Serving capacity

Plan context length and concurrency together

Choose the context and expected simultaneous requests first, then use the remaining VRAM to test a serving capacity boundary.

Decision 01

Context is a capacity decision

Longer prompts and responses reserve more KV Cache. Start with the useful context for the workload instead of selecting the largest number available.

Decision 02

Concurrency multiplies the cache

Each active request needs its own cache budget. Serving plans should state their target concurrency rather than borrowing a single-user result.

Decision 03

Tune the pressure point

When capacity is tight, compare reduced context, reduced concurrency, a smaller weight format, or a larger aggregate VRAM target as distinct paths.

Continue planning

Related decisions

Plan vLLM serving capacity

vLLM model length

Set vLLM max model length from the KV Cache budget

vLLM concurrency

Plan vLLM concurrency before raising max_num_seqs

Sources

Reviewed 2026-07-15. Open the primary references behind this planning path.

vLLM cache configurationGPU memory utilization, KV Cache capacity, and per-engine concurrency at the configured model length.Open source ↗vLLM model configurationHow max model length is derived or set as a capacity boundary.Open source ↗vLLM parallelism and scalingTensor-parallel sizing for multi-GPU inference and serving.Open source ↗vLLM data-parallel deploymentSeparate DP engines, independent KV caches, and the per-engine meaning of max_num_seqs.Open source ↗