vLLM model length

Set vLLM max model length from the KV Cache budget

In vLLM, maximum model length is not just a model capability: it is a capacity reservation for prompt and output tokens across active requests.

Decision 01

Start from the useful request envelope

Define the longest prompt-plus-output path your endpoint needs to complete. Set the model-length target from that workload envelope, not from the model card maximum by default.

Decision 02

Make the reservation visible

After loaded weights and runtime headroom, the remaining GPU memory becomes the KV Cache budget. At a fixed model length, that budget determines how many full-length requests can coexist.

Decision 03

Test the boundary in the serving planner

Enter the proposed model length and target request count together. If cache pressure is tight, lower one of those two requirements before changing the GPU target or tensor-parallel shape.

Continue planning

Related decisions

Plan max model length

Serving capacity

Plan context length and concurrency together

vLLM concurrency

Plan vLLM concurrency before raising max_num_seqs

Sources

Reviewed 2026-07-15. Open the primary references behind this planning path.

vLLM cache configurationGPU memory utilization, KV Cache capacity, and per-engine concurrency at the configured model length.Open source ↗vLLM model configurationHow max model length is derived or set as a capacity boundary.Open source ↗vLLM parallelism and scalingTensor-parallel sizing for multi-GPU inference and serving.Open source ↗vLLM data-parallel deploymentSeparate DP engines, independent KV caches, and the per-engine meaning of max_num_seqs.Open source ↗