Memory model

Separate model weights, KV Cache, and runtime reserve

Inference planning begins with the loaded weights, then adds the KV Cache for each active request and a visible runtime reserve.

Decision 01

Weights set the starting point

The selected weight format changes the largest persistent memory block. Lower-bit formats reduce weight memory, while metadata and runtime implementation still consume capacity.

Decision 02

KV Cache grows with the request

Context length and concurrent requests both increase KV Cache demand. Architectures with fewer KV heads can materially change this part of the plan.

Decision 03

Leave operating room

Use a visible runtime reserve rather than treating every reported GB as model capacity. The result should leave room for the runtime, allocations, and normal variation.

Continue planning

Related decisions

Build an inference plan

Serving capacity

Plan context length and concurrency together

vLLM model length

Set vLLM max model length from the KV Cache budget

Sources

Reviewed 2026-07-15. Open the primary references behind this planning path.

Hugging Face quantization documentationWeight formats, 8-bit loading, and QLoRA implementation context.Open source ↗vLLM cache configurationGPU memory utilization, KV Cache capacity, and per-engine concurrency at the configured model length.Open source ↗