Memory model

Separate model weights, KV Cache, and runtime reserve

Inference planning begins with the loaded weights, then adds the KV Cache for each active request and a visible runtime reserve.

Weights set the starting point

The selected weight format changes the largest persistent memory block. Lower-bit formats reduce weight memory, while metadata and runtime implementation still consume capacity.

KV Cache grows with the request

Context length and concurrent requests both increase KV Cache demand. Architectures with fewer KV heads can materially change this part of the plan.

Leave operating room

Use a visible runtime reserve rather than treating every reported GB as model capacity. The result should leave room for the runtime, allocations, and normal variation.

Continue planning

Test this memory constraint with a live capacity plan.

Build an inference plan

Serving capacity

Plan context length and concurrency together

vLLM model length

Set vLLM max model length from the KV Cache budget

Editorial record

Maintained by Make Your Own Tools to turn “Fit an LLM workload” into a defined weights, context or activations, runtime reserve, and the next hardware boundary. The references below provide the technical context for this planning path. Its calculation rules and planning assumptions are documented in the methodology, and affected calculations pass regression checks before the review date advances.

Last reviewed
Evidence set
2 primary references
Calculation coverage
3 affected rules checked
Read methodology →View change history ↗
Planning scope
LLM inference memory planning from model architecture and workload inputs.
Next check
Carry the capacity range into the exact model, quantization, runtime, kernel, and hardware configuration.
Hugging Face quantization documentationWeight formats, 8-bit loading, and QLoRA implementation context.Open source ↗vLLM cache configurationGPU memory utilization, KV Cache capacity, and per-engine concurrency at the configured model length.Open source ↗