vLLM concurrency
Plan vLLM concurrency before raising max_num_seqs
Higher sequence concurrency only works when the KV Cache can support the chosen model length; a single-user fit is not a serving-capacity result.
Decision 01
Use full-length concurrency as the planning baseline
Plan for the concurrent active requests that may reach the configured maximum model length. Shorter real-world requests can create headroom, but they should not be the only capacity assumption.
Decision 02
Treat cache pressure as a scheduling signal
When the cache budget is close to the requested total, queueing, recomputation, or request limits can become part of normal operation. Keep a planning margin rather than treating the theoretical maximum as the default setting.
Decision 03
Change one lever at a time
Reduce the model-length target, reduce concurrent active requests, select a smaller weight footprint, or increase aggregate VRAM. Each path has a different effect on product behavior and cost.