Training memory

Set sequence length and micro-batch before increasing VRAM

Adapter training can still exhaust a GPU when sequences or micro-batches grow. Gradient checkpointing exchanges some compute time for a lower activation-memory requirement.

Sequence length is a first-order memory lever

Activation memory grows with the batch and sequence dimensions, and long-context attention can add further pressure. Choose a sequence target that matches the training examples rather than the largest supported window.

Micro-batch and effective batch are separate choices

Lower the per-device micro-batch when memory is the constraint, then use gradient accumulation when you need a larger effective batch. This preserves the distinction between GPU fit and optimization schedule.

Use checkpointing after setting the workload

Gradient checkpointing saves fewer activations and recomputes them during backward passes. Enable it when the desired sequence and micro-batch do not leave enough activation headroom, then account for the additional compute time.

Continue planning

Test this memory constraint with a live capacity plan.

Plan a LoRA training run

LoRA mode

Choose LoRA or QLoRA from the base-weight budget

Serving capacity

Plan context length and concurrency together

Editorial record

Maintained by Make Your Own Tools to turn “Fit an LLM workload” into a defined weights, context or activations, runtime reserve, and the next hardware boundary. The references below provide the technical context for this planning path. Its calculation rules and planning assumptions are documented in the methodology, and affected calculations pass regression checks before the review date advances.

Last reviewed
Evidence set
4 primary references
Calculation coverage
2 affected rules checked
Read methodology →View change history ↗
Planning scope
Adapter-training memory planning from sequence, micro-batch, and checkpointing choices.
Next check
Carry the workload into the exact model, framework, optimizer, precision, distributed strategy, and measured training step.
Hugging Face gradient checkpointingActivation-memory reduction through recomputation and its training-time tradeoff.Open source ↗Hugging Face GPU memory usageHow weights, optimizer state, activations, sequence length, and batch size occupy GPU memory.Open source ↗Hugging Face PEFT LoRA guideFrozen base weights, low-rank adapters, and QLoRA-oriented implementation context.Open source ↗Hugging Face FSDP2 guideHow FSDP shards model, gradient, and optimizer state across GPUs, unlike replicated DDP training.Open source ↗