LoRA mode

Choose LoRA or QLoRA from the base-weight budget

LoRA reduces the number of trainable parameters, while QLoRA also changes the frozen base-weight footprint. Neither choice removes the need to plan activations.

Decision 01

Separate frozen base memory from trainable state

The base model remains present during adapter training. LoRA adds small trainable matrices, while a quantized base can materially reduce the persistent weight portion of the memory plan.

Decision 02

Use the mode to solve the actual constraint

If the base weights leave too little room for a useful training setup, compare a QLoRA baseline. If the base fits with room for activations, compare the simpler LoRA path at the same sequence and micro-batch target.

Decision 03

Keep the workload fixed while comparing

Use the same model size, sequence length, micro-batch size, and GPU capacity in both planner modes. That makes the source of the capacity difference explicit.

Continue planning

Related decisions

Compare LoRA and QLoRA

Training memory

Set sequence length and micro-batch before increasing VRAM

Memory model

Separate model weights, KV Cache, and runtime reserve

Sources

Reviewed 2026-07-15. Open the primary references behind this planning path.

Hugging Face quantization documentationWeight formats, 8-bit loading, and QLoRA implementation context.Open source ↗Hugging Face PEFT LoRA guideFrozen base weights, low-rank adapters, and QLoRA-oriented implementation context.Open source ↗Hugging Face GPU memory usageHow weights, optimizer state, activations, sequence length, and batch size occupy GPU memory.Open source ↗Hugging Face FSDP2 guideHow FSDP shards model, gradient, and optimizer state across GPUs, unlike replicated DDP training.Open source ↗