Low-rank updates and the real memory bill
How to study this lesson
Read one section, trace the worked example, and try the knowledge check. Bookmark saves a shortcut; notes appear in My notebook. Mark lesson complete is your own assessment and does not mark its lab passed.
By the end, you can
- Derive the parameter count of a low-rank update.
- Separate frozen weights, optimizer state, and activation memory.
- Explain the role and limitations of QLoRA.
Factor the change, not the whole model
LoRA represents an update to a frozen weight matrix using a product of two smaller trainable matrices. For a weight W with output dimension d_out and input dimension d_in, write the adapted transformation as W x plus (alpha/r) B A x. A has shape r by d_in, B has shape d_out by r, and r is the chosen rank. The original LoRA paper establishes this parameter-efficient adaptation construction. The trainable update is constrained to a low-dimensional family; the frozen base matrix itself need not be low rank.
For an invented 4,096 by 4,096 projection, full updating exposes 16,777,216 parameters. Rank 8 exposes 8 × (4,096 + 4,096) = 65,536 adapter parameters, before any separately trained biases or other modules. That is 256 times fewer trainable parameters for this matrix, not a promise of 256 times less total memory or training time.
Follow a tiny forward pass
Take a two-output, three-input layer. Let W contain rows [1, 0, 0] and [0, 1, 0], A be [1, 2, 3], B contain [0.1] and [0.2], and alpha/r equal two. For x = [1, 0, -1], the frozen output is [1, 0]. A x is -2, B A x is [-0.2, -0.4], and the scaled update is [-0.4, -0.8]. The adapted output becomes [0.6, -0.8].
This arithmetic shows why two matrix multiplications can describe a useful change without materializing a full update during training. Initialization must allow useful gradients: setting both factors to zero prevents either factor from receiving a learning signal through their product initially. A common construction initializes one factor nonzero and the other zero, preserving the base output at initialization while permitting the zero factor to learn.
Account for every category of memory
Trainable parameter count is only one memory term. Frozen base weights still occupy space. Trainable parameters may require gradients, optimizer moments, and higher-precision copies depending on the optimizer and mixed-precision implementation. Activations retained for backward computation can be large, especially at long sequence lengths. Temporary workspaces and allocator behavior add further overhead.
A hypothetical seven-billion-parameter base uses about 14 billion bytes at two bytes per parameter, approximately 13.04 GiB, before other allocations. Rank reduction cannot remove those frozen weights. Gradient checkpointing trades repeated forward computation for fewer retained activations. Shorter sequences, smaller microbatches, and accumulation can affect memory differently. Record measured peak allocated memory and the device capacity separately; a configuration that fits a tiny batch does not establish that an intended production training workload fits.
What quantized adaptation changes
QLoRA combines frozen quantized base weights with trainable low-rank adapters. Its paper describes a four-bit storage approach, a quantization scheme designed for normally distributed weights, double quantization of quantization constants, and paged optimizers. Computation involves suitable higher-precision representations; four-bit weight storage does not mean every operation, gradient, activation, and optimizer value uses four bits. Treat each precision choice as a separate configuration field.
For the same invented seven-billion-parameter count, four bits per base weight implies an ideal payload of 3.5 billion bytes, about 3.26 GiB. Scales, metadata, unquantized components, adapters, activations, and workspaces increase the real requirement. This estimate is useful for rejecting impossible budgets, but it is not a substitute for a measured run. Quantization may change numerical behavior, so the adapted model needs evaluation against both the unadapted quantized base and any full-precision reference relevant to deployment.
Choose rank by evidence and deploy the right artifact
Rank is a capacity and cost choice, not a quality score. Try a small set of ranks while holding the data and evaluation procedure fixed, and compare task success, regression checks, memory, and training time. Targeting different projections changes the set of possible updates. State exactly which modules were adapted and which additional parameters were trainable; otherwise two runs labeled LoRA may be incomparable.
Package the adapter with the base-model identifier, revision, tokenizer, template, training configuration, and evaluation manifest. Loading an adapter onto a different base revision can invalidate assumptions even if shapes match. Merging an adapter into a compatible base can remove separate adapter computation, but quantized merging and requantization may introduce numerical differences or operational constraints. Re-evaluate the exact deployed artifact, including its precision and serving format, before transferring any training-time result to production.
Work through the code
This is an exact low-rank forward calculation using lists, not a training implementation or quantizer. The tiny dimensions make the adapter only slightly smaller than the base. Larger square matrices with low rank produce a much larger parameter reduction.
def matvec(matrix, vector):
if any(len(row) != len(vector) for row in matrix):
raise ValueError("incompatible dimensions")
return [sum(a * b for a, b in zip(row, vector))
for row in matrix]
weights = [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0]]
factor_a = [[1.0, 2.0, 3.0]]
factor_b = [[0.1], [0.2]]
vector = [1.0, 0.0, -1.0]
rank, alpha = 1, 2.0
base = matvec(weights, vector)
low_rank = matvec(factor_b, matvec(factor_a, vector))
adapted = [w + alpha / rank * delta
for w, delta in zip(base, low_rank)]
trainable = sum(map(len, factor_a)) + sum(map(len, factor_b))
print("base:", base)
print("adapted:", [round(value, 3) for value in adapted])
print("adapter parameters:", trainable)
base: [1.0, 0.0] adapted: [0.6, -0.8] adapter parameters: 5
Run Python snippets locally with the prerequisites named above. The module coding lab runs directly in your browser.
Pause and reason
For a 2,048 by 8,192 weight matrix with rank 16, compute the full and adapter parameter counts. Explain why the ratio cannot directly predict peak GPU memory.
Check your understanding
A four-bit adapter-training configuration runs out of memory only when context length increases. Which explanation is most plausible?
Your notes
Explain the mechanism in your own words. Add a failure you want to test.