Quantization is a numerical and systems experiment
How to study this lesson
Read one section, trace the worked example, and try the knowledge check. Bookmark saves a shortcut; notes appear in My notebook. Mark lesson complete is your own assessment and does not mark its lab passed.
By the end, you can
- Derive a simple symmetric quantization rule.
- Distinguish weight, activation, and KV-cache precision.
- Evaluate speed, memory, and quality together.
Map values onto a finite grid
A simple symmetric integer quantizer selects a scale s and maps a value x to an integer q by rounding x divided by s and clipping to a supported integer range. Dequantization reconstructs approximately s times q. The NVIDIA TensorRT documentation describes explicit quantization semantics and supported formats; real implementations differ in integer ranges, scaling granularity, rounding, and hardware paths. The demonstration here deliberately uses a symmetric range from -qmax to qmax so the arithmetic is easy to inspect.
If the largest absolute value is one and a toy three-bit signed scheme uses qmax equal to three, the scale is one third. A value 0.2 maps to integer one and reconstructs as approximately 0.3333, producing error 0.1333. Zero remains exactly zero. The endpoints reconstruct exactly in this example, but the small interior value is relatively distorted. Quantization error depends on the values as well as the number of bits.
Scale granularity trades metadata for fidelity
One scale for a whole tensor is cheap to store but can let a single outlier dominate resolution. Suppose most weights lie between -0.1 and 0.1 while one is 10. A shared low-bit grid may represent many ordinary values poorly. Per-channel or grouped scales can give smaller regions a more appropriate range, at the cost of scale metadata and potentially different kernel behavior. The choice must match the quantized format and serving implementation.
Calibration, when required by a method, estimates useful ranges or other statistics from representative data. A tiny calibration sample that contains only short English prompts may miss activation behavior under long documents, other languages, or tool-heavy interactions. Keep calibration separate from the final evaluation set and record its provenance. Methods designed to preserve important outliers or compensate for errors can help, but their benefits remain workload- and implementation-dependent.
Storage precision is not every operation's precision
Weight-only quantization compresses stored model weights while computation may use a different arithmetic format after dequantization or in specialized kernels. Activation quantization and KV-cache quantization are separate choices with different memory and accuracy consequences. A four-bit model label is therefore incomplete unless it states what is stored in four bits, how scales are represented, and which operations remain at higher precision.
Hugging Face's cache documentation distinguishes dynamic, static, offloaded, and quantized cache strategies and notes that reducing cache memory can add latency in some workloads. A smaller payload may permit a larger useful batch, yet dequantization overhead can hurt a lightly loaded short-context request. Likewise, a format supported in software may not have the fastest kernel on a particular device. Measure the exact artifact, runtime, and hardware combination that will be deployed.
A local numerical error can have a discontinuous outcome
Mean squared reconstruction error describes tensor approximation, not end-to-end behavior. A small change in logits can alter the highest-probability token when two candidates are close, after which autoregressive generation follows a different context. Some changes are harmless wording variations; others break a schema, choose an incorrect tool, or remove a required condition. Evaluate task outcomes and sensitive slices rather than assuming small weight error guarantees small product impact.
Use paired examples to compare a reference and quantized artifact under matched decoding settings. Separate format failures, factual failures, routing changes, and abstention differences. Include long-context cases if cache precision changes. A quality metric that averages across easy prompts may conceal damage in rare but important operations. Report both average performance and the specific failure modes relevant to the application; numerical fidelity and task fidelity answer complementary questions.
Choose from a constrained tradeoff surface
Make a small comparison matrix with precision configuration, measured peak memory, accepted requests per second, first-token latency, completion latency, and task success. Hold workload and hardware fixed where possible. Mark unavailable combinations honestly rather than filling missing cells with extrapolated speedups. Include warmup and compilation policy, since a short benchmark can be dominated by setup costs or omit them entirely.
The best configuration is one that meets the task's quality and latency requirements within its resource budget. The smallest file may not be the fastest service, and the fastest microbenchmark may not sustain the desired concurrency. Treat analytical memory calculations as preflight estimates, then validate them under load. Keep a rollback path to the reference artifact and version the quantization procedure, so a numerical regression can be reproduced and corrected without guessing which export option changed.
Work through the code
This list-based toy quantizer uses one symmetric scale and Python rounding. It does not implement a production model format or hardware kernel. Change the input to include a large outlier and examine how that changes reconstruction of small values.
def quantize(values, bits):
if bits < 2:
raise ValueError("at least two bits required")
qmax = 2 ** (bits - 1) - 1
maximum = max((abs(value) for value in values), default=0.0)
scale = maximum / qmax if maximum else 1.0
integers = [max(-qmax, min(qmax, round(value / scale)))
for value in values]
restored = [integer * scale for integer in integers]
error = max((abs(a - b) for a, b in zip(values, restored)), default=0.0)
return integers, error
values = [-1.0, -0.2, 0.0, 0.2, 1.0]
for bits in (3, 8):
integers, error = quantize(values, bits)
print(f"{bits}-bit integers: {integers}")
print(f"maximum absolute error: {error:.6f}")
assert quantize([0.0, 0.0], 3) == ([0, 0], 0.0)
3-bit integers: [-3, -1, 0, 1, 3] maximum absolute error: 0.133333 8-bit integers: [-127, -25, 0, 25, 127] maximum absolute error: 0.003150
Run Python snippets locally with the prerequisites named above. The module coding lab runs directly in your browser.
Pause and reason
A quantized artifact reduces model-weight memory by 50 percent but increases single-request latency by 15 percent. Give two reasons it may still be useful and two measurements needed before deciding.
Check your understanding
Why can a quantized model with low weight reconstruction error still make different tool choices?
Your notes
Explain the mechanism in your own words. Add a failure you want to test.