Quantization Theory

Source Excerpt

The mathematical foundations and practical techniques for reducing neural network precision — from FP32/FP16 floating point to INT8, INT4, and beyond. Quantization enables running large models on constrained hardware with minimal quality degradation.

Why Quantization Works

Neural networks are remarkably robust to precision reduction. Key reasons:

1. Over-parameterization: Models have far more parameters than needed. Reducing precision removes redundancy.
2. Weight distribution: Most weights are small. Large weights (outliers) dominate computation and can be preserved at higher precision.
3. Activation saturation: Non-linearities (ReLU, GELU) saturate, making small precision differences irrelevant.
4. Ensemble effect: Millions of small errors average out, preserving overall model behavior.

Floating Point Formats

FP32 (32-bit)

Sign (1 bit) | Exponent (8 bits) | Mantissa (23 bits)

• Range: ±1.4×10⁻⁴⁵ to ±3.4×10³⁸
• Precision: ~7 decimal digits
• Standard for training

FP16 (16-bit)

Sign (1 bit) | Exponent (5 bits) | Mantissa (10 bits)

• Range: ±6.1×10⁻⁵ to ±65504
• Precision: ~3 decimal digits
• Standard for inference (GPU optimized)

BF16 (Brain Floating Point, 16-bit)

Sign (1 bit) | Exponent (8 bits) | Mantissa (7 bits)

• Range: Same as FP32 (±3.4×10³⁸)
• Precision: ~2 decimal digits
• Better numerical stability than FP16

Comparison

Integer Quantization

Symmetric Quantization

Map floating-point range [-max, +max] to integer range [-127, +127] (INT8):

scale = max(|W|) / 127
W_int8 = round(W_fp32 / scale)
W_fp32 ≈ W_int8 × scale

Advantage: Simple, fast dequantization (multiply by scale).

Disadvantage: Wastes range if weights are asymmetric.

Asymmetric Quantization

Map floating-point range [min, max] to integer range [0, 255] (UINT8):

scale = (max - min) / 255
zero_point = round(-min / scale)
W_uint8 = round(W_fp32 / scale) + zero_point

Advantage: Better utilization of integer range.

Disadvantage: Slightly more complex dequantization.

Post-Training Quantization (PTQ)

Quantize a pre-trained model without additional training. Fastest approach, good for INT8.

Methods

Min-max calibration: Use a small calibration dataset to find min/max activation values.

For each layer:
  1. Run calibration data through layer
  2. Record min/max of activations and weights
  3. Compute scale and zero_point
  4. Quantize weights

Percentile calibration: Use percentiles (e.g., 99.9th) instead of min/max to handle outliers.

K-means clustering: Cluster weight values, assign cluster centers to integer levels.

Quality

Quantization-Aware Training (QAT)

Simulate quantization effects during training. Better quality than PTQ, especially for INT4 and below.

How It Works

1. Forward pass: Simulate quantized weights and activations
2. Straight-through estimator (STE): Pass gradients through quantization (non-differentiable)
3. Backward pass: Update full-precision weights with simulated gradients

W_quantized = round(W / scale) × scale  (forward)
∂L/∂W = ∂L/∂W_quantized  (STE: gradient passes through)

Quality

Advanced Quantization Techniques

AWQ (Activation-Aware Weight Quantization)

Identify and protect "outlier" weights that are important for model quality.

1. Analyze activation magnitudes across channels
2. Identify important channels (high activation magnitude)
3. Keep important channels at higher precision
4. Quantize remaining channels aggressively

Result: Better INT4 quality than uniform quantization.

SpQR (Sparsification + Quantization)

Combine weight sparsification with quantization:

1. Prune small weights (sparsification)
2. Quantize remaining weights
3. Result: Sparse + quantized model

Result: Higher compression than quantization alone.

OPTQ (Optimized Parameter-aware Text Quantization)

Iteratively select and quantize layers, optimizing for overall model quality:

1. Evaluate layer importance (sensitivity analysis)
2. Quantize least important layers first
3. Update importance scores
4. Repeat until target precision reached

Result: Better quality than layer-by-layer quantization.

NF4 (NormalFloat4) — QLoRA

4-bit normal浮点 format designed for QLoRA:

• Information-theoretically optimal for normally-distributed weights
• 4-bit floating point (not integer)
• Better than INT4 for model weights (which are approximately normal)

Quantization and Hardware

GPU Support

CPU Support

Practical Guidance

When to Use Which Precision

Quality Preservation Tips

1. Keep attention weights at higher precision (more sensitive to quantization)
2. Use mixed precision (K-quantization: some layers Q8, others Q4)
3. Calibrate with representative data (domain-matched calibration set)
4. Evaluate before deploying (run benchmark prompts on quantized model)
5. Keep FP16 original (for re-quantization if needed)

Source excerpt truncated at 220 of 225 lines. Open the canonical wiki path above for the full page.