A 2-input, 2-hidden, 1-output binary classifier. Every edge is a learned weight stored as a 32-bit float. These are the numbers we want to compress to INT4.
Input layer
Hidden layer
Output layer
Edge weight (fp32)
Six weights in total: four from input→hidden (0.43, 0.07, 0.91, 0.02) and two from hidden→output (0.65, 0.38). In fp32, each weight uses 4 bytes. Goal: compress each to 4 bits (INT4) — 8× smaller — without destroying the model's predictions.
The core challenge
One outlier ruins the whole scale
INT4 gives you exactly 16 integer buckets. The scale is set by the full weight range. One large outlier forces a scale so coarse that all normal weights collapse into the same bucket.
Outlier weight
Normal weights (clustered)
INT4 buckets
This is the enemy. Not just theoretical — transformer residual streams regularly contain activation values 100× larger than the typical magnitude. The three methods on the next slides each solve this differently.
Method 1
GPTQ — quantize by importance, compensate errors
GPTQ runs a calibration set through the model to compute the Hessian diagonal — a measure of how sensitive the loss is to each weight. High-sensitivity weights get quantized carefully; their rounding error is pushed to neighbors.
High Hessian (very important)
Medium importance
Low importance
Error propagation
GPTQ treats quantization as an optimization problem over the full weight matrix. It uses the Hessian to find the order that minimizes total output error, compensating downstream weights for the rounding error introduced at each step.
Limitation: Computing the Hessian needs a calibration dataset and takes hours for large models. GPTQ is weight-only — activations still run at fp16. Excellent quality at INT3/INT4 but expensive to run.
Method 2
SmoothQuant — migrate the difficulty
The problem with quantizing activations: they vary per input, so you can't measure their range ahead of time. SmoothQuant's insight: mathematically move the outlier "difficulty" from activations into weights — which are static and can be pre-calibrated.
Activation outlier
Smooth factor s = √outlier
After smoothing — safe to quantize
SmoothQuant is the key to quantizing activations, not just weights. By migrating the outlier scale from X into W, both tensors become well-behaved — and you can quantize them both to INT8 using standard methods.
What this enables: W8A8 inference — weight 8-bit, activation 8-bit. NVIDIA's INT8 tensor cores run integer matrix multiplications natively, giving up to 4× throughput vs fp16. SmoothQuant is used in TensorRT-LLM and vLLM for high-throughput serving.
Method 3
AWQ — protect the salient 1%
AWQ asks a simpler question than GPTQ: which weight channels matter most? Its answer: the ones with consistently large activations. Scale those channels up before quantization so they use more of the INT4 range.
Salient channel (large avg |x|)
Medium activation
Inactive channel
AWQ protects the 1% of weight channels that matter most. Finding them takes one forward pass — no Hessian needed. Scale up before quantization, scale outputs back down — the math is exact, the quality gain is large.
Why AWQ won in practice: Nearly as accurate as GPTQ but runs in minutes instead of hours. Most production INT4 checkpoints today (Mistral, Qwen, LLaMA) use AWQ. It's the default in Hugging Face's AutoAWQ library.
Summary
Three methods, three angles of attack
All three share the same goal — minimize output error under INT4/INT8 quantization — but each attacks the problem differently. In production, they're often combined.
Migrate activation outliers into weights via scale factor
Quant target
Weights + activations
Speed
Fast — one forward pass
Best for
W8A8 on INT8 tensor core hardware
AWQ
Core idea
Scale up salient channels (top 1% by activation)
Quant target
Weights only
Speed
Very fast — minutes
Best for
Production INT4 — most widely used today
The progression tells a story: GPTQ proved INT4 was viable for LLMs (2022). SmoothQuant extended that to activations, unlocking hardware speedups (2022). AWQ simplified the whole thing into something fast enough for everyday use (2023). Each paper fixed a specific failure mode of the previous one.
In practice today: Local inference (llama.cpp) → GGUF Q4_K_M uses AWQ-style channel scaling. Production serving (vLLM) → AWQ INT4 for memory, SmoothQuant W8A8 for throughput. Fine-tuning → QLoRA, which quantizes base model weights with NF4 (a 4-bit float variant), then fine-tunes small adapters in fp16.