Our toy network — before quantization

6 weights, all fp32. We'll quantize these three ways and watch what happens to each weight.

w₁₁=0.43 w₁₂=0.07 w₂₁=0.91 w₂₂=0.02 w₃₁=0.65 w₃₂=0.38 x₁ 1.20 x₂ 0.80 h₁ 0.56 h₂ 0.74 ŷ 0.81 Input Hidden Output Weights (fp32) w₁₁ = 0.43 w₁₂ = 0.07 w₂₁ = 0.91 w₂₂ = 0.02 w₃₁ = 0.65 w₃₂ = 0.38
Weight range
0.02 → 0.91
Naïve INT4 scale s
(0.91−0.02)/15 = 0.059
Max rounding error
±0.030 per weight

GPTQ — step by step

Second-order quantization: use the Hessian to decide what to quantize first, compensate errors downstream.

Run calibration data — compute H per weight
Pass a small calibration dataset through the frozen model. For each weight, compute the diagonal of the Hessian of the loss — how steeply the loss changes when you perturb that weight.
calib data x₁ x₂ h₁ h₂ ŷ H=8.4 HIGH H=0.3 low H=3.1 med H=0.1 low H=5.2 HIGH H=2.8 med Hessian diagonal = ∂²Loss / ∂w² Large H = weight strongly affects loss. Round it carelessly → big output error.
Each edge now has a Hessian score. w₁₁ (H=8.4) and w₃₁ (H=5.2) are the most sensitive — small perturbations to these weights cause large changes in the loss. These get quantized first with the finest possible bucket placement.
Rank weights by Hessian — set the quantization order
Sort all weights by H (descending). High-H weights are quantized first — their precision matters most. Low-H weights go last and can tolerate coarser rounding.
Weightfp32 valueHessian HPrioritySensitivity
w₁₁0.43 H = 8.4 ① First Very high — quantize with finest care
w₃₁0.65 H = 5.2 ② Second High — also needs careful placement
w₂₁0.91 H = 3.1 ③ Third Medium — absorbs error from above
w₃₂0.38 H = 2.8 ④ Fourth Medium
w₁₂0.07 H = 0.3 ⑤ Fifth Low — coarse rounding acceptable
w₂₂0.02 H = 0.1 ⑥ Last Very low — almost zero impact
Why order matters: GPTQ quantizes greedily — one weight at a time — and the rounding error from each step gets propagated forward to remaining weights. Starting with the most important weights means subsequent compensation steps have more "budget" to absorb errors without affecting the loss much.
Quantize w₁₁ first — show the rounding
With scale s = (0.91−0.02)/15 = 0.059 and zero-point z = round(0.02/s) = 0, we map w₁₁ = 0.43 to its nearest INT4 bucket.
0 1 2 3 4 5 6 7 8 9 15 7 w₁₁ = 0.430 INT4=7 → 0.433 δ = −0.003 (very small — good bucket fit) 0.02 0.91 s = 0.059 per bucket
w₁₁ = 0.430 maps to INT4 bucket 7 (reconstructed as 0.433). Rounding error δ = −0.003 — tiny because the weight range is compact and the scale is fine. This error now gets propagated to the next weight (w₃₁) for compensation.
Compensate — push the error into w₃₁
The rounding error δ from quantizing w₁₁ is "pushed" to w₃₁ (the next highest-H weight) by adjusting its value before quantizing it. The adjustment is scaled by the relative Hessian values.
BEFORE compensation
w₁₁ error δ = −0.003
w₃₁ original = 0.650
H(w₁₁) = 8.4,  H(w₃₁) = 5.2
AFTER compensation
adjustment = δ × H(w₁₁)/H(w₃₁)
              = −0.003 × 8.4/5.2
              = −0.005
w₃₁ adjusted = 0.650 − 0.005 = 0.645
The compensation slightly shifts w₃₁ before quantizing it. When w₃₁ is then rounded to its nearest INT4 bucket, the rounding error from w₁₁ has been partially absorbed. The layer's total output error is minimized, not just each individual weight's error.
This is what makes GPTQ "second-order": It's not just minimizing per-weight rounding error. It's minimizing the layer's output error, accounting for the curvature of the loss surface (the Hessian). Naïve INT4 minimizes each weight in isolation; GPTQ minimizes the system jointly.
Post-quantization — GPTQ result network
All 6 weights are now stored as INT4 integers (0–15). The network edges show both the INT4 value and the reconstructed float. Error is small everywhere because GPTQ compensated each step.
INT4:7 → 0.433 err: −0.003 ✓ INT4:1 → 0.079 err: +0.009 ✓ INT4:15→ 0.905 err: −0.005 ✓ INT4:0 → 0.020 err: 0.000 ✓ INT4:11→ 0.649 err: −0.001 ✓ INT4:6 → 0.374 err: −0.006 ✓ x₁ x₂ h₁ h₂ ŷ GPTQ result Max error: 0.009 Avg error: 0.004 All INT4 ✓ Output ŷ: 0.809 (was 0.81)
All weights are INT4. Every edge shows the integer bucket, reconstructed float, and rounding error. The output prediction barely changed: 0.809 vs original 0.810. Error compensation across weights is what makes this work.
Step 1 of 5

SmoothQuant — step by step

Migrate activation outliers into weights using a per-channel scale factor, then quantize both to INT8.

Identify the activation outlier
During a calibration forward pass, record the actual activation values flowing through the network. One activation dimension (channel 3, corresponding to h₂→ŷ) is consistently large: 48.2 vs typical values of 0.5–1.5.
activation = 48.2 ⚠ outlier! act = 0.74 x₁ x₂ h₁ h₂ ⚠ outlier ŷ The problem act range: 0–48.2 INT8 scale = 48.2/127 = 0.380 per bucket Normal acts → 1–4 buckets!
The h₂ activation (48.2) is ~65× larger than the typical h₁ activation (0.74). If we quantize activations naively to INT8, the scale becomes 0.380 — meaning normal activations only span 1–4 of the 255 available INT8 buckets. All precision is wasted on the empty range 1.5 → 48.2.
Compute per-channel smooth factor s
For each weight channel, compute s = (max |activation|)^α where α ∈ [0,1] controls how much difficulty to move from activations to weights. α=0.5 (geometric mean) works well in practice.
Channel h₁ → ŷ (normal)
max |act| = 0.74
s₁ = 0.740.5 = 0.860
x̂ = x / 0.860  ← mild shift
Ŵ = W × 0.860 ← mild shift
Channel h₂ → ŷ (outlier)
max |act| = 48.2
s₂ = 48.20.5 = 6.943
x̂ = x / 6.943  ← big shift
Ŵ = W × 6.943 ← big shift
The smooth factor s is channel-specific. For the normal channel, s ≈ 0.86 — a mild adjustment. For the outlier channel, s = 6.94 — a large shift that will shrink the activation from 48.2 down to 6.94, making it much easier to quantize.
Apply the transform — x̂ = x/s, Ŵ = W×s
Divide each activation by its channel's s, multiply each weight by s. The output y = x·W is unchanged because (x/s)·(W×s) = x·W. But now both tensors are quantization-friendly.
ChannelBefore: act / weights factorAfter: act / weightChange
h₁→ŷ (w₃₁) act=0.74 / w=0.65 s₁ = 0.860 act=0.860 / w=0.559 mild
h₂→ŷ (w₃₂) act=48.2 / w=0.38 s₂ = 6.943 act=6.943 / w=2.638 outlier tamed
Before (act range): 0.74 48.2 INT8 scale = 48.2/127 = 0.380 — wasteful! After smoothing (act range): 0.860 6.943 INT8 scale = 6.943/127 = 0.055 — 7× better!
The activation range shrank from 0–48.2 down to 0–6.943. The INT8 scale improved 7×. Both activations and weights can now be safely quantized to INT8, with the output unchanged mathematically.
Quantize both to INT8 — W8A8
Now quantize the smoothed activations x̂ and the scaled weights Ŵ both to INT8. Since both tensors are well-behaved, precision is high. The final matmul runs as integer arithmetic on hardware.
QuantitySmoothed valueINT8 bucketReconstructedError
act h₁ (x̂₁)0.860INT8: 160.875+0.015
act h₂ (x̂₂)6.943INT8: 1276.985+0.042
weight Ŵ₃₁0.559INT8: 180.567+0.008
weight Ŵ₃₂2.638INT8: 852.646+0.008
W8A8 unlocks hardware speedups: NVIDIA Ampere and Hopper GPUs have INT8 tensor cores that run matrix multiplications 2–4× faster than FP16. With SmoothQuant taming the activations, the entire attention and FFN matmul can run in INT8 — not just weight storage.
Post-quantization — SmoothQuant result network
Edges show INT8 integer values (not INT4 — SmoothQuant targets W8A8). The weights absorbed the outlier scale; activations flow as INT8. Both with small error everywhere.
INT8:55 → 0.429 err: −0.001 ✓ INT8:9 → 0.071 err: +0.001 ✓ INT8:116→ 0.908 err: −0.002 ✓ INT8:3 → 0.024 err: +0.004 ✓ Ŵ: INT8:18→0.567 s=0.86 absorbed Ŵ: INT8:85→2.646 s=6.94 absorbed act: INT8 act: INT8 x₁ x₂ h₁ h₂ ŷ SmoothQuant Format: INT8 Weights: INT8 ✓ Activations: INT8 ✓ Output ŷ: 0.808 (was 0.81) W8A8 matmul ✓
Purple edges show the smoothed output-layer weights (Ŵ) — they absorbed the outlier scale. Both weights and activations are INT8. The full matmul now runs as integer arithmetic. Output prediction: 0.808 vs original 0.810.
Step 1 of 5

AWQ — step by step

Find the 1% of weight channels that matter most (salient), scale them up before INT4 quantization so they get finer buckets.

Score each weight channel by activation magnitude
Run a calibration forward pass and record the average absolute activation for each hidden-layer channel. The idea: if a channel's activation is always large, its weight contributes a lot to the output — and rounding errors in that weight get amplified.
avg|h₁|=0.65 avg|h₂|=1.82 ★ output error risk: δw × 1.82 ← amplified δw × 0.65 ← ok x₁ x₂ h₁ h₂ salient ŷ
h₂ has avg activation 1.82 vs h₁'s 0.65. A rounding error δw in w₃₂ (h₂→ŷ) causes output error of δw × 1.82 — 2.8× larger than the same error in w₃₁. Channel h₂ is flagged as salient.
Rank channels — identify the salient ones
Sort weight channels by average activation magnitude. The top 1% are "salient" — their quantization errors will be amplified the most in the output. For our tiny network, channel h₂ is clearly the one to protect.
ChannelWeightavg |activation|Output error riskVerdict
h₂ → ŷ w₃₂ = 0.38 1.82  ★ highest δw₃₂ × 1.82 — amplified Salient — protect
h₁ → ŷ w₃₁ = 0.65 0.65 δw₃₁ × 0.65 — moderate Normal
x₁ → h₁ w₁₁ = 0.43 0.58 moderate Normal
x₂ → h₂ w₂₁ = 0.91 0.52 moderate Normal
x₁ → h₂ w₁₂ = 0.07 0.41 low Normal
x₂ → h₁ w₂₂ = 0.02 0.38 lowest Normal
Why this works so fast: Finding salient channels requires just one calibration forward pass to record activation statistics. No Hessian computation, no iterative optimization. That's why AWQ runs in minutes vs GPTQ's hours — same idea (protect important weights), massively simpler implementation.
Scale up the salient channel before quantizing
Multiply w₃₂ (the salient weight) by a scale factor s > 1 before quantizing. This forces it to occupy more of the INT4 range, giving it finer bucket resolution. After quantizing, divide the output by s to compensate.
Without AWQ — naive INT4: w₃₁ w₃₂ Both share the same bucket grid — w₃₂ gets no special treatment With AWQ — w₃₂ scaled up by s=2.0 before quantizing: w₃₁ w₃₂×s w₃₁ group: fine buckets w₃₂ group (scaled ×2): finer relative buckets Salient channel uses more of the integer range → smaller rounding error → smaller amplified output error
By scaling w₃₂ up by s=2.0 before quantization, it occupies the upper range of the bucket grid — meaning the bucket spacing is effectively halved relative to its value. After quantization, we divide the output by s, so the final computation is unchanged but the rounding error is smaller.
Quantize all weights to INT4 — salient gets better precision
All weights quantize to INT4. The salient weight (w₃₂) was scaled up first, so it lands in a higher bucket with finer effective precision. Its rounding error, when amplified by the activation, is much smaller.
Weightfp32Pre-quant valueINT4ReconstructedTrue errorAmplified error
w₁₁0.430.430 INT4: 70.433 −0.003−0.002 (×0.58)
w₁₂0.070.070 INT4: 10.079 +0.009+0.004 (×0.41)
w₂₁0.910.910 INT4: 150.905 −0.005−0.003 (×0.52)
w₂₂0.020.020 INT4: 00.020 0.0000.000 (×0.38)
w₃₁0.650.650 INT4: 110.640 −0.010−0.007 (×0.65)
w₃₂ ★0.38 0.760 (×2.0) INT4: 13 0.387 (÷2.0) −0.007 −0.013 (×1.82) ↓ better!
The salient weight w₃₂ has a true error of −0.007, similar to other weights. But without AWQ, naive quantization would give it an error of ~−0.025, amplified to −0.046 by the 1.82 activation. AWQ reduces the amplified output error for the salient channel by ~3×.
Post-quantization — AWQ result network
All weights are INT4. The salient edge (h₂→ŷ) is highlighted — it has a larger absolute weight after absorbing the scale, but a smaller effective error relative to its output contribution.
INT4:7 → 0.433 err: −0.003 ✓ INT4:1 → 0.079 err: +0.009 ✓ INT4:15→ 0.905 err: −0.005 ✓ INT4:0 → 0.020 err: 0.000 ✓ INT4:11→ 0.640 err: −0.010 ✓ INT4:13 → 0.387 ★ AWQ protected — err: −0.007 x₁ x₂ h₁ h₂ salient ★ ŷ AWQ result Format: INT4 Salient ch: ★ ✓ Max err: 0.010 Avg err: 0.006 Output ŷ: 0.808 (was 0.81)
The amber edge (h₂→ŷ) is the protected salient channel. Its reconstructed weight (0.387) is very close to the original (0.38). Output: 0.808 vs original 0.810. All in INT4, with just minutes of calibration time — no Hessian required.
Step 1 of 5