Humming against CUTLASS on the GB10: 10 to 19% faster NVFP4 matrix multiplies at the 27B shapes

24 October 2026 · DGX Spark (GB10, sm_121) · humming 0.1.12 · flashinfer 0.6.18 · torch 2.13 cu130

Our record serving stack for Qwen3.8-27B spends about 60 of its 120 milliseconds per decode step inside NVFP4 matrix multiplies. On a unified-memory box with 273 GB/s, those multiplies are bound by how fast the weights stream from memory, so a kernel that reads closer to the bandwidth ceiling is worth real tokens per second. Humming is an open NVFP4 kernel from the vLLM project. We measured it against the FlashInfer CUTLASS kernel our stack uses today.

Setup

Both kernels run W4A4 NVFP4: FP4 e2m1 weights and activations, group size 16, FP8 scales. The shapes are the dense projections of the 27B model: hidden size 5120, intermediate 17408. M is the number of tokens per verify pass: 13 for one user with 12 draft tokens, 26 for two users, 65 for one user with 64 drafts.

The first run kept the weights warm in L2 and overstated Humming on the small shapes by a wide margin, because a real decode step reads every layer's weights from memory once. The numbers below come from a cache-cold run: eight rotating copies of each weight matrix, so every multiply streams from DRAM the way the server does.

Result, cache cold

Effective weight bandwidth: CUTLASS reaches 181 to 200 GB/s on the big shapes, Humming 209 to 224 GB/s, against a 273 GB/s peak. CUTLASS also has a floor of about 56 µs on small matrices regardless of size; Humming does the same multiply in 17 µs. Every small projection in the model pays that floor today.

The accuracy number that was wrong

Against a BF16 reference on random Gaussian data, the first run gave a relative error of 0.157 for Humming and 0.134 for CUTLASS. That looked like a kernel problem. It was a test problem. With weights at a standard deviation of 0.02 and no per-tensor scale, the FP8 group scales fall into the subnormal range of e4m3. Real NVFP4 checkpoints carry a per-tensor scale. With that scale enabled, Humming's error is 0.136 against 0.135 for CUTLASS, and the tensor scale costs 1 to 2% of speed, leaving Humming 9 to 16% ahead on the big shapes.

What it is worth

At 10 to 19% on the big shapes, plus the small-matrix floor removed, the saving is about 7 to 12 milliseconds per step on our record configuration. On the public ranking cell that projects from 62.5 to roughly 66 to 69 tokens per second, before any drafter gains. The projection is a projection: the next step is a server-level A/B on the real checkpoint with the quality gates we use for every change, greedy agreement first, then GSM8K and MATH-500.

Reproduce

CUDA_VISIBLE_DEVICES=0 python -I humming_l2cold.py
# shapes (5120,5120) (5120,17408) (17408,5120) (5120,1024); M = 13, 26, 65
# 8 rotating weight copies; LayerConfig: a and b float4e2m1, scales float8e4m3, group 16
# accuracy: humming_accuracy.py, weight_scale_2_type tensor

Scripts and raw output are in our engineering repository and shared on request.