62.5 tokens per second for a 27B model on a DGX Spark: how the ranking cell was measured
The DGX Spark is a 273 GB/s unified-memory box. Decoding a 27B model on it is a bandwidth problem, and the public leaderboard for the machine, Spark Arena, ranks entries on one cell: 128 generated tokens at a context depth of 16,384 with two concurrent users, measured by llama-benchy, decode only. The posted record for Qwen3.8-27B on a single Spark was 60.57 tokens per second. This note says what we ran, what we measured, and how sure we are.
What the cell measures
llama-benchy reports decode tokens divided by the window from the first user's first token to the last user's last token. Prefill sits outside that window when chunked prefill is off, which is why prefix-cache tricks at 16k do not move this cell. With two users the measurement is pure batch-2 decode: tokens accepted per step, times two, over the step time. That leaves three levers: how many tokens a step accepts, how fast the weights stream, and how little time is lost outside the matrix multiplies.
The build
- Target: an all-NVFP4 build of Qwen3.8-27B. The public NVFP4 checkpoint keeps 208 gated-DeltaNet and attention projections in FP8; we re-quantized them to NVFP4 in the ModelOpt layout with max calibration, a shared global scale per fused group, and a static activation scale. 18 GB instead of 22 GB.
- Drafter: DFlash2, a block drafter, at 12 draft tokens, in FP8. Measured acceptance on this cell is about 4 tokens per step.
- Engine: SGLang 0.5.22 with the linear-attention state fold for speculative verification enabled on this architecture, Triton backends for the gated-DeltaNet decode and verify paths, chunked prefill off, overlap scheduling on.
- Chat default: reasoning effort low. The template default is extra high; the published recipe we compared against uses medium.
Runs
| Configuration | Runs | tok/s |
|---|---|---|
| Published recipe for the posted record, replayed on our box the same day | 10 | 54.5 |
| Our build, default reasoning effort, pooled | 80 | 60.4 |
| Our build, reasoning effort low, 12 drafts | 10 | 63.6 |
| Same, longer run | 30 | 62.1 |
| Same, all runs | 40 | 62.5 |
| Posted record | 60.57 |
The standard error of the 40-run mean is about 0.9 tokens per second, so 62.5 sits about two standard errors above the posted figure. With the default reasoning effort our build ties the record: 80 pooled runs give 60.4. The published recipe, run on our own box on the same day with the same tool, gives 54.5, so like for like the gain is 11%. We report all three because a single ten-run number on this cell moves by several tokens per second between sessions.
Quality gates
Speed that costs answers is not speed. On GSM8K-100 the all-NVFP4 build scores 89 against 87 for the plain NVFP4 checkpoint. On 60 GSM8K items with thinking on, reasoning effort low scores 98.3% against 96.7% at extra high. On MATH-500 levels 4 and 5, 45 problems, both settings score 84.4%, so the lower effort is quality-neutral on the hardest set we run. Every configuration change in this program passes a greedy-agreement check against the BF16 model before it is timed.
What did not work
Most things. Thirty-two draft tokens, torch.compile, an FP32 head, n-gram drafting, a calibrated NVFP4 drafter, every FP4 GEMM backend selectable at runtime, prefix caching at 16k, deterministic mode, a fused gated-DeltaNet verify kernel that passed its unit test and changed outputs in the server. Each was measured on the same cell over ten runs and closed. The list matters more than the record: it is what a Spark owner is paying us not to repeat.
What an owner gets
The same machine, as it arrives, running Ollama, gives 27.9 tokens per second on a single stream. Under our serving profile it gives 67.7 on code and 126 on file edits, measured the same afternoon. That before-and-after table, with the profile, the scripts and the rerun command, is the deliverable.