GB10 Inference Engine Shootout: llama.cpp -np4 vs vLLM vs NVIDIA's Official NGC Container

Verdict first

On one DGX Spark, over the same chat pipeline (retrieval + rerank + LLM generation over real SSE, not a decode-only synthetic run), we head-to-head'd three ways of serving GPT-OSS-120B^[1][2]:

We kept the self-built image (vllm-node, vLLM 0.15.2rc1, spark-vllm-docker community lineage) over NVIDIA's official NGC container: 2.5x faster single-user TTFT beats NGC's +9% aggregate throughput edge^[2]. TTFT is our highest-frequency user-facing metric, so that's the axis we optimized for.

Method

All three legs ran the same real pipeline, not each engine's own microbenchmark: the same load driver (scripts/load_test_chat.py), the same concurrency levels (c=1/2/4), against the full retrieval+rerank+LLM chat SSE endpoint, authenticated with a real member JWT (an earlier "all requests failed" false alarm turned out to be the guest rate limiter, not engine concurrency — fixed by switching to a minted member token)^[1]. Single-stream raw decode was measured separately (200-token generation ×3) as an engine-only ceiling, excluding retrieval/rerank overhead. All three legs share one docker-compose-models.yml network alias (gpt-oss-120b) — swapping engines is stop one container, start another, zero backend code changes^[2].

The numbers

A→B delta (llama.cpp -np4 → vLLM, real chat pipeline, 18/18 requests succeeded, 0 errors)^[1]:

vLLM was also probed at c=6: 8/8 requests succeeded, TTFT p95 22.4s, aggregate throughput 8.6 tok/s (+56% over c=4); c=8 was already saturated at the same 8.6 tok/s with p95 stretching to 44.7s — that's this configuration's practical concurrency ceiling on this box. Against the earlier single-slot llama.cpp baseline (a flat 2.4–2.7 tok/s regardless of concurrency), c=6 vLLM is a +244% jump^[3].

Self-built vLLM vs NVIDIA's official NGC image, head-to-head (same box, same flags, warm caches)^[2]: single-stream decode was identical on both — 34.3 tok/s. Chat c=1 TTFT p50 was 3.96s on the self-built image vs 9.83s on NGC (confirmed on a warm rerun, not a cache artifact). c=4 p95 was roughly tied (16.5s vs 17.2s). NGC edged ahead on aggregate throughput (6.0 vs 5.5 tok/s, +9%).

Why

vLLM's chat-pipeline win comes down to automatic prefix caching (APC) + chunked prefill: RAG requests share the same system prompt and overlapping retrieved context across calls, so APC reuses already-computed KV state instead of re-running the full prefill every time — that's where the TTFT drop comes from. llama.cpp's multi-slot mode has no equivalent mechanism.

But single-stream decode tied at 34.3 tok/s on both vLLM images tells you the bottleneck isn't "official vs community image" — it's the attention backend. Both images can only use TRITON_ATTN for gpt-oss on this box (the only eligible backend available), which also disproves the "the official image ships better attention" hypothesis. A community-reported 59 tok/s figure comes from a third-party fork with extra quantized layers and a custom attention backend — we're flagging that explicitly as community-sourced, not something we reproduced ourselves^[1].

Trade-offs and recommendation

If your workload is high-frequency, multi-user chat/RAG with a shared system prompt, vLLM with APC + chunked prefill is close to mandatory — the TTFT drop is directly user-visible. If you only care about raw single-stream generation speed (batch jobs, offline summarization), llama.cpp's ~45 tok/s is actually the better and lighter-weight choice. NVIDIA's official NGC image does win on aggregate throughput, but for a single-user-interaction-dominated product, a 2.5x TTFT gap matters more than a 9% throughput gap — which is exactly why we didn't switch to it.

FAQ

Q: Why not just use NVIDIA's official NGC image and skip maintaining a custom one? A: We ran a direct head-to-head: NGC's aggregate throughput is 9% higher, but single-user TTFT is 2.5x slower (9.83s vs 3.96s). TTFT is our highest-frequency user-facing metric, so we kept the self-built image and left the NGC image pulled locally for re-testing on future releases^[2].

Q: Why does vLLM only hit 34 tok/s single-stream on GB10, not the community-cited 59 tok/s? A: Both vLLM images we tested (self-built and official NGC) land on the TRITON_ATTN backend for gpt-oss-120b, and single-stream decode was identical between them — 34.3 tok/s. The 59 tok/s figure is from a community fork with extra quantized layers and a custom attention backend; we haven't reproduced it in our own environment, so we're flagging it as community-sourced, not our own result^[1].

Q: Is this comparison a one-off benchmark run? A: All three legs used the same real chat pipeline, the same load-test script, and the same concurrency levels. The vLLM results were confirmed consistent across two separate checkpoints — the initial migration window and a stability re-check the next morning (0 restarts, 0 errors)^[1].

About HTZL.AI

HTZL.AI is the enterprise private-AI brand of Sichuan Huitu Zhilian Technology Co., Ltd., focused on on-premise deployment where data and models never leave the customer's own infrastructure — and an NVIDIA Inception member company. Every number in this post comes from our own engineering records on a real production DGX Spark.