Multi-GPU Chassis Benchmark: 287ms TTFT and 310/310 Zero Errors on a 122B MoE Model
We ran a full inference benchmark of a 122B-parameter MoE model (10B active, Int4) on our in-house liquid-cooled multi-GPU chassis: 164 tok/s output at the production concurrency point, 287ms median time-to-first-token, zero errors across 310 requests, and five out of five quality tests passed. Full numbers and the concurrency knee analysis below.
On this page
Why run this benchmark
The single most common question in private deployment is: "If I leave the cloud, how much inference performance do I give up?"
That question cannot be answered from a spec sheet. It can only be answered with measurements taken on the same real inference path the product actually runs. So we benchmarked a 122B-parameter MoE model (10B active parameters) at Int4 quantization with a 32K running context on our in-house liquid-cooled multi-GPU chassis, across three dimensions: a concurrency sweep, sustained load, and output quality.
One point about the hardware up front: the GPUs in this chassis are interchangeable. A multi-card PCIe architecture is designed for configurability — the same liquid-cooled enclosure and thermal design accepts accelerators of different generations and memory capacities depending on a customer's budget and workload. What follows is therefore a measured baseline for this architecture in one specific configuration, not a performance claim about any particular card. Change the cards and the baseline changes; the method does not.
Concurrency sweep: locating the production operating point
| Concurrency | Output throughput | Request throughput | TTFT | End-to-end | Regime |
|---|---|---|---|---|---|
| 1 | 35.5 tok/s | 0.31 req/s | 243 ms | 3.13s | Interactive |
| 2 | 48.3 tok/s | 0.42 req/s | 255 ms | 4.98s | Interactive |
| 4 | 65.8 tok/s | 0.58 req/s | 296 ms | 6.60s | Sweet spot |
| 6 | 131 tok/s | 1.20 req/s | 292 ms | 4.10s | Optimal |
| 8 | 164 tok/s | 1.44 req/s | 287 ms | 4.85s | Production |
| 16 | 92.9 tok/s | 0.82 req/s | 7,183 ms | 13.66s | Queuing |
| 32 | 122.8 tok/s | 1.08 req/s | 9,823 ms | 17.48s | Batch only |
The valuable part of this table is not the peak — it is where the knee is.
From c=1 to c=8, output throughput rises 4.6× while time-to-first-token barely moves (243ms → 287ms). Across that range, added concurrency is essentially free. c=8 is this configuration's production operating point: 164 tok/s output, 287ms TTFT, with peak output reaching 248 tok/s.
Past c=8 the behaviour changes character. At c=16, TTFT jumps to 7.2 seconds — 25× the c=8 figure. The cause is not compute starvation but KV cache capacity acting as a hard ceiling: at a full 32K context only about 5 sequences fit concurrently, and everything beyond that queues. This is why we set the concurrency limit before the knee rather than after it: a system with a 287ms first token and a system with a 7.2-second first token are two different products.
Sustained load: 331 seconds without a break
Point samples flatter; sustained load tells the truth. We ran 200 requests (1024-token input, 512-token output) back to back:
- Sustained 0.60 req/s, 149 tok/s output
- Median TTFT 457 ms
- 331 seconds total, success rate 200/200
Across the concurrency sweep and the stress run — 310 requests — the error rate was 0%.
Extrapolating from that sustained figure: 0.60 req/s × (1024 + 512) tokens ≈ 922 tok/s. Run continuously at full load, that is roughly 2.4 billion tokens per month per chassis (input plus output), or about 390 million tokens/month counting output alone. Treat it as a capacity ceiling — real workloads do not run flat out around the clock — but it establishes the order of magnitude a single chassis serves.
Quality tests: fast is not the same as correct
Throughput bought at the cost of output quality is not throughput worth having. All five quality tests passed:
| Test | Result | Detail |
|---|---|---|
| Long context (8K input) | PASS | Correct summarization, no degradation |
| Reasoning chain | PASS | Correct answer, full reasoning trace intact |
| 6-way concurrent quality | PASS | 6/6 math answers correct, 0.54–0.59s latency |
| Long generation (16K) | PASS | 8,023 tokens / 308s, graceful stop |
| Repetition-loop detection | PASS | 1–100 count, no loops, clean finish |
The last one deserves its own note. Repetition looping is the characteristic failure mode of quantized models under long generation, and it is among the easiest things for a benchmark to skip. We treat it as mandatory, because when it happens the user sees a wall of text that never stops — and that damages trust far more than being slow.
Architecture and constraints, stated plainly
| Property | Value |
|---|---|
| Model architecture | MoE, 256 experts, 8+1 active |
| Parameters | 122B total / 10B active |
| Quantization | Int4 (moe_wna16) |
| Native context | 262,144 tokens (256K) |
| Running context | 32,768 tokens (32K) |
| VRAM footprint | 68.5 GB across cards |
| Inference engine | SGLang, tensor parallel TP=4 |
The constraints are equally clear and we list them alongside: KV cache is the hard ceiling, allowing roughly 5 concurrent sequences at a full 32K context; the MoE architecture prevents overlapped prefill/decode scheduling; and on a pure PCIe topology without peer-to-peer interconnect, CUDA Graph runs piecewise. These are deterministic architectural constraints, not tuning problems — and stating them is what lets a customer judge whether their workload fits.
What this chassis is for
We offer two private-deployment paths. They are not alternatives but a ladder:
- Single-node DGX Spark — Grace Blackwell unified memory, whole-system power draw comparable to a high-end workstation, deployable straight into an office rack. Suited to customers where data sovereignty comes first and no machine room exists.
- Liquid-cooled multi-GPU chassis — the system benchmarked here. Multi-card PCIe expansion carries larger models and higher concurrency, with GPU selection configurable. Suited to scaled inference and full-size model deployment.
Both paths run the same software stack — the same RAG engine, the same multi-tenant isolation, the same API gateway. A customer growing from the entry configuration into scaled deployment changes hardware, not platform, and does not redo the integration.
FAQ
Q: Do these numbers hold with different GPUs? A: No, and they shouldn't. The chassis is built for configurability — card model, memory capacity and generation can all change, and the performance baseline moves with them. What this post provides is the method and the knee analysis: sweep concurrency to find the operating point, sustain load to verify stability, run quality tests to catch degradation. We re-baseline with the same method after any configuration change.
Q: Why cap concurrency at 8 rather than higher? A: Because TTFT at c=16 is 25× that at c=8. In a conversational product, time-to-first-token governs perceived responsiveness, and the throughput gained by pushing past the knee costs the user an extra seven seconds of waiting. We set the limit before the knee.
Q: Is 2.4 billion tokens/month measured or extrapolated? A: Extrapolated. It derives from the measured sustained throughput (0.60 req/s at 1024 in / 512 out) run continuously at full load, counting input plus output. Real workloads do not saturate around the clock; the figure is for sizing the order of magnitude a chassis serves, not a throughput commitment.
About HTZL.AI
HTZL.AI is the enterprise private-AI brand of Sichuan Huitu Zhilian Technology Co., Ltd., focused on a deployment path where data never leaves the enterprise and models never enter the cloud. HTZL.AI is an NVIDIA Inception member. All figures in this post come from measurements on our own hardware.