A Production RAG Upgrade in 48 Hours: Cross-Encoder Reranking, vLLM Migration, v1.0.0

Goal: wire cross-encoder reranking into the production chat path

The first objective was moving chat-path retrieval from single-stage vector recall to two-stage precision ranking: a rerank gate, a candidate pool widened from 8 to 24 and narrowed back to 8, matching REST-endpoint widening, and an SSE load-test driver to go with it^[1].

The first external constraint came from the hardware architecture: HuggingFace TEI ships no linux/arm64 image and cannot start on DGX Spark (GB10, ARM64). We served the cross-encoder from the llama.cpp image already running our LLM, via its --reranking mode — one fewer component to deploy and one fewer surface to operate (details in a separate post)^[1].

Quantifying the engine bottleneck

With reranking wired in, we measured the single-slot llama.cpp concurrency baseline: c=1 TTFT p50 12.3s, c=2 33.8s, c=4 51.2s, aggregate throughput flat at 2.4–2.7 tok/s — engine serialization was the bottleneck itself. The first step was switching to 4 slots (-np 4): c=4 TTFT dropped to roughly 21s, aggregate throughput rose from 2.7 to 4.6 tok/s (+70%), with no regression on single-stream decode^[1].

Closed-loop load testing as the release gate

We do not let code review decide whether a change ships. The gate is closed-loop load testing with real auth and real concurrency, and this pass located and resolved five integration issues before any of them reached users^[1]: MCP subprocess environment stripped down to just HOME/PATH, silently dropping RERANKER_HOST (fixed by passing the full environment through); MCP logs written to stdout polluting the JSONRPC transport channel with 392 parse errors (fixed by routing to stderr); a cross-context ContextVar reset throwing on client disconnect (fixed with a guard); an initial "everything failed" baseline that turned out to be the guest rate limiter blocking unauthenticated load-test traffic (fixed by switching to a real member JWT); and the reranker returning 500s on ~1K-token chunks because the default batch size (512) couldn't fit them (fixed by raising ubatch to 4096). One more surfaced in parallel: newer llama.cpp had removed the --cache-prompt flag, and continuing to pass it caused a crash-restart loop; it was removed^[1]. This batch landed alongside the core feature work, and the full suite — 2,743 tests — passed green.

The rollback plan, measured under real conditions

-np 4 was an interim step; phase two was migrating to vLLM for automatic prefix caching and chunked prefill.

The first maintenance window (2026-07-25, 01:31–01:35) executed a controlled rollback in 4.5 minutes: vLLM startup returned a SafetensorError — 4 of 15 weight shards were remnants of an earlier interrupted download. The positive signal was that every planned flag (mxfp4, Marlin backend, fp8 KV, prefix caching, chunked prefill) was accepted without complaint. Rollback was verified complete at 01:35:57 with llama.cpp serving public traffic at 200 throughout: zero interruption to the primary path^[2].

That window produced a permanent addition to our pre-flight checklist: verify shard header integrity and cross-check size against the offset table, not merely that the files exist. We requested the second window only after that check was in place — being able to diagnose and exit safely inside 4.5 minutes is exactly what the plan is designed for.

Window #2: migration complete, metrics improved across the board

The second window (02:39–02:58) had a cold start of only ~10 minutes (budgeted 30–40). B-side results on the real chat pipeline^[2]:

Concurrency	TTFT p50 (A→B)	Aggregate throughput (A→B)
c=1	14.0s → 3.96s (−72%)	3.1 → 3.7 tok/s
c=2	28.0s → 11.27s (−60%)	2.5 → 5.0 tok/s
c=4	21.0s → 14.74s (−30%)	4.6 → 5.5 tok/s

18/18 requests succeeded, 0 errors. Two items went onto the optimization roadmap: single-stream decode at 34.3 tok/s (the community-cited 59 depends on a FlashInfer backend not included in this build); and a faithfulness sub-score that, at a sample size of one and skewed by query caching, was not statistically meaningful and was re-baselined after the golden set was expanded. The call was to keep vLLM — every user-facing metric improved.

NGC head-to-head + stability re-check

At 03:59–04:35 we pulled NVIDIA's official NGC image for a direct comparison: single-stream decode was identical on both (34.3 tok/s, both capped by TRITON_ATTN); c=1 TTFT p50 was 3.96s on our image vs 9.83s on NGC; c=4 was roughly tied; NGC edged ahead 9% on aggregate throughput — we kept our own image, weighing a 2.5x TTFT advantage as worth more than 9% aggregate^[2]. A 05:47 re-check confirmed 0 restarts, 0 errors, memory stable at 89–91GB; a c=6/8 probe showed c=6 aggregate throughput at 8.6 tok/s (+56%), c=8 already saturated — concurrency guards were raised from 4 to 6 (global) / 4 (per-tenant)^[1]^[3].

Guardrails and the numbers

The blocking CI gate's golden question set was established at 17 questions, covering source recall, keyword recall, anti-fabrication negatives, citation correctness, and faithfulness — mined from real traffic and knowledge-base coverage; a nightly 03:10 cron runs the same harness against production^[3]. Across these 48 hours we located and resolved 13 integration issues across the stack; at close, the backend suite stood at 2,810 tests and the frontend at 842, all green^[3].

Engineering method

What carried these 48 hours was not overtime but three standing disciplines: closed-loop load testing as the release gate (real auth, real concurrency — not a code-review judgement call), controlled maintenance windows (every change carries explicit rollback criteria and a time budget), and a golden question set as a blocking CI gate (quality regressions are caught by machine, not by spot checks). The 4.5-minute safe exit in window #1 is that discipline measured under real conditions.

FAQ

Q: Was live service affected during the upgrade? A: Zero interruption to the primary path. Window #1 exercised the rollback plan with llama.cpp serving at 200 throughout; window #2 cold-started in about 10 minutes with 18/18 requests succeeding and 0 errors.

Q: Why a self-built vLLM image rather than the official NGC container? A: Head-to-head: c=1 TTFT p50 of 3.96s self-built vs 9.83s for NGC, with NGC ahead only on aggregate throughput, by 9%. In a conversational product, single-user time-to-first-token matters far more than aggregate throughput.

Q: What does the golden question set actually test? A: 17 questions spanning source recall, keyword recall, anti-fabrication negatives, citation correctness, and faithfulness, sourced from real traffic — both a CI gate and a nightly production health check.

About HTZL.AI

HTZL.AI is the enterprise private-AI brand of Sichuan Huitu Zhilian Technology Co., Ltd., focused on on-premise deployment where data and models never leave the customer's own infrastructure — and an NVIDIA Inception member company. Every number in this post comes from our own engineering records on a real production DGX Spark.