TEI Has No arm64 Image: Running bge-reranker-v2-m3 on DGX Spark with llama.cpp

The problem: TEI cannot boot on GB10

Our original reranking plan was HuggingFace's Text Embeddings Inference (TEI) — the most common cross-encoder serving framework in the RAG ecosystem, with a simple API and plenty of community docs. But DGX Spark's CPU is ARM64 (GB10 is a Grace CPU + Blackwell GPU unified-memory architecture), and we verified that TEI ships no linux/arm64 image across the entire 1.5–1.8 and cpu tag series — checked tag by tag with docker manifest inspect, and every result came back amd64-only^[1]. This wasn't a configuration hassle to work around; it was a dead end on this hardware, full stop.

The fix: reuse the image we already run

Instead of standing up a second serving stack, we started a second instance of the same local/llama.cpp:server-cuda image already serving our LLM, using its built-in --reranking mode to serve the cross-encoder^[2]. Concretely:

1. Downloaded a Q8_0 GGUF quantization of BAAI/bge-reranker-v2-m3 (~600MB, one-time), from gpustack's GGUF conversion repo;
2. Added a reranker service to compose, reusing llama.cpp's base config with a --reranking flag added to the command;
3. The critical detail: llama.cpp's /rerank endpoint returns the exact same response shape as TEI ([{"index", "score"}, ...], sorted by score descending) — which meant the parser we'd already written for TEI (rag/reranker.py) needed zero code changes to consume llama.cpp's output. We confirmed this live with a probe script before trusting it^[2].

That contract compatibility is what let this be a near-zero-code-change swap — if the response shapes had differed, we'd have had to change infrastructure and business logic at the same time, which would have been a materially riskier change.

Tuning: sizing the batch for real RAG chunk lengths

With the serving approach settled, the batch parameters needed sizing for the real workload. llama.cpp's default physical batch size (ubatch) is 512, and reranking is non-causal: each query+chunk pair must fit entirely inside one physical batch, and 512 cannot hold a 1024-token chunk plus the query. Load testing with production-length RAG chunks settled the configuration: --ctx-size to 8192, --batch-size/--ubatch-size to 4096 (covering a 1024 chunk_size plus query with headroom), and slots reduced to -np 2 to bound VRAM^[3].

This is a step worth taking with any cross-encoder deployment: reranking has batch-capacity constraints quite unlike generative inference, and inheriting the LLM's default batch parameters will fail outright on real chunk lengths. Load-testing with production chunk sizes before rollout is far more reliable than tuning after the fact.

Results: two-stage retrieval measured on the production path

In production the chat path now recalls 24 candidates and hands them to the cross-encoder, which narrows to the 8 that become answer context. Production logs confirm it: "Chat-path rerank applied, method: cross-encoder, candidates: 24, returned: 8"^[1]. Sampling the REST retrieval endpoint shows rerank_score with a real, discriminating distribution (for example 5.47 / 5.28 / 4.72 within one candidate batch) — the precision stage is genuinely reordering candidates rather than going through the motions^[1].

FAQ

Q: Is llama.cpp the only ARM64 alternative to TEI? A: It's the one path we verified and adopted — its advantage is reusing infrastructure we already run for the LLM, rather than standing up a new stack. We haven't done an exhaustive survey of other ARM64-native reranker serving frameworks; if you're evaluating alternatives, checking arm64 support tag-by-tag with docker manifest inspect is the first step — don't assume a popular framework supports ARM64 by default.

Q: Why not use vector retrieval alone? A: Vector recall is strong at coarse semantic filtering but has limited resolving power when ordering candidates against each other. With the cross-encoder in place we observe a real, discriminating score distribution, and the precision stage measurably changes which documents reach answer context — for cited enterprise Q&A, which 8 documents get in directly determines how accurate the answer is.

Q: Does raising --ubatch-size from 512 to 4096 slow down reranking itself or use more memory? A: It costs more per-batch memory, which is exactly why we dropped the concurrent slot count to -np 2 — trading batch capacity for the ability to handle real chunk lengths, while controlling total memory pressure. We didn't run a dedicated before/after latency A/B on the batch-size change, so we're not quoting a specific latency number here — only describing the configuration trade-off itself.

About HTZL.AI

HTZL.AI is the enterprise private-AI brand of Sichuan Huitu Zhilian Technology Co., Ltd., focused on on-premise deployment where data and models never leave the customer's own infrastructure — and an NVIDIA Inception member company. Every number in this post comes from our own engineering records on a real production DGX Spark.