Moore Threads Unveils Prefill Decoupling Solution: Single-Server Throughput Nears 6M Tokens at 64K Context — BigGo Finance

As large-model input context lengths rapidly scale into the hundreds of thousands to 1 million token range, inference costs and time-to-first-token (TTFT) latency in long-text scenarios continue to climb, becoming a critical bottleneck constraining enterprise AI service efficiency and total cost of ownership (TCO). On August 24, Moore Threads officially released its “MTT S5000 Prefill-as-a-Service Technical White Paper,” introducing a “Prefill As a Service” paradigm built on its flagship MTT S5000 AI training-inference integrated accelerator card. The solution targets long-context inference scenarios such as AI agents, code generation, and ultra-long document analysis, offering a practical path that balances inference efficiency, cost control, and compute monetization.

The white paper identifies the core pain point behind today’s high inference costs as a structural mismatch in compute resource allocation. In online large-model inference, the Prefill phase (input processing) is compute-intensive and constrained by floating-point throughput, while the Decode phase (output generation) is memory-bandwidth-intensive and constrained by VRAM bandwidth. Running both in the same physical resource pool inevitably leads to bidirectional waste—if hardware is selected based on Decode bandwidth requirements, the large-capacity VRAM needed for Prefill remains chronically underutilized; if selected based on Prefill compute requirements, Decode compute units sit largely idle.

Prefill-Decode Heterogeneous Decoupling

To address this bottleneck, the white paper’s core proposal is to fully decouple Prefill from Decode, allowing each to run on hardware resource pools optimally matched to its own computational characteristics. Through tiered hardware investment, compute configuration shifts from “universal redundancy” to “demand-matched allocation,” achieving substantial reductions in per-token infrastructure costs while satisfying service-level objective (SLO) constraints.

As the compute foundation for this approach, the MTT S5000 is highly aligned with Prefill task requirements at the hardware level. The accelerator delivers high dense compute throughput to compress TTFT, natively supports full-precision FP8 computation, and is fully compatible with CUDA as well as mainstream inference frameworks including PyTorch and vLLM, demonstrating strong engineering adaptability and scalability for production deployment.

Benchmark Results Under Stringent SLO Constraints

The white paper conducted extreme stress testing across 64K to 400K sequence lengths using Zhipu’s ultra-large-parameter GLM-5.2-FP8 model, under stringent hard SLO constraints of 90% prefix cache hit rate and P50 TTFT not exceeding 6,000 milliseconds. Measured results show that at 64K context, single-server Prefill throughput reached 95,920 tokens per second, with overall single-server throughput hitting 5.8 MTokens TPM. At the extreme 400K long-sequence scenario, TPM still reached 2.6 MTokens, demonstrating stable and controllable throughput performance.

Using Zhipu’s official API pricing as the baseline (8 yuan per million input tokens, 2 yuan per million cache-hit tokens, yielding a blended rate of 2.6 yuan per million tokens), a single server at full load under ideal conditions could generate monthly revenue of approximately 646,000 yuan (approximately $96,000). Under typical agent-driven mixed-context workloads, a single server offers more deterministic compute monetization capability and a clear investment payback cycle.

Moore Threads stated that the second half of the large-model industry competition is fundamentally a contest of inference efficiency versus commercial returns. Through the MTT S5000 and its Prefill-as-a-Service solution, the company aims not only to redefine the performance and cost boundaries of long-context inference, but also to help enterprises seize TCO initiative in the era of trillion-parameter models and AI agents. The complete white paper is available on Moore Threads’ official website.