Paper, researchers, and primary source
Inference, evaluation & serving / inference_systems
Sarathi-Serve uses chunked prefills and iteration-level scheduling to reduce interference between long prompt processing and ongoing token generation.
Contribution 1
Split large prefills into chunks that can share decode iterations.
Contribution 2
Implemented stall-free batching to control prefill-induced generation pauses.
Contribution 3
Evaluated throughput and latency tradeoffs across online serving workloads.
Research context
inference_systems / 2024
Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve places sarathi serve inside the broader inference systems discussion at Microsoft Research, with chunked prefill supplying a second analytical lens. Its contribution chain has three links: Split large prefills into chunks that can share decode iterations; Implemented stall-free batching to control prefill-induced generation pauses; and Evaluated throughput and latency tradeoffs across online serving workloads. This framing makes stall free batching a property to inspect within the study, not a label that settles later deployments. Its builder-facing implication is that chunk size and scheduling policy are workload-specific; extra preemption or batching can affect tail latency and fairness.
Methods and evidence reading
1 cataloged method notes
Method 1
The experimental design in Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve tests split large prefills into chunks that can share decode iterations and implemented stall-free batching to control prefill-induced generation pauses against the paper's documented baselines, datasets, model variants, or systems workloads.
How to read the evidence
A careful reading of Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve starts with the experiment's declared scope, not the reputation of Microsoft Research. The editorial method record pairs two moves: Split large prefills into chunks that can share decode iterations; and Implemented stall-free batching to control prefill-induced generation pauses. The outcome-facing contribution is: Evaluated throughput and latency tradeoffs across online serving workloads. This supports the bounded implication that chunk size and scheduling policy are workload-specific; extra preemption or batching can affect tail latency and fairness. It does not remove the source limit that the empirical reach of Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve stops at accelerator hardware, reported models, comparison baselines, numerical precision, software revisions, service-level objectives, and request shapes; broader chunked prefill use therefore requires fresh measurements. Follow-on evaluation should therefore vary chunked prefill while retaining an explicit sarathi serve baseline. A reproduction ledger for Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve should preserve sarathi serve, vary chunked prefill, and retain a counterexample tied to Implemented stall-free batching to control prefill-induced generation pauses before judging transfer.
Findings in the source record
1 paper-specific findings
- The reported evidence in Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve supports evaluated throughput and latency tradeoffs across online serving workloads.
Practical implication for AI builders
Microsoft Research / 2024
Proposed BrokenGPT application
Research blueprint / proposed status
Proposed route placement / sarathi serve
For a proposed BrokenGPT experiment based on Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve, tune chunking only against representative short and long requests, then gate on time to first token, inter-token jitter, p99 latency, throughput, and starvation. Keep the sarathi serve path isolated, versioned, and attributable to this research record.
Proposed acceptance test / chunked prefill
Validate the proposed sarathi serve route against the paper's reported outcome: Evaluated throughput and latency tradeoffs across online serving workloads. The proposed Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve test should capture inter-token delay, memory headroom, goodput, and time to first token, with chunked prefill error slices reported apart from the headline sarathi serve result.
Proposed decision boundary / stall free batching
Balance latency, throughput, and operational complexity before promoting the proposed stall free batching design. Because the paper leaves networking overhead, failure recovery, safety checks, authentication, tokenization, and workload drift as open implementation variables rather than consequences established by its experiments, adoption remains conditional on replay under BrokenGPT's selected model, runtime, and policy configuration.
Limitations, verification, and source
Boundaries recorded with the paper
Limitations
- The empirical reach of Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve stops at accelerator hardware, reported models, comparison baselines, numerical precision, software revisions, service-level objectives, and request shapes; broader chunked prefill use therefore requires fresh measurements.
- The paper leaves networking overhead, failure recovery, safety checks, authentication, tokenization, and workload drift as open implementation variables rather than consequences established by its experiments.
PRIMARY SOURCES
- 01Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
Microsoft Research — Primary primary arXiv paper / 4 March 2024 / Amey Agrawal, Nitin Kedia, Ashish Panwar, and 5 more