Paper, researchers, and primary source
Inference, evaluation & serving / inference_systems
Splitwise studies phase splitting for generative inference, placing compute-heavy prompt processing and memory-heavy token generation on hardware suited to each phase.
Contribution 1
Characterized distinct prefill and decode resource behavior.
Contribution 2
Designed clusters that split inference phases across heterogeneous machines.
Contribution 3
Evaluated cost, power, throughput, and latency tradeoffs under serving workloads.
Research context
inference_systems / 2023
Splitwise: Efficient generative LLM inference using phase splitting places splitwise inside the broader inference systems discussion at Microsoft Research / University of Washington, with phase splitting supplying a second analytical lens. Read together, the source records three advances: Characterized distinct prefill and decode resource behavior; Designed clusters that split inference phases across heterogeneous machines; and Evaluated cost, power, throughput, and latency tradeoffs under serving workloads. Keeping those moves together prevents heterogeneous hardware from being detached from its evidence. For an implementation review, the relevant consequence is that hardware specialization can improve efficiency but adds transfer latency, capacity-planning complexity, and sensitivity to workload mix.
Methods and evidence reading
1 cataloged method notes
Method 1
The experimental design in Splitwise: Efficient generative LLM inference using phase splitting tests characterized distinct prefill and decode resource behavior and designed clusters that split inference phases across heterogeneous machines against the paper's documented baselines, datasets, model variants, or systems workloads.
How to read the evidence
The evidentiary value of Splitwise: Efficient generative LLM inference using phase splitting comes from the relationship among its reported moves. Two entries define the method-level claim: Characterized distinct prefill and decode resource behavior; and Designed clusters that split inference phases across heterogeneous machines. The cataloged result is: Evaluated cost, power, throughput, and latency tradeoffs under serving workloads. On that basis, hardware specialization can improve efficiency but adds transfer latency, capacity-planning complexity, and sensitivity to workload mix. The catalog nevertheless records that evidence for splitwise in Splitwise: Efficient generative LLM inference using phase splitting covers reported models, software revisions, request shapes, numerical precision, comparison baselines, accelerator hardware, and service-level objectives; behavior beyond that documented envelope remains untested. Reproduction work should separate genuine splitwise transfer from behavior caused by a changed phase splitting setup. To retest Splitwise: Efficient generative LLM inference using phase splitting, hold the splitwise baseline visible while changing phase splitting, then log where Designed clusters that split inference phases across heterogeneous machines no longer predicts the reported outcome.
Findings in the source record
1 paper-specific findings
- The reported evidence in Splitwise: Efficient generative LLM inference using phase splitting supports evaluated cost, power, throughput, and latency tradeoffs under serving workloads.
Practical implication for AI builders
Microsoft Research / University of Washington / 2023
Proposed BrokenGPT application
Research blueprint / proposed status
Proposed route placement / splitwise
For a proposed BrokenGPT experiment based on Splitwise: Efficient generative LLM inference using phase splitting, compare split and unified clusters with the same model and trace, recording quality, both latency phases, energy, utilization, transfer overhead, and failover. Keep the splitwise path isolated, versioned, and attributable to this research record.
Proposed acceptance test / phase splitting
Validate the proposed splitwise route against the paper's reported outcome: Evaluated cost, power, throughput, and latency tradeoffs under serving workloads. Assess the proposed Splitwise: Efficient generative LLM inference using phase splitting route through time to first token, inter-token delay, memory headroom, and goodput, and treat phase splitting failures as their own splitwise decision input.
Proposed decision boundary / heterogeneous hardware
Balance latency, throughput, and operational complexity before promoting the proposed heterogeneous hardware design. Because even if the reported result reproduces, authentication, safety checks, failure recovery, workload drift, networking overhead, and tokenization can reverse its product value and must be measured separately, adoption remains conditional on replay under BrokenGPT's selected model, runtime, and policy configuration.
Limitations, verification, and source
Boundaries recorded with the paper
Limitations
- Evidence for splitwise in Splitwise: Efficient generative LLM inference using phase splitting covers reported models, software revisions, request shapes, numerical precision, comparison baselines, accelerator hardware, and service-level objectives; behavior beyond that documented envelope remains untested.
- Even if the reported result reproduces, authentication, safety checks, failure recovery, workload drift, networking overhead, and tokenization can reverse its product value and must be measured separately.
PRIMARY SOURCES
- 01Splitwise: Efficient generative LLM inference using phase splitting
Microsoft Research / University of Washington — Primary primary arXiv paper / 30 November 2023 / Pratyush Patel, Esha Choukse, Chaojie Zhang, and 4 more