Paper, researchers, and primary source
Inference, evaluation & serving / inference_systems
FlexGen schedules model weights, activations, and key-value cache across GPU, CPU, and storage to run large language models on a single memory-limited accelerator.
Contribution 1
Formulated offloading as a search over tensor placement and movement schedules.
Contribution 2
Used compression and overlapped I/O to relieve accelerator-memory limits.
Contribution 3
Demonstrated high-throughput offline generation for models too large for one GPU.
Research context
inference_systems / 2023
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU places flexgen inside the broader inference systems discussion at Stanford University, with offloading supplying a second analytical lens. The editorial sequence connects three claims: Formulated offloading as a search over tensor placement and movement schedules; Used compression and overlapped I/O to relieve accelerator-memory limits; and Demonstrated high-throughput offline generation for models too large for one GPU. The combination matters because single gpu only has meaning under the paper's stated setup. Operationally, the record points to one consequence: flexGen targets throughput-oriented batch inference; storage bandwidth and offload latency can make it unsuitable for interactive token streaming.
Methods and evidence reading
1 cataloged method notes
Method 1
The experimental design in FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU tests formulated offloading as a search over tensor placement and movement schedules and used compression and overlapped i/o to relieve accelerator-memory limits against the paper's documented baselines, datasets, model variants, or systems workloads.
How to read the evidence
For FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU, the useful question is what the documented setup supports and where transfer remains untested. Its evidence chain begins with two documented moves: Formulated offloading as a search over tensor placement and movement schedules; and Used compression and overlapped I/O to relieve accelerator-memory limits. Reported evidence then addresses: Demonstrated high-throughput offline generation for models too large for one GPU. The resulting interpretation is practical but conditional: flexGen targets throughput-oriented batch inference; storage bandwidth and offload latency can make it unsuitable for interactive token streaming. Its boundary is that the empirical reach of FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU stops at numerical precision, comparison baselines, reported models, software revisions, request shapes, accelerator hardware, and service-level objectives; broader offloading use therefore requires fresh measurements. Any extension should report how altered offloading conditions affect the original flexgen result. Testing FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU beyond its source setting requires a stable flexgen control, explicit offloading slices, and documented exceptions to Used compression and overlapped I/O to relieve accelerator-memory limits.
Findings in the source record
1 paper-specific findings
- The reported evidence in FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU supports demonstrated high-throughput offline generation for models too large for one gpu.
Practical implication for AI builders
Stanford University / 2023
Proposed BrokenGPT application
Research blueprint / proposed status
Proposed route placement / flexgen
For a proposed BrokenGPT experiment based on FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU, keep FlexGen as an offline batch candidate and compare completed jobs per dollar, wall time, quality parity, device wear, and queue delay. Keep the flexgen path isolated, versioned, and attributable to this research record.
Proposed acceptance test / offloading
Validate the proposed flexgen route against the paper's reported outcome: Demonstrated high-throughput offline generation for models too large for one GPU. Measure goodput, time to first token, memory headroom, and inter-token delay for the FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU candidate, then isolate offloading regressions before judging the proposed flexgen route.
Proposed decision boundary / single gpu
Balance latency, throughput, and operational complexity before promoting the proposed single gpu design. Because operational use of flexgen introduces workload drift, failure recovery, tokenization, authentication, networking overhead, and safety checks, so a matched replay is necessary before a release decision, adoption remains conditional on replay under BrokenGPT's selected model, runtime, and policy configuration.
Limitations, verification, and source
Boundaries recorded with the paper
Limitations
- The empirical reach of FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU stops at numerical precision, comparison baselines, reported models, software revisions, request shapes, accelerator hardware, and service-level objectives; broader offloading use therefore requires fresh measurements.
- Operational use of flexgen introduces workload drift, failure recovery, tokenization, authentication, networking overhead, and safety checks, so a matched replay is necessary before a release decision.
PRIMARY SOURCES
- 01FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
Stanford University — Primary primary arXiv paper / 13 March 2023 / Ying Sheng, Lianmin Zheng, Binhang Yuan, and 11 more