Paper, researchers, and primary source
Major lab research / alignment_and_safety
This work trains summarizers from human comparisons, using a reward model to optimize outputs toward judgments of summary quality.
Contribution 1
Applied preference modeling and reinforcement learning to abstractive summarization.
Contribution 2
Compared human-feedback optimization with supervised and likelihood-based baselines.
Contribution 3
Studied transfer of learned preferences to a different news domain.
Research context
alignment_and_safety / 2020
Learning to summarize from human feedback places summarization inside the broader alignment and safety discussion at OpenAI, with human feedback supplying a second analytical lens. The paper's through-line contains three reported moves: Applied preference modeling and reinforcement learning to abstractive summarization; Compared human-feedback optimization with supervised and likelihood-based baselines; and Studied transfer of learned preferences to a different news domain. That sequence keeps reward model tied to the reported work instead of treating it as a free-standing promise. The practical stake is equally bounded: directly optimizing human judgments can improve subjective output quality, but the reward model and annotator population define what is rewarded.
Methods and evidence reading
1 cataloged method notes
Method 1
The experimental design in Learning to summarize from human feedback tests applied preference modeling and reinforcement learning to abstractive summarization and compared human-feedback optimization with supervised and likelihood-based baselines against the paper's documented baselines, datasets, model variants, or systems workloads.
How to read the evidence
Evidence for Learning to summarize from human feedback is best read as a scoped argument rather than a universal verdict. The source record contains two linked moves: Applied preference modeling and reinforcement learning to abstractive summarization; and Compared human-feedback optimization with supervised and likelihood-based baselines. Its reported outcome is: Studied transfer of learned preferences to a different news domain. The defensible takeaway remains directly optimizing human judgments can improve subjective output quality, but the reward model and annotator population define what is rewarded. That conclusion must travel with the recorded boundary that for Learning to summarize from human feedback, the supported boundary runs through rater instructions, model revisions, prompt sampling, evaluator models, selected threat model, and construct validity; extrapolation past it needs an independently matched baseline. A replication should preserve the disclosed setup and test whether summarization still holds when human feedback conditions change. To retest Learning to summarize from human feedback, hold the summarization baseline visible while changing human feedback, then log where Compared human-feedback optimization with supervised and likelihood-based baselines no longer predicts the reported outcome.
Findings in the source record
1 paper-specific findings
- The reported evidence in Learning to summarize from human feedback supports studied transfer of learned preferences to a different news domain.
Practical implication for AI builders
OpenAI / 2020
Proposed BrokenGPT application
Research blueprint / proposed status
Proposed route placement / summarization
For a proposed BrokenGPT experiment based on Learning to summarize from human feedback, create a summarization evaluator from explicit accuracy, coverage, and brevity rubrics, while retaining source-linked human audits to detect reward gaming. Keep the summarization path isolated, versioned, and attributable to this research record.
Proposed acceptance test / human feedback
Validate the proposed summarization route against the paper's reported outcome: Studied transfer of learned preferences to a different news domain. Use refusal precision, adversarial coverage, and helpful-answer retention to evaluate Learning to summarize from human feedback, but retain a distinct human feedback ledger so the proposed summarization path cannot hide concentrated failures.
Proposed decision boundary / reward model
Balance usefulness, oversight burden, and residual risk before promoting the proposed reward model design. Because even if the reported result reproduces, deployment drift, adversarial adaptation, judge bias, unsampled behaviors, and language coverage can reverse its product value and must be measured separately, adoption remains conditional on replay under BrokenGPT's selected model, runtime, and policy configuration.
Limitations, verification, and source
Boundaries recorded with the paper
Limitations
- For Learning to summarize from human feedback, the supported boundary runs through rater instructions, model revisions, prompt sampling, evaluator models, selected threat model, and construct validity; extrapolation past it needs an independently matched baseline.
- Even if the reported result reproduces, deployment drift, adversarial adaptation, judge bias, unsampled behaviors, and language coverage can reverse its product value and must be measured separately.
PRIMARY SOURCES
- 01Learning to summarize from human feedback
OpenAI — Primary primary arXiv paper / 2 September 2020 / Nisan Stiennon, Long Ouyang, Jeff Wu, and 6 more