01
A large context window is capacity, not guaranteed recall.
The BrokenGPT public contract defaults to a 1,048,576-token context cap, with the live configured value published on the system page. Input, tool definitions, response schemas, and any explicit output allowance must fit together.
More context can reduce truncation, but it can also increase latency, cost, distraction, and the chance that relevant evidence is buried. Measure answer quality at the document positions and lengths your product will use.
02
Reserve context for instructions and the answer.
- Count or estimate the serialized messages, tools, and schemas.
- Choose an output allowance that matches the task rather than the maximum remaining capacity.
- Leave a margin for serialization and tokenizer differences.
- If the request is too large, reduce history, retrieve passages, summarize in stages, or split the job.
03
Choose a pattern based on the evidence shape.
| Pattern | Use when | Main risk |
|---|---|---|
| Full document | One bounded source must be read together | Cost and weak attention to middle sections |
| Retrieval | A small set of passages answers each query | Missing evidence during retrieval |
| Map-reduce | Many independent sections can be summarized | Losing cross-section relationships |
| Rolling summary | Conversation history grows over time | Summary drift and omitted commitments |
| Hierarchical analysis | Large corpora need section and corpus views | Complex orchestration and traceability |
04
Test recall, citation, latency, and cost at realistic lengths.
- Place answer-bearing evidence near the start, middle, and end of samples.
- Include conflicting and irrelevant passages to measure grounding.
- Require source identifiers or quoted spans that your application can verify.
- Record input/output tokens, first-token latency, total latency, and failure rate.
- Test context_length_exceeded and cancellation paths explicitly.