BrokenGPT is being built in three phases. This journal will publish the methods, evaluations, costs, failures, and model decisions behind each one—not just the launch announcement.
DISPATCH 001PHASE 01 / LIVE
01
JULY 2026 · FOUNDATION NOTE
Why we are starting with a private open-source mixture.
Phase 1 gives us a measurable baseline. A privately routed mixture of open-source models powers chat and the compatible API while the same gateway records usage, latency, and product behavior. Starting here lets us separate improvements to the model from improvements to the service around it.
The result is a working foundation and a record we can compare against when Phase 2 begins. Model identity, context limits, metering, and the service boundary remain visible throughout the process.
From refusal reduction to full-precision evaluation.
How we select checkpoints, build refusal and accuracy evaluations, test regressions, and decide whether a change is actually better.
PHASE 02 REPORT / PLANNEDDISPATCH 003 · TARGET EARLY 2027
Training a BrokenGPT-native model.
The data decisions, training approach, capability results, limitations, and deployment architecture behind our own uncensored model.
PHASE 03 REPORT / PLANNED
RESEARCH LIBRARY / 100 SOURCE-LED REVIEWS
The paper desk.
Browse 80 foundational papers from major AI labs and 20 papers on inference, evaluation, and serving. Every review keeps the original authors, organizations, and primary source attached.
The paper replaces recurrent sequence processing with a Transformer built entirely from attention and feed-forward layers, making token interactions parallelizable.
BERT pretrains a deep bidirectional Transformer encoder by masking tokens and then adapts the same representation to many language-understanding tasks.
T5 casts every studied language problem as text-to-text, unifying inputs, targets, training objectives, and evaluation under one Transformer interface.
Switch Transformer simplifies sparse mixture-of-experts routing so very large models activate only one expert per token while keeping computation manageable.
PaLM trains a 540-billion-parameter dense decoder model with the Pathways system and examines how language, reasoning, coding, and multilingual capability change with scale.
PaLM 2 reports a family of multilingual foundation models trained with revised data, objectives, and scaling choices to improve reasoning and language coverage.
Gemini presents a family of natively multimodal models that process text, images, audio, and video and are evaluated across reasoning and perception tasks.
Gemini 1.5 introduces a multimodal mixture-of-experts model and studies retrieval and understanding over context windows extending to millions of tokens.
Gemma describes lightweight open models derived from Gemini-era research, including training, evaluation, responsible-release work, and deployment-oriented variants.
RETRO augments a language model with nearest-neighbor chunks retrieved from a database containing trillions of tokens, reducing reliance on parameters alone.
Flamingo connects pretrained vision and language components through gated cross-attention so one model can perform new image-and-text tasks from a few demonstrations.
Minerva further trains a large language model on technical material and uses step-by-step prompting plus answer aggregation for quantitative reasoning.
Chain-of-thought prompting shows that a few worked natural-language rationales can elicit multi-step reasoning from sufficiently large language models.
Self-consistency samples multiple reasoning paths and selects the most common answer, replacing a single greedy chain with consensus over diverse derivations.
Perceiver IO extends the latent-bottleneck architecture with flexible output queries, allowing structured outputs of many different sizes and semantics.
MuZero learns a model tailored to planning without reconstructing the environment, predicting only rewards, values, and policies needed by tree search.
InstructGPT combines demonstrations, preference comparisons, a learned reward model, and reinforcement learning to make language models follow user instructions more reliably.
WebGPT fine-tunes a language model to search and navigate the web, collect references, and compose cited answers that humans compare for usefulness and factual support.
The GPT-4 report describes a large multimodal model, its benchmark performance, predictable scaling work, post-training, safety evaluation, and known limitations without disclosing full architecture details.
Whisper trains an encoder-decoder Transformer on a very large, weakly supervised collection of multilingual speech to perform transcription and translation robustly.
Let's Verify Step by Step trains verifiers to judge individual reasoning steps and finds process supervision more effective than outcome-only supervision on the studied math problems.
Weak-to-strong generalization studies whether a strong model can learn a task from labels produced by a weaker model, using controlled model-size gaps as an analogy for future oversight.
Constitutional AI uses written principles and model-generated critiques and revisions to reduce dependence on direct human labels for harmlessness training.
This paper uses language models to generate adversarial test questions for other language models, increasing the scale and diversity of red-team evaluation.
The paper evaluates whether language models can estimate the probability that their own answers are correct, including via verbalized confidence and learned probes.
Model-written evaluations use language models to propose test cases for behaviors such as bias, deception, power seeking, and other difficult-to-enumerate tendencies.
This work tests scalable-oversight protocols on book-length tasks where models answer questions and human judges receive different forms of model assistance.
The paper studies whether instruction-following models can reduce biased responses when prompts explicitly ask them to consider fairness or avoid stereotypes.
Collective Constitutional AI develops a process for gathering public input into model principles and compares a model trained with those principles against a developer-authored baseline.
Toy Models of Superposition uses small neural networks to study how more features than available dimensions can be represented through overlapping directions.
DeepSeek LLM reports a large-scale open language-model project spanning data construction, scaling experiments, pretraining, supervised tuning, and capability evaluation.
DeepSeek-V3 scales a sparse mixture-of-experts model with multi-head latent attention and auxiliary-loss-free load balancing, alongside extensive training-efficiency work.
DeepSeek-R1 studies reinforcement learning for reasoning, including a model trained without an initial supervised reasoning phase and a production pipeline that adds curated post-training data.
DeepSeekMath trains an open mathematical language model on a large web-derived math corpus and studies reinforcement learning with group-relative policy optimization.
DeepSeek-VL couples a vision encoder with language models and uses staged training to preserve language ability while learning real-world visual understanding.
DeepSeek-Prover-V1.5 trains formal theorem-proving models with proof-assistant feedback and uses tree search to improve proof discovery at inference time.
LLaMA trains smaller foundation models on publicly described data and shows that data-rich training can make compact checkpoints competitive with much larger systems.
Llama 2 documents pretrained and chat-tuned model families, including supervised instruction tuning, reinforcement learning from human feedback, and safety evaluation.
The Llama 3 report presents a multilingual model family, its data and scaling choices, long-context work, post-training stack, multimodal extensions, and safety evaluations.
DINOv2 learns general-purpose visual features through large-scale self-supervision and a curated image pipeline without requiring labels for pretraining.
OPT releases a suite of decoder-only pretrained models and a detailed logbook intended to support reproducible research on large-language-model training.
Phi-3 studies compact language models trained on carefully selected and synthetic data, including a model designed to run within phone-class memory limits.
Orca trains a smaller model on complex explanation traces from stronger teachers while varying prompts and tasks to encourage broader imitation than answer-only distillation.
StyleGAN redesigns the generator so a learned style representation controls image synthesis at multiple resolutions while stochastic inputs model fine variation.
Mistral 7B combines grouped-query attention and sliding-window attention to deliver a compact open-weight language model with efficient inference characteristics.
The Lottery Ticket Hypothesis finds sparse subnetworks inside randomly initialized dense networks that can train effectively when reset to their original initialization.
Denoising diffusion probabilistic models learn to reverse a gradual noising process, turning random noise into data through a parameterized denoising chain.
ResNet uses identity shortcut connections so very deep convolutional networks can optimize residual functions without the degradation seen in plain networks.
Dropout regularizes neural networks by randomly omitting units during training, reducing co-adaptation and approximating an ensemble of many thinned networks.
Vision Transformer applies a largely standard Transformer encoder to sequences of image patches and studies when large-scale pretraining makes it competitive with convolutional networks.
Mamba makes state-space model parameters input-dependent and couples the selective recurrence with a hardware-aware parallel algorithm for long sequences.
vLLM introduces PagedAttention and a serving engine that manages key-value cache blocks like virtual memory so request batches waste less accelerator memory.
DeepSpeed Inference co-designs Transformer kernels, tensor parallelism, and quantization to serve very large dense and sparse models across accelerators.
LMSYS Org / University of California, Berkeley / Stanford University
SGLang provides a language and runtime for structured language-model programs, using shared-prefix caching and constrained-generation optimizations across multi-call workflows.
FlexGen schedules model weights, activations, and key-value cache across GPU, CPU, and storage to run large language models on a single memory-limited accelerator.
FlashAttention-2 reorganizes exact attention computation to improve accelerator occupancy, reduce non-matrix work, and parallelize more effectively across sequence length.
Grouped-query attention interpolates between multi-head and multi-query attention by sharing key-value heads within groups and uptrains existing checkpoints.
Speculative decoding lets a faster draft model propose token blocks that a larger target model verifies in parallel while preserving the target distribution.
Peking University / University of California, San Diego
DistServe separates prompt prefill from autoregressive decoding onto different accelerator pools so each phase can be provisioned for its own latency objective.
Splitwise studies phase splitting for generative inference, placing compute-heavy prompt processing and memory-heavy token generation on hardware suited to each phase.
S-LoRA combines unified paging and custom kernels to keep many adapters in host memory and bring active LoRA weights into shared-model batches efficiently.
SmoothQuant moves quantization difficulty from activation outliers into weights through an offline, mathematically equivalent channel-wise scaling transform.
GPTQ quantizes large Transformer weights after training using approximate second-order information and an efficient layer-wise reconstruction procedure.
AWQ protects weight channels associated with large activation magnitudes and searches scaling choices using a small calibration set for low-bit deployment.
HELM defines a transparent, scenario-based framework for evaluating language models across accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency.
