RESEARCH LIBRARY / 100 SOURCE-LED REVIEWS
Browse 80 foundational papers from major AI labs and 20 papers on inference, evaluation, and serving. Every review keeps the original authors, organizations, and primary source attached.
MAJOR LAB RESEARCH
Google Brain / Google Research
The paper replaces recurrent sequence processing with a Transformer built entirely from attention and feed-forward layers, making token interactions parallelizable.
Read source-led reviewMAJOR LAB RESEARCH
Google AI Language
BERT pretrains a deep bidirectional Transformer encoder by masking tokens and then adapts the same representation to many language-understanding tasks.
Read source-led reviewMAJOR LAB RESEARCH
Google Research
T5 casts every studied language problem as text-to-text, unifying inputs, targets, training objectives, and evaluation under one Transformer interface.
Read source-led reviewMAJOR LAB RESEARCH
Google Research
Switch Transformer simplifies sparse mixture-of-experts routing so very large models activate only one expert per token while keeping computation manageable.
Read source-led reviewMAJOR LAB RESEARCH
Google Research
PaLM trains a 540-billion-parameter dense decoder model with the Pathways system and examines how language, reasoning, coding, and multilingual capability change with scale.
Read source-led reviewMAJOR LAB RESEARCH
Google Research / Google DeepMind
PaLM 2 reports a family of multilingual foundation models trained with revised data, objectives, and scaling choices to improve reasoning and language coverage.
Read source-led reviewMAJOR LAB RESEARCH
Google / Google DeepMind
Gemini presents a family of natively multimodal models that process text, images, audio, and video and are evaluated across reasoning and perception tasks.
Read source-led reviewMAJOR LAB RESEARCH
Google / Google DeepMind
Gemini 1.5 introduces a multimodal mixture-of-experts model and studies retrieval and understanding over context windows extending to millions of tokens.
Read source-led reviewMAJOR LAB RESEARCH
Google / Google DeepMind
Gemma describes lightweight open models derived from Gemini-era research, including training, evaluation, responsible-release work, and deployment-oriented variants.
Read source-led reviewMAJOR LAB RESEARCH
DeepMind
The Chinchilla study argues that, under a fixed compute budget, many language models were too large and trained on too few tokens.
Read source-led reviewMAJOR LAB RESEARCH
DeepMind
RETRO augments a language model with nearest-neighbor chunks retrieved from a database containing trillions of tokens, reducing reliance on parameters alone.
Read source-led reviewMAJOR LAB RESEARCH
DeepMind
Flamingo connects pretrained vision and language components through gated cross-attention so one model can perform new image-and-text tasks from a few demonstrations.
Read source-led reviewMAJOR LAB RESEARCH
DeepMind
Gato trains one sequence model on tokenized observations and actions from hundreds of embodied, game, vision, and language tasks.
Read source-led reviewMAJOR LAB RESEARCH
DeepMind
The Sparrow work trains a dialogue agent with targeted human judgments about helpfulness, correctness, evidence, and conversational rules.
Read source-led reviewMAJOR LAB RESEARCH
Google Research / DeepMind
Med-PaLM adapts a large language model to medical question answering and introduces clinician-focused evaluation beyond automatic benchmark scores.
Read source-led reviewMAJOR LAB RESEARCH
Google Research
Minerva further trains a large language model on technical material and uses step-by-step prompting plus answer aggregation for quantitative reasoning.
Read source-led reviewMAJOR LAB RESEARCH
Google Research
Chain-of-thought prompting shows that a few worked natural-language rationales can elicit multi-step reasoning from sufficiently large language models.
Read source-led reviewMAJOR LAB RESEARCH
Google Research
Self-consistency samples multiple reasoning paths and selects the most common answer, replacing a single greedy chain with consensus over diverse derivations.
Read source-led reviewMAJOR LAB RESEARCH
DeepMind
Perceiver uses a small latent array to repeatedly attend to very large inputs, decoupling computational cost from raw input size.
Read source-led reviewMAJOR LAB RESEARCH
DeepMind
Perceiver IO extends the latent-bottleneck architecture with flexible output queries, allowing structured outputs of many different sizes and semantics.
Read source-led reviewMAJOR LAB RESEARCH
DeepMind
MuZero learns a model tailored to planning without reconstructing the environment, predicting only rewards, values, and policies needed by tree search.
Read source-led reviewMAJOR LAB RESEARCH
DeepMind
AlphaZero learns chess and shogi from self-play using one general reinforcement-learning and tree-search algorithm without human game data.
Read source-led reviewMAJOR LAB RESEARCH
OpenAI
GPT-3 studies a 175-billion-parameter autoregressive language model that performs many tasks from instructions or examples without gradient updates.
Read source-led reviewMAJOR LAB RESEARCH
OpenAI
InstructGPT combines demonstrations, preference comparisons, a learned reward model, and reinforcement learning to make language models follow user instructions more reliably.
Read source-led reviewMAJOR LAB RESEARCH
OpenAI
This work trains summarizers from human comparisons, using a reward model to optimize outputs toward judgments of summary quality.
Read source-led reviewMAJOR LAB RESEARCH
OpenAI
WebGPT fine-tunes a language model to search and navigate the web, collect references, and compose cited answers that humans compare for usefulness and factual support.
Read source-led reviewMAJOR LAB RESEARCH
OpenAI
The GPT-4 report describes a large multimodal model, its benchmark performance, predictable scaling work, post-training, safety evaluation, and known limitations without disclosing full architecture details.
Read source-led reviewMAJOR LAB RESEARCH
OpenAI
DALL-E treats text and image tokens as one autoregressive sequence and studies zero-shot image generation from natural-language descriptions.
Read source-led reviewMAJOR LAB RESEARCH
OpenAI
CLIP learns aligned image and text representations by predicting which captions belong to which images across a large web-derived dataset.
Read source-led reviewMAJOR LAB RESEARCH
OpenAI
GLIDE conditions a diffusion model on text and compares classifier-free guidance with CLIP guidance for photorealistic generation and editing.
Read source-led reviewMAJOR LAB RESEARCH
OpenAI
DALL-E 2 uses a prior to map text into CLIP image representations and a diffusion decoder to generate images conditioned on those representations.
Read source-led reviewMAJOR LAB RESEARCH
OpenAI
Whisper trains an encoder-decoder Transformer on a very large, weakly supervised collection of multilingual speech to perform transcription and translation robustly.
Read source-led reviewMAJOR LAB RESEARCH
OpenAI
This study fits empirical power laws relating language-model loss to parameter count, dataset size, and training compute over the ranges examined.
Read source-led reviewMAJOR LAB RESEARCH
OpenAI
Jukebox models music as compressed audio tokens with hierarchical autoregressive Transformers conditioned on artist, genre, and optionally lyrics.
Read source-led reviewMAJOR LAB RESEARCH
OpenAI
Let's Verify Step by Step trains verifiers to judge individual reasoning steps and finds process supervision more effective than outcome-only supervision on the studied math problems.
Read source-led reviewMAJOR LAB RESEARCH
OpenAI
Weak-to-strong generalization studies whether a strong model can learn a task from labels produced by a weaker model, using controlled model-size gaps as an analogy for future oversight.
Read source-led reviewMAJOR LAB RESEARCH
Anthropic
Constitutional AI uses written principles and model-generated critiques and revisions to reduce dependence on direct human labels for harmlessness training.
Read source-led reviewMAJOR LAB RESEARCH
Anthropic / DeepMind
This paper uses language models to generate adversarial test questions for other language models, increasing the scale and diversity of red-team evaluation.
Read source-led reviewMAJOR LAB RESEARCH
Anthropic
The paper evaluates whether language models can estimate the probability that their own answers are correct, including via verbalized confidence and learned probes.
Read source-led reviewMAJOR LAB RESEARCH
Anthropic
Model-written evaluations use language models to propose test cases for behaviors such as bias, deception, power seeking, and other difficult-to-enumerate tendencies.
Read source-led reviewMAJOR LAB RESEARCH
Anthropic
This work tests scalable-oversight protocols on book-length tasks where models answer questions and human judges receive different forms of model assistance.
Read source-led reviewMAJOR LAB RESEARCH
Anthropic
The paper studies whether instruction-following models can reduce biased responses when prompts explicitly ask them to consider fairness or avoid stereotypes.
Read source-led reviewMAJOR LAB RESEARCH
Anthropic / Collective Intelligence Project
Collective Constitutional AI develops a process for gathering public input into model principles and compares a model trained with those principles against a developer-authored baseline.
Read source-led reviewMAJOR LAB RESEARCH
Anthropic / Redwood Research
Sleeper Agents constructs models with conditional deceptive behavior and tests whether standard safety training removes or conceals that behavior.
Read source-led reviewMAJOR LAB RESEARCH
Anthropic
This paper investigates sycophancy, where assistants mirror a user's stated beliefs or preferences even when doing so conflicts with truthfulness.
Read source-led reviewMAJOR LAB RESEARCH
Anthropic
Toy Models of Superposition uses small neural networks to study how more features than available dimensions can be represented through overlapping directions.
Read source-led reviewMAJOR LAB RESEARCH
DeepSeek-AI
DeepSeek LLM reports a large-scale open language-model project spanning data construction, scaling experiments, pretraining, supervised tuning, and capability evaluation.
Read source-led reviewMAJOR LAB RESEARCH
DeepSeek-AI
DeepSeek-V2 combines mixture-of-experts routing with multi-head latent attention to reduce training and inference costs while supporting long context.
Read source-led reviewMAJOR LAB RESEARCH
DeepSeek-AI
DeepSeek-V3 scales a sparse mixture-of-experts model with multi-head latent attention and auxiliary-loss-free load balancing, alongside extensive training-efficiency work.
Read source-led reviewMAJOR LAB RESEARCH
DeepSeek-AI
DeepSeek-R1 studies reinforcement learning for reasoning, including a model trained without an initial supervised reasoning phase and a production pipeline that adds curated post-training data.
Read source-led reviewMAJOR LAB RESEARCH
DeepSeek-AI
DeepSeek-Coder builds open code models from a two-trillion-token corpus with a large code share, project-level packing, and long-context adaptation.
Read source-led reviewMAJOR LAB RESEARCH
DeepSeek-AI
DeepSeekMath trains an open mathematical language model on a large web-derived math corpus and studies reinforcement learning with group-relative policy optimization.
Read source-led reviewMAJOR LAB RESEARCH
DeepSeek-AI
DeepSeek-VL couples a vision encoder with language models and uses staged training to preserve language ability while learning real-world visual understanding.
Read source-led reviewMAJOR LAB RESEARCH
DeepSeek-AI
DeepSeekMoE restructures sparse expert models so experts specialize more finely while some experts remain shared across tokens.
Read source-led reviewMAJOR LAB RESEARCH
DeepSeek-AI
DeepSeek-Coder-V2 extends the code-model line with sparse mixture-of-experts training, broader language coverage, and longer contexts.
Read source-led reviewMAJOR LAB RESEARCH
DeepSeek-AI
DeepSeek-Prover-V1.5 trains formal theorem-proving models with proof-assistant feedback and uses tree search to improve proof discovery at inference time.
Read source-led reviewMAJOR LAB RESEARCH
Meta AI
LLaMA trains smaller foundation models on publicly described data and shows that data-rich training can make compact checkpoints competitive with much larger systems.
Read source-led reviewMAJOR LAB RESEARCH
Meta AI
Llama 2 documents pretrained and chat-tuned model families, including supervised instruction tuning, reinforcement learning from human feedback, and safety evaluation.
Read source-led reviewMAJOR LAB RESEARCH
Meta AI
The Llama 3 report presents a multilingual model family, its data and scaling choices, long-context work, post-training stack, multimodal extensions, and safety evaluations.
Read source-led reviewMAJOR LAB RESEARCH
Meta AI Research
Segment Anything trains a promptable segmentation model and pairs it with a large automatically assisted mask dataset for broad zero-shot transfer.
Read source-led reviewMAJOR LAB RESEARCH
Meta AI Research
DINOv2 learns general-purpose visual features through large-scale self-supervision and a curated image pipeline without requiring labels for pretraining.
Read source-led reviewMAJOR LAB RESEARCH
Meta AI Research
ImageBind aligns images with text, audio, depth, thermal, and inertial signals in one embedding space using images as the binding modality.
Read source-led reviewMAJOR LAB RESEARCH
Meta AI
OPT releases a suite of decoder-only pretrained models and a detailed logbook intended to support reproducible research on large-language-model training.
Read source-led reviewMAJOR LAB RESEARCH
Microsoft Research
LoRA freezes a pretrained model and learns low-rank updates inside selected weight matrices, reducing trainable parameters and deployment overhead.
Read source-led reviewMAJOR LAB RESEARCH
Microsoft Research
Phi-3 studies compact language models trained on carefully selected and synthetic data, including a model designed to run within phone-class memory limits.
Read source-led reviewMAJOR LAB RESEARCH
Microsoft Research
Orca trains a smaller model on complex explanation traces from stronger teachers while varying prompts and tasks to encourage broader imitation than answer-only distillation.
Read source-led reviewMAJOR LAB RESEARCH
Microsoft Research
DeBERTa separates content and position representations in attention and adds an enhanced masked-token decoder for stronger language understanding.
Read source-led reviewMAJOR LAB RESEARCH
NVIDIA
Megatron-LM implements model-parallel Transformer training with carefully placed tensor partitions and communication operations.
Read source-led reviewMAJOR LAB RESEARCH
NVIDIA
StyleGAN redesigns the generator so a learned style representation controls image synthesis at multiple resolutions while stochastic inputs model fine variation.
Read source-led reviewMAJOR LAB RESEARCH
Mistral AI
Mistral 7B combines grouped-query attention and sliding-window attention to deliver a compact open-weight language model with efficient inference characteristics.
Read source-led reviewMAJOR LAB RESEARCH
Mistral AI
Mixtral uses sparse mixture-of-experts layers so each token activates a subset of feed-forward experts while retaining an open deployment path.
Read source-led reviewMAJOR LAB RESEARCH
Alibaba Cloud
Qwen describes a multilingual decoder model family and its chat post-training, code and mathematics variants, tool use, and broad evaluation.
Read source-led reviewMAJOR LAB RESEARCH
Alibaba Cloud
Qwen2 expands the Qwen family across dense and mixture-of-experts sizes with more languages, longer context, and revised training and alignment.
Read source-led reviewMAJOR LAB RESEARCH
Massachusetts Institute of Technology
The Lottery Ticket Hypothesis finds sparse subnetworks inside randomly initialized dense networks that can train effectively when reset to their original initialization.
Read source-led reviewMAJOR LAB RESEARCH
University of California, Berkeley
Denoising diffusion probabilistic models learn to reverse a gradual noising process, turning random noise into data through a parameterized denoising chain.
Read source-led reviewMAJOR LAB RESEARCH
University of Washington / Allen Institute for AI
YOLO reframes object detection as a single regression problem that predicts boxes and class probabilities directly from a full image.
Read source-led reviewMAJOR LAB RESEARCH
Microsoft Research
ResNet uses identity shortcut connections so very deep convolutional networks can optimize residual functions without the degradation seen in plain networks.
Read source-led reviewMAJOR LAB RESEARCH
University of Toronto
Dropout regularizes neural networks by randomly omitting units during training, reducing co-adaptation and approximating an ensemble of many thinned networks.
Read source-led reviewMAJOR LAB RESEARCH
Google Research
Vision Transformer applies a largely standard Transformer encoder to sequences of image patches and studies when large-scale pretraining makes it competitive with convolutional networks.
Read source-led reviewMAJOR LAB RESEARCH
Carnegie Mellon University / Princeton University
Mamba makes state-space model parameters input-dependent and couples the selective recurrence with a hardware-aware parallel algorithm for long sequences.
Read source-led reviewINFERENCE / EVALUATION / SERVING
University of California, Berkeley
vLLM introduces PagedAttention and a serving engine that manages key-value cache blocks like virtual memory so request batches waste less accelerator memory.
Read source-led reviewINFERENCE / EVALUATION / SERVING
Microsoft Research / Microsoft DeepSpeed
DeepSpeed Inference co-designs Transformer kernels, tensor parallelism, and quantization to serve very large dense and sparse models across accelerators.
Read source-led reviewINFERENCE / EVALUATION / SERVING
LMSYS Org / University of California, Berkeley / Stanford University
SGLang provides a language and runtime for structured language-model programs, using shared-prefix caching and constrained-generation optimizations across multi-call workflows.
Read source-led reviewINFERENCE / EVALUATION / SERVING
University of California, Berkeley
AlpaServe uses statistical multiplexing and model-parallel placement to serve many large models whose traffic varies over time.
Read source-led reviewINFERENCE / EVALUATION / SERVING
Stanford University
FlexGen schedules model weights, activations, and key-value cache across GPU, CPU, and storage to run large language models on a single memory-limited accelerator.
Read source-led reviewINFERENCE / EVALUATION / SERVING
BigScience / Hivemind
Petals coordinates volunteer or independently operated machines so users can run and adapt large models without one participant hosting every layer.
Read source-led reviewINFERENCE / EVALUATION / SERVING
Stanford University
FlashAttention-2 reorganizes exact attention computation to improve accelerator occupancy, reduce non-matrix work, and parallelize more effectively across sequence length.
Read source-led reviewINFERENCE / EVALUATION / SERVING
Google Research
Multi-query attention shares one set of key and value heads across attention queries, reducing memory bandwidth during autoregressive decoding.
Read source-led reviewINFERENCE / EVALUATION / SERVING
Google Research
Grouped-query attention interpolates between multi-head and multi-query attention by sharing key-value heads within groups and uptrains existing checkpoints.
Read source-led reviewINFERENCE / EVALUATION / SERVING
Google DeepMind
Speculative decoding lets a faster draft model propose token blocks that a larger target model verifies in parallel while preserving the target distribution.
Read source-led reviewINFERENCE / EVALUATION / SERVING
Together AI
Medusa adds multiple decoding heads to a frozen language model so several candidate future tokens can be proposed and verified together.
Read source-led reviewINFERENCE / EVALUATION / SERVING
Peking University / University of California, San Diego
DistServe separates prompt prefill from autoregressive decoding onto different accelerator pools so each phase can be provisioned for its own latency objective.
Read source-led reviewINFERENCE / EVALUATION / SERVING
Microsoft Research / University of Washington
Splitwise studies phase splitting for generative inference, placing compute-heavy prompt processing and memory-heavy token generation on hardware suited to each phase.
Read source-led reviewINFERENCE / EVALUATION / SERVING
Microsoft Research
Sarathi-Serve uses chunked prefills and iteration-level scheduling to reduce interference between long prompt processing and ongoing token generation.
Read source-led reviewINFERENCE / EVALUATION / SERVING
University of Washington / University of California, San Diego
Punica serves many LoRA-specialized models over shared base weights using batching and GPU kernels designed for mixed adapters.
Read source-led reviewINFERENCE / EVALUATION / SERVING
Stanford University
S-LoRA combines unified paging and custom kernels to keep many adapters in host memory and bring active LoRA weights into shared-model batches efficiently.
Read source-led reviewINFERENCE / EVALUATION / SERVING
Massachusetts Institute of Technology / NVIDIA
SmoothQuant moves quantization difficulty from activation outliers into weights through an offline, mathematically equivalent channel-wise scaling transform.
Read source-led reviewINFERENCE / EVALUATION / SERVING
Institute of Science and Technology Austria
GPTQ quantizes large Transformer weights after training using approximate second-order information and an efficient layer-wise reconstruction procedure.
Read source-led reviewINFERENCE / EVALUATION / SERVING
Massachusetts Institute of Technology / NVIDIA
AWQ protects weight channels associated with large activation magnitudes and searches scaling choices using a small calibration set for low-bit deployment.
Read source-led reviewINFERENCE / EVALUATION / SERVING
Stanford Center for Research on Foundation Models
HELM defines a transparent, scenario-based framework for evaluating language models across accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency.
Read source-led review