The Machine That Researches Itself
Deep Dive · Open Source · AI Systems — Machine Intelligence Review, March 2026
Andrej Karpathy's autoresearch is a deceptively simple framework with a radical premise: let an AI agent run your machine learning experiments autonomously, overnight, while you sleep. Wake up to 100 experiments and a better model.
karpathy/autoresearch · 8.5k stars · MIT License · Single GPU
01 · The Big Idea — A Closed Loop Between Agent and GPU
At its core, autoresearch is a hill-climbing search algorithm where the search space is code itself — not just hyperparameters. An AI agent reads a training file, forms a hypothesis, modifies the code, runs a 5-minute training session, and decides whether the result was better or worse. Then it does it again. Forever.
What makes this elegant is what it doesn't do. There's no distributed training, no YAML configs, no experiment tracker servers, no complex orchestration. One file to edit. One GPU. One metric. One agent.
The simplicity is the point. The constraint is the feature.
HUMAN defines goal: minimize val_bpb
↓
AGENT searches code space (train.py)
↓
GPU evaluates fitness (5-min budget)
↓
GIT maintains state + history
↓
BPB guides next hypothesis
↓
↑___________________________↑
repeat indefinitely
"Research is now entirely the domain of autonomous swarms of AI agents running across compute cluster megastructures. This repo is the story of how it all began." — @karpathy, March 2026
At a Glance
| 5 min | Per experiment (fixed time budget) |
| ~12 | Experiments per hour |
| ~100 | Experiments overnight |
| 1 | File the agent edits |
02 · Repository Architecture — Three Files That Matter
The entire system is built on a deliberate separation of concerns. One file is sacred, one is the agent's playground, and one is the human's control surface.
| File | Owner | Role |
|---|---|---|
prepare.py 🔒 READ-ONLY |
Fixed Infrastructure | Data download, BPE tokenizer, dataloader with best-fit packing, and the evaluate_bpb() ground-truth metric. The agent cannot touch this. It is the referee. |
train.py 🤖 AGENT EDITS |
AI Agent | Full GPT model, MuonAdamW optimizer, and training loop. Everything is fair game: architecture, hyperparameters, attention patterns, activation functions, batch size. |
program.md 👤 HUMAN EDITS |
Human Researcher | The agent's instruction manual and finite state machine spec. The "research org code" that governs how the AI behaves. This is what humans iterate on over time. |
results.tsv |
Agent (writes), Human (reads) | Flat append-only log of every experiment: commit hash, val_bpb, memory usage, keep/discard decision, description. Human-readable, grep-able, notebook-loadable. |
03 · System Design — Four Layers, One Loop
The system decomposes cleanly into four architectural layers, each with a single responsibility.
Layer 1 — Orchestration: Agent + program.md
The AI agent is the controller. program.md is its FSM spec:
IDLE → HYPOTHESIZE → MODIFY → COMMIT → EVALUATE → DECIDE → IDLE
Crucially, the agent maintains no internal memory between experiments — all state lives in git and the TSV log, making the system fully restartable and inspectable at any checkpoint.
Layer 2 — State: Git as Database
Git does double duty as both version control and experiment database:
- Commits = experiment records
- Branches = research runs
git reset --hard= rollback / discardgit log= full experiment lineage
You get diff, blame, and bisect for free — you can always trace exactly what change caused an improvement, with full reproducibility built in.
Layer 3 — Compute: Single-GPU Training Runtime
Intentionally single-node, single-GPU, self-contained. No distributed training, no config files. Just uv run train.py. The time budget (not epoch count) as the termination condition is the key design decision: it makes all experiments directly comparable regardless of whether the model grew or shrank, enabling fair cross-experiment comparison on your specific hardware.
train.py
│
├── GPTConfig (model hyperparams — agent edits these)
├── Model (architecture — agent edits this)
├── MuonAdamW (optimizer — agent edits this)
└── TrainLoop (calls prepare.py's evaluate_bpb at end)
│
▼
stdout: val_bpb, peak_vram_mb, mfu_percent, ...
Layer 4 — Evaluation: BPB Metric + results.tsv
Bits-per-byte (BPB) is the single ground truth metric. It's vocabulary-size-independent, which means architectural changes that alter vocab size can still be fairly compared — a critical property when the agent is free to experiment with tokenizer and model dimensions simultaneously.
04 · The Experiment Loop — Eight Steps, Repeated Forever
The agent runs this loop indefinitely until the human manually stops it. Each iteration is roughly 6–7 minutes wall-clock time — 5 minutes of training plus overhead.
1. Check Git State Verify the working directory is clean, branch is correct, and identify the baseline commit to roll back to if needed.
2. Generate Hypothesis Form an idea based on past results, literature knowledge, or random exploration. E.g. "increasing LR may help since the loss curve suggests underfitting."
3. Modify train.py
Make targeted, interpretable changes with explanatory comments. Only train.py is touched — never prepare.py.
4. Commit A descriptive git commit becomes the experiment's permanent identifier in the log.
5. Run Training (5 min)
uv run train.py runs exactly 300 seconds, then emits val_bpb, memory, MFU, and token throughput to stdout.
6. Extract Results Parse key metrics from the training log using grep.
7. Log to results.tsv
Append commit hash, val_bpb, memory, and description to the append-only results file.
8. Keep or Discard
If improvement: keep the commit and continue. If regression or crash: git reset --hard HEAD~1 and try again.
Decision Logic
Training complete (val_bpb = X)
↓
Crashed?
/ \
YES NO → bpb < best?
↓ / \
discard IMPROVED REGRESSION
fix or try keep commit git reset
different update best --hard HEAD~1
idea continue try next idea
05 · Technical Highlights — What the Agent Starts With
The baseline train.py is not a naive GPT-2 clone. It's packed with modern ML research insights — which also serves as fertile ground for the agent to improve upon.
Model Architecture
MODEL · 01 — RoPE + QK-Norm Rotary position embeddings for relative position awareness, combined with query-key normalization to prevent attention logit explosion.
MODEL · 02 — Sliding Window Attention
Pattern-based attention windows (e.g. SSSL) reduce O(n²) complexity. Early layers focus locally, later layers attend globally.
MODEL · 03 — ReLU² + Logit Softcap Squared ReLU activation is faster than GeLU and works well with Muon. Final logits are soft-capped via tanh for numerical stability.
Optimizer
OPTIMIZER · 04 — Muon for Matrices Spectral normalization via approximate SVD (Polar Express orthogonalization) for 2D weight matrices. Prevents gradient explosion without full SVD cost.
OPTIMIZER · 05 — AdamW for Embeddings Hybrid optimizer: Muon handles matrices, AdamW handles embeddings and scalars — each with tuned learning rates suited to their update characteristics.
┌──────────────────────────────────────────────────────────────┐
│ MuonAdamW Hybrid │
├──────────────────────┬────────────┬──────────────────────────┤
│ Parameter Type │ Optimizer │ Learning Rate │
├──────────────────────┼────────────┼──────────────────────────┤
│ 2D Matrices (Wq, Wk) │ Muon │ 0.04 × √(d_model) │
│ Embeddings (wte) │ AdamW │ 0.6 │
│ Unembedding (lm_head)│ AdamW │ 0.004 │
│ Scalars (lambdas) │ AdamW │ 0.5 │
└──────────────────────┴────────────┴──────────────────────────┘
OPTIMIZER · 06 — Warmdown LR Schedule Linear warmdown over the final 50% of training steps. Momentum ramps from 0.85 → 0.95, giving early responsiveness and late-stage stability.
LR
│ ┌──────────────────┐
│ / \
│ / \
│ / \
└────────────────────────────► Steps
↑ ↑ ↑
Warmup Constant Warmdown
(last 50%)
06 · Design Philosophy
Intentional Constraints as Features
Every limitation in autoresearch is a deliberate design decision, not a gap to be filled later.
Single file to modify keeps diffs reviewable and scope manageable. An agent that can touch any file is an agent that's hard to reason about.
Time budget over epoch count means your results are optimized for your specific GPU — not a theoretical training curve. The downside is that results aren't portable across hardware. That's accepted.
Git as the state machine means you get reproducibility, lineage, and rollback for free — using a tool every developer already knows.
BPB as the sole metric eliminates the entire class of "my model is better but it cheated by using a bigger vocab" comparisons.
Where It Breaks at Scale
| Bottleneck | Fix |
|---|---|
| Single agent, sequential experiments | Multi-agent swarm with shared results log |
| 5-min budget limits model size | Time-normalize across GPU tiers |
| Flat TSV, no structured queries | SQLite or MLflow experiment tracker |
| No inter-run memory | RAG over past results across branches |
| Agent rediscovers dead ends | Separate strategist agent reads logs and steers |
07 · The Vision — A Template for the Future
Autoresearch isn't trying to replace ML researchers. It's a proof of concept that the loop itself — hypothesize, test, keep or discard — can be run by an AI agent with minimal human scaffolding.
The repo is opinionated about what matters: a clear fitness metric, a constrained search space, and a stateless agent that relies on external state. These properties make the system auditable, extensible, and surprisingly robust for something so small.
Whether you point Claude, Codex, or any other capable agent at it, the result is the same: you program the research process in program.md, and the machine does the experiments. The human moves up the stack from "person who runs training jobs" to "person who designs research strategies."
That shift — subtle as it looks in a 300-line repo — is the whole point.
Quick Start
# 1. Install uv project manager
curl -LsSf https://astral.sh/uv/install.sh | sh
# 2. Install dependencies
uv sync
# 3. Prepare data + tokenizer (one-time, ~2 min)
uv run prepare.py
# 4. Verify setup with a single run
uv run train.py
# 5. Point your agent at the repo and prompt:
# "Hi, have a look at program.md and let's kick off
# a new experiment run. Let's do the setup first."
# Then go to sleep. Wake up to:
cat results.tsv | sort -t$'\t' -k2 -n | head -10
Requirements: Single NVIDIA GPU (tested on H100) · Python 3.10+ · uv
Source: github.com/karpathy/autoresearch · MIT License · 8.5k ★ The Research Letter · Machine Intelligence Review · March 2026