A benchmark and open call for valuable, reproducible RL runs. Each task is a real training run that went wrong. The agent has to find the root cause (issue) and, where a repaired run exists, fix it and rerun (solution).
- 6 Nov 2026 — We release the second version of the task list: 32 tasks, 24 of which support solution-level evaluation.
- 2 Nov 2026 — We release the first version of the task list: 18 tasks, 13 of which support solution-level evaluation.
- 24 Oct 2026 — Call for contributions officially open, and Kimi-K3.1-Pro added to the leaderboard.
- 19 Oct 2026 — GitHub repo and website released.
AutoRLBench Leaderboard, issue + solution evaluation. Overall is the combined pass rate across all evaluated runs (issue + solution): passes over total runs, not an average of the two levels.
| # | Model | Agent | Issue Score | Solution Score | Overall |
|---|---|---|---|---|---|
| 1 | GPT-6-Astra | Codex | 33.3 % | 38.9 % | 35.2 % |
| 2 | GPT-6.1-Sol | Codex | 30.6 % | 38.9 % | 33.3 % |
| 3 | Gemini-4-Argon | Claude Code | 16.7 % | 55.6 % | 29.6 % |
| 4 | GLM-5.3 | Claude Code | 22.2 % | 33.3 % | 25.9 % |
| 5 | Kimi-K3.1-Pro | Claude Code | 16.7 % | 38.9 % | 24.1 % |
| 6 | Claude-Fable-5.1 | Claude Code | 16.7 % | 27.8 % | 20.4 % |
| 7 | Qwen3.8-Max | Qwen Code | 13.9 % | 27.8 % | 18.5 % |
| 8 | Grok-4.7 | Grok Build | 16.7 % | 16.7 % | 16.7 % |
| 9 | Kimi-K3 | Claude Code | 0.0 % | 16.7 % | 5.6 % |
| 10 | DeepSeek-V4-Pro | Claude Code | 0.0 % | 5.6 % | 1.9 % |
Six domains, one representative task each. The domains come from where in the RL pipeline the failure lives:
Configuration · issueTRL GSM8K EOS mismatch Missing <|im_end|> in eos_token_id lets rollouts continue past the answer.
|
Reward · issue · solutionLength bonus reward hack An unconditional length bonus teaches the policy to write longer, not better. |
Data · issueTraining data is too hard Train is TACO VERY_HARD while validation is EASY; held-out accuracy ends below its start.
|
Algorithm · issueGRPO advantage broadcast An advantage broadcast replaces per-sample advantages, causing policy collapse. |
Data · issueTraining data too hard Training on VERY_HARD while validating on EASY, held-out accuracy falls below the starting point. |
Infrastructure · issue · solutionMoE auxiliary gradients miss token normalization The per-token loss callback omits the token count, so MoE z-loss gradients are never normalized. |
prepare ──▶ agent ──▶ [rerun] ──▶ verifier
│ │ │ │
│ │ │ └─ issue: LLM judge on agent_answer.md vs. hidden oracle
│ │ │ solution: a metric threshold on the rerun's logs
│ │ └─ solution only: the harness re-runs the experiment on the agent's
│ │ workspace, per the task's run_spec.json
│ └─ reads instruction.md and a writable copy of workspace/
└─ fetches pinned models and datasets (the only stage with network)
Each stage runs in its own container from the same prebuilt image. Oracles and verifiers are mounted only into the verifier stage.
At the solution level the agent repairs the workspace and submits report.md — it does not launch training itself. The harness clears the task's log roots, re-runs the experiment on the repaired workspace, and grades only that fresh run. No report.md, no rerun and no grading: the task scores 0. A solution task without a run_spec.json (a pure protocol repair, say) skips the rerun but is bound by the same report gate.
| Path | Contents |
|---|---|
tasks/ |
Benchmark tasks, grouped by domain |
autorlbench/ |
Evaluation harness: task loader, validator, runtimes, agent adapters, CLI |
ENVIRONMENTS.md |
The eight fixed shared environment images every task runs in |
skills/package-task/ |
Agent skill that turns a raw task into a task |
tests/ |
Harness unit tests |
page/ |
Website source, deployed to GitHub Pages |
tasks/<domain>/<task_id>/
├── README.md # summary, difficulty, reference solution, verification
├── task.toml # metadata and environment image
├── workspace/ # the run as the agent sees it: code, config, logs, metrics
├── environment/prepare.sh # fetches pinned models and datasets (only stage with network)
└── evaluations/
├── issue/ # instruction.md · oracle.md · verifier/
└── solution/ # instruction.md · oracle/ · verifier/ (grader_spec.json · run_spec.json)
# Validate a task
python3 -m autorlbench validate tasks/reward/slime_retool_length_reward_bonus
# Rerun a task's experiment on its original workspace (prepare + run_spec.json,
# no agent, no verifier) — for task submitters to confirm their environment works
python3 -m autorlbench rerun tasks/reward/slime_retool_length_reward_bonus
# Run an agent through Harbor (pip install harbor)
python3 -m autorlbench harbor-evaluate tasks/reward --agent codex --model <model>harbor-evaluate records each run under outputs/harbor-evaluate/<task_id>/<agent>-<model>/<timestamp>/: the command as an executable command.sh, the effective Harbor command, a live prepare.log and harbor.log (both tail -f-able while the run is going), and the final report.json. Only stage-start lines go to the terminal; everything else lands in those log files.
rerun is the submitter-side environment check: it runs the task's prepare.sh and then the experiment its run_spec.json declares, on the workspace exactly as the task author left it — no agent, no verifier, no report gate. Each run records itself under outputs/rerun/<task_id>/<timestamp>/ in the same layout as harbor-evaluate: the command as an executable command.sh, live tail -f-able prepare.log and rerun.log, the final report.json, the experiment's logs under results/, and the workspace copy. passed in the report means both stages ran to completion, while the experiment's own exit code is recorded but not judged, since a buggy run is expected to fail.
Run python3 -m autorlbench --help for every option. The issue verifier calls an OpenAI-compatible judge through OPENAI_BASE_URL and OPENAI_API_KEY.
We welcome both problematic runs and healthy reference runs. Bring the issue materials, or add the repaired run when it exists; an agent does the packaging — inside a clone of this repository, using the package-task skill. Every task runs in one of the eight fixed shared images in ENVIRONMENTS.md: run your buggy run in one of them and ship that run's artifacts — a defect that reproduces in none of the eight may submit its own environment configuration with an explanation instead. Contribute two valid tasks and you become a co-author on the benchmark paper. See CONTRIBUTING.md and the contribution guide.
Apache-2.0. Each task's workspace keeps the license of its upstream code.

