Skip to content

About

A benchmark and open call for valuable, reproducible RL runs.

Topics

Resources

Contributing

Stars

26 stars

Watchers

0 watching

Forks

Repository files navigation

AutoRLBench

Website Leaderboard Tasks Contribute License Chinese (Simplified)

A benchmark and open call for valuable, reproducible RL runs. Each task is a real training run that went wrong. The agent has to find the root cause (issue) and, where a repaired run exists, fix it and rerun (solution).

AI Trains AI

🚀 News

  • 6 Nov 2026 — We release the second version of the task list: 32 tasks, 24 of which support solution-level evaluation.
  • 2 Nov 2026 — We release the first version of the task list: 18 tasks, 13 of which support solution-level evaluation.
  • 24 Oct 2026 — Call for contributions officially open, and Kimi-K3.1-Pro added to the leaderboard.
  • 19 Oct 2026 — GitHub repo and website released.

Leaderboard

AutoRLBench Leaderboard, issue + solution evaluation. Overall is the combined pass rate across all evaluated runs (issue + solution): passes over total runs, not an average of the two levels.

# Model Agent Issue Score Solution Score Overall
1 GPT-6-Astra Codex 33.3 % 38.9 % 35.2 %
2 GPT-6.1-Sol Codex 30.6 % 38.9 % 33.3 %
3 Gemini-4-Argon Claude Code 16.7 % 55.6 % 29.6 %
4 GLM-5.3 Claude Code 22.2 % 33.3 % 25.9 %
5 Kimi-K3.1-Pro Claude Code 16.7 % 38.9 % 24.1 %
6 Claude-Fable-5.1 Claude Code 16.7 % 27.8 % 20.4 %
7 Qwen3.8-Max Qwen Code 13.9 % 27.8 % 18.5 %
8 Grok-4.7 Grok Build 16.7 % 16.7 % 16.7 %
9 Kimi-K3 Claude Code 0.0 % 16.7 % 5.6 %
10 DeepSeek-V4-Pro Claude Code 0.0 % 5.6 % 1.9 %

Tasks

Six domains, one representative task each. The domains come from where in the RL pipeline the failure lives:

Configuration Configuration · issue
TRL GSM8K EOS mismatch
Missing <|im_end|> in eos_token_id lets rollouts continue past the answer.
Reward Reward · issue · solution
Length bonus reward hack
An unconditional length bonus teaches the policy to write longer, not better.
Data Data · issue
Training data is too hard
Train is TACO VERY_HARD while validation is EASY; held-out accuracy ends below its start.
Algorithm Algorithm · issue
GRPO advantage broadcast
An advantage broadcast replaces per-sample advantages, causing policy collapse.
Data Data · issue
Training data too hard
Training on VERY_HARD while validating on EASY, held-out accuracy falls below the starting point.
Infrastructure Infrastructure · issue · solution
MoE auxiliary gradients miss token normalization
The per-token loss callback omits the token count, so MoE z-loss gradients are never normalized.

How a task works

prepare ──▶ agent ──▶ [rerun] ──▶ verifier
  │           │          │            │
  │           │          │            └─ issue:    LLM judge on agent_answer.md vs. hidden oracle
  │           │          │               solution: a metric threshold on the rerun's logs
  │           │          └─ solution only: the harness re-runs the experiment on the agent's
  │           │             workspace, per the task's run_spec.json
  │           └─ reads instruction.md and a writable copy of workspace/
  └─ fetches pinned models and datasets (the only stage with network)

Each stage runs in its own container from the same prebuilt image. Oracles and verifiers are mounted only into the verifier stage.

At the solution level the agent repairs the workspace and submits report.md — it does not launch training itself. The harness clears the task's log roots, re-runs the experiment on the repaired workspace, and grades only that fresh run. No report.md, no rerun and no grading: the task scores 0. A solution task without a run_spec.json (a pure protocol repair, say) skips the rerun but is bound by the same report gate.

Repository layout

Path Contents
tasks/ Benchmark tasks, grouped by domain
autorlbench/ Evaluation harness: task loader, validator, runtimes, agent adapters, CLI
ENVIRONMENTS.md The eight fixed shared environment images every task runs in
skills/package-task/ Agent skill that turns a raw task into a task
tests/ Harness unit tests
page/ Website source, deployed to GitHub Pages

Task layout

tasks/<domain>/<task_id>/
├── README.md               # summary, difficulty, reference solution, verification
├── task.toml               # metadata and environment image
├── workspace/              # the run as the agent sees it: code, config, logs, metrics
├── environment/prepare.sh  # fetches pinned models and datasets (only stage with network)
└── evaluations/
    ├── issue/              # instruction.md · oracle.md · verifier/
    └── solution/           # instruction.md · oracle/ · verifier/ (grader_spec.json · run_spec.json)

Quick start

# Validate a task
python3 -m autorlbench validate tasks/reward/slime_retool_length_reward_bonus

# Rerun a task's experiment on its original workspace (prepare + run_spec.json,
# no agent, no verifier) — for task submitters to confirm their environment works
python3 -m autorlbench rerun tasks/reward/slime_retool_length_reward_bonus

# Run an agent through Harbor (pip install harbor)
python3 -m autorlbench harbor-evaluate tasks/reward --agent codex --model <model>

harbor-evaluate records each run under outputs/harbor-evaluate/<task_id>/<agent>-<model>/<timestamp>/: the command as an executable command.sh, the effective Harbor command, a live prepare.log and harbor.log (both tail -f-able while the run is going), and the final report.json. Only stage-start lines go to the terminal; everything else lands in those log files.

rerun is the submitter-side environment check: it runs the task's prepare.sh and then the experiment its run_spec.json declares, on the workspace exactly as the task author left it — no agent, no verifier, no report gate. Each run records itself under outputs/rerun/<task_id>/<timestamp>/ in the same layout as harbor-evaluate: the command as an executable command.sh, live tail -f-able prepare.log and rerun.log, the final report.json, the experiment's logs under results/, and the workspace copy. passed in the report means both stages ran to completion, while the experiment's own exit code is recorded but not judged, since a buggy run is expected to fail.

Run python3 -m autorlbench --help for every option. The issue verifier calls an OpenAI-compatible judge through OPENAI_BASE_URL and OPENAI_API_KEY.

Contributing

We welcome both problematic runs and healthy reference runs. Bring the issue materials, or add the repaired run when it exists; an agent does the packaging — inside a clone of this repository, using the package-task skill. Every task runs in one of the eight fixed shared images in ENVIRONMENTS.md: run your buggy run in one of them and ship that run's artifacts — a defect that reproduces in none of the eight may submit its own environment configuration with an explanation instead. Contribute two valid tasks and you become a co-author on the benchmark paper. See CONTRIBUTING.md and the contribution guide.

License

Apache-2.0. Each task's workspace keeps the license of its upstream code.

About

A benchmark and open call for valuable, reproducible RL runs.

Topics

Resources

Contributing

Stars

26 stars

Watchers

0 watching

Forks

Contributors

Languages