Repository navigation
Add model-level leaderboards + Arena.ai + refresh Terminal Bench top-3 - #52
Merged
Merged
Conversation
Adds three categories of benchmark coverage the page was missing:
- Model-level leaderboards section: Chatbot Arena (human-preference Elo),
Artificial Analysis (cost/speed/intelligence Pareto), ARC Prize / ARC-AGI
(reasoning generalization). Framed as "pick a model substrate before
picking an agent benchmark" — these rank LLMs, not harnesses.
- Arena.ai Agent Leaderboard added as the live-traffic agent benchmark
("agent Chatbot Arena") in the Other benchmarks table — 889K+ sessions,
28 models, multi-dimensional scoring (tool reliability, task completion,
steerability, bash recovery, hallucination).
- Refreshes Terminal Bench 2.0 top-3 with current June 2026 numbers
(NexAU-AHE + GPT-5.5 84.7%, LemonHarness 84.5%, Capy + GPT-5.5 83.1%);
drops the stale "ForgeCode #1 and #3" insight that no longer holds.
- openlm.ai/swe-bench added as an alt-view link inside the SWE-bench
section header (side-by-side standard + Verified + IOI scores with
explicit harness column).
- τ-Bench section gets the canonical pass^8 < 25% retail number from the
arXiv 2406.12045 paper — single concrete consistency-collapse stat the
section was missing.
- Choosing-a-benchmark table picks up Arena.ai and a new "picking your
base model" row pointing at the three model-level leaderboards.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C9gzSkgNa3yNtob4gBeeit
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds model-level leaderboard coverage (the page only had agent benchmarks), refreshes stale Terminal Bench top-3 with current June 2026 numbers, and pulls in the Arena.ai Agent Leaderboard — the live-traffic agent benchmark that the field has been missing on this page.
What changed in
content/benchmarks.mdNew "Model-level leaderboards" section (between τ-Bench and "Other benchmarks worth knowing") — framed as "pick a model substrate before picking an agent benchmark."
Refreshed Terminal Bench 2.0 top performers with live data:
The "ForgeCode #1 and #3" insight no longer holds; replaced with the GPT-5.5 #1-and-#3 framing.
Added Arena.ai Agent Leaderboard as a row in "Other benchmarks worth knowing" — 889K+ sessions across 28 models, scoring on tool reliability, task completion, steerability, bash recovery, and tool hallucination. Closest analog to "agent Chatbot Arena."
SWE-bench section —
openlm.ai/swe-benchadded as an alt-view link in the header (side-by-side standard + Verified + IOI competition scores with explicit harness column).τ-Bench section — adds the canonical Pass^8 < 25% in retail number from the original arXiv 2406.12045 paper. Single concrete stat the page was missing on the consistency-collapse claim.
Choosing-a-benchmark table — adds Arena.ai row + a new "Picking your base model" row pointing at the three model-level leaderboards.
URLs covered
All 7 user-requested benchmark URLs are now referenced on the page:
build.shsucceeds; rebuiltindex.htmlincluded.Test plan
🤖 Generated with Claude Code
https://claude.ai/code/session_01C9gzSkgNa3yNtob4gBeeit
Generated by Claude Code