Hugging Face • arXiv • BIGAI • Humanlaya • xbench
As AI learns to gather the scattered sixpence, human gains the liberty to look up at the moon.
A rubric-based automated evaluation system for language agents capabilities on economically conceptual tasks across professional domains. It evaluates 50+ models or agent systems from 6 API-based providers with weighted binary grading, async concurrency, cost tracking, and Excel/JSON reporting.
Contact: liuyang@bigai.ai
- [2026.09.28] $OMB are now available to evaluate with EvalScope.
- [2026.09.26] $OMB has been accepted by NeurIPS 2026 ✨. See you in 🇺🇸 Atlanta!
- [2026.08.28] $OMB has been adopted by Tencent Hy4-preview.
- [2026.08.03] $OMB has been adopted by Alibaba Qwen3.8-Max.
- [2026.06.23] $OMB has been adopted by ByteDance Doubao-Seed-2.1.
- [2026.03.10] $OMB has been widely reported by 机器之心 and launched in xbench.
Table of contents
- Overview
- Dataset
- Installation
- Quick start
- Using EvalScope
- Task format
- Output format
- Frequently asked questions
- License
- Citation
We maintains the $OneMillion-Bench evaluation toolkit and the latest leaderboard performance is available here.
$OneMillion-Bench is a CLI tool (omb) that generates LLM agent responses, grades them against weighted rubrics using judge models, and produces reports.
- 50+ models across 6 providers (OpenRouter, Qwen/DashScope, VolcEngine, Hunyuan, Ling-1T, LiteLLM)
- Concurrent async processing with up to 128 parallel requests
- Weighted rubric scoring with binary (yes/no) evaluation
- Repeated sampling & judging for variance estimation
- Web search augmentation for search-enabled generation
- Automatic cost tracking per model with token usage and pricing
- Rich reporting: Gruvbox-themed Excel workbooks, JSON summaries
5 professional domains, bilingual (Chinese + English), 400 questions:
| Domain | Directory | CN | EN | Total | Description |
|---|---|---|---|---|---|
| Healthcare & Medicine | healthcare_and_medicine |
40 | 40 | 80 | Clinical medicine, oncology, pharma, gene & cell therapy |
| Economics & Finance | economics_and_finance |
40 | 40 | 80 | Investment, equities, financial analysis |
| Industry | industry |
40 | 40 | 80 | Systems engineering, embedded systems, robotics |
| Law | law |
40 | 40 | 80 | Corporate/commercial law, M&A |
| Natural Science | natural_science |
40 | 40 | 80 | Chemistry, materials science |
Each test case contains a prompt, optional system prompt, domain tags, and 5-23 weighted rubrics labeled as: Factual Information, Analytical Reasoning, Instructions Following, or Structure and Formatting.
Requires Python 3.10+. conda recommended.
conda create -n evals python=3.10 -y && conda activate evals
git clone https://github.com/humanlaya/OneMillion-Bench.git
cd OneMillion-Bench
pip install -e .
omb --version # verify installationmkdir datasets
hf download humanlaya-data-lab/OneMillion-Bench --repo-type=dataset --local-dir datasets/OneMillion-Benchexport OPENROUTER_API_KEY="your-key" # Required (recommended for 50+ models)
export DASHSCOPE_API_KEY="your-key" # Optional: Qwen / DashScope
export VOLC_API_KEY="your-key" # Optional: VolcEngine (Doubao)
export HUNYUAN_API_KEY="your-key" # Optional: Tencent Hunyuan
export LING_API_KEY="your-key" # Optional: Ling-1T
export LITELLM_API_KEY="your-key" # Optional: LiteLLMp.s. add these to ~/.bashrc or ~/.zshrc to persist across sessions.
cp src/omb/config/default.yaml my_config.yamlEdit my_config.yaml to select models (API keys are read from environment variables, not the config file):
JUDGE_MODELS:
- "google/gemini-3-pro-preview"
GENERATOR_MODELS:
- "openai/gpt-5.4"
- "anthropic/claude-opus-4.6"
- "google/gemini-3.1-pro-preview"omb eval --dataset datasets/OneMillion-Bench/healthcare_and_medicine --config my_config.yaml # single domain
omb eval --dataset datasets/OneMillion-Bench --config my_config.yaml --recursive # all domains
omb eval --dataset datasets/OneMillion-Bench/law --config my_config.yaml --grade-only # grade onlypython examples/auto_grading.py --dataset datasets/OneMillion-Bench/healthcare_and_medicine --config my_config.yamlResults are written to outputs/result_YYYYMMDD_HHMMSS/.
EvalScope now supports $OneMillion-Bench, providing a standardized evaluation workflow with OpenAI-compatible model endpoints, saved predictions, scoring, and report generation.
This may be useful for users who want a quick path to evaluate hosted or locally served models.
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key YOUR_API_KEY \
--datasets one_million_bench \
--limit 10For detailed setup, configuration, and usage instructions, see the EvalScope $OneMillion-Bench documentation.
Each test case is a JSON file:
| Field | Type | Description |
|---|---|---|
prompt |
string | Question or instruction |
system_prompt |
string | Optional system prompt |
tags |
list[string] | Domain hierarchy, e.g., ["Healthcare", "Pharma", "Gene Therapy"] |
case_id |
int | Unique test case ID |
rubrics |
list[object] | Evaluation rubrics (see below) |
Each rubric:
| Field | Type | Description |
|---|---|---|
rubric_number |
int | Index within the case |
rubric_detail |
string | Evaluation criterion |
rubric_weight |
int | Score weight (positive to award, negative to penalize) |
rubricLabel |
string | Category label |
Example:
{
"prompt": "Explain the concept of viral titer in gene therapy ...",
"system_prompt": "",
"tags": ["Healthcare", "Pharma", "Gene Therapy"],
"case_id": 2860,
"rubrics": [
{"rubric_number": 1, "rubric_detail": "Clearly defines viral titer and distinguishes infectious vs. physical titer.", "rubric_weight": 10, "rubricLabel": "Analytical Reasoning"},
{"rubric_number": 2, "rubric_detail": "Lists at least 3 common titer measurement methods.", "rubric_weight": 8, "rubricLabel": "Factual Information"}
]
}Results are saved to outputs/result_YYYYMMDD_HHMMSS/:
- Updated JSON files — input files augmented with
model_response,rubric_auto_score,judge_cot, andrubric_auto_vs_humanfields. - grading_results.xlsx — Gruvbox-themed workbook with per-judge scores, aggregate sheets, and cost breakdowns.
- grading_results.json — Machine-readable summary with per-model metrics and costs.
What models are supported? 50+ models through 6 providers. Specified as provider/model (e.g., openai/gpt-5.4). See src/omb/config/default.yaml for the full list.
Can I add a new LLM provider? Yes. Extend BaseLLMClient in src/omb/clients/ and implement the call() method.
Can I use custom rubrics? Yes. Create JSON files following the task format schema.
How does grading work? The judge evaluates each rubric with a binary yes/no judgment and chain-of-thought justification. Scores are weighted by rubric weights. Multiple judges and repeated runs are supported.
How is cost tracked? Token usage is tracked per model per call. Costs are computed from the pricing table in the config and reported in both Excel and JSON outputs.
@inproceedings{
liu2026onemillionbench,
title={\$OneMillion-Bench: How Far are Language Agents from Human Experts?},
author={Liu, Yang and Li, Jiaqi and Bai, Jun and Yang, Qianyu and Hu, Xiaobo and Peng, Tao and Wang, Zaiyuan and Dong, Jiayun and Zhang, Chun and Jia, Zixia and Chen, Kaiyuan and Xiao, Yanglihong and Yin, Lingyue and Duan, Tiliang and Zhang, Ge and Yao, Gang and Chen, Hao and Jiao, Jianpeng and Zheng, Zilong},
booktitle={The Fortieth Annual Conference on Neural Information Processing Systems Evaluations and Datasets Track},
year={2026},
url={https://openreview.net/forum?id=2gAr3pMVHH}
}