AI evaluation integrity.
We break the benchmarks, judges and scoring pipelines AI is measured with, and attest the ones that hold.

Authensor is a security research firm in San Francisco, founded by John Daniel Kearney. Most repositories on this profile are forks kept for upstream pull requests; the work lives in etb-scan, etb-harness and the repositories listed below. It audits the instruments the AI industry measures itself with: evaluation harnesses, benchmarks, LLM judges, leaderboards and scoring pipelines.

Every defect we report is proven with a running reproduction against the genuine code at a pinned commit, and disclosed to the maintainer first. Fixed-scope audits end in an attestation with a scope, a standard version and an expiry date.

• The Evaluator Trust Boundary (ETB-01 to ETB-10). An evaluator has an ETB failure when its score depends on an artifact the evaluated system can influence. The page lists dated cases with upstream fixes in SWE-bench, Promptfoo, NVIDIA garak and the UK AI Security Institute's Inspect and ControlArena. Preprint: 10.5281/zenodo.21633620.
• etb-scan, an offline scanner that measures ETB-01 (injected verdicts) against your own judge. Standard library only, no API keys, MIT.

pip install etb-scan
etbscan --judge mypkg:judge --trials 10

• etb-harness, which measures how gameable an agent benchmark is before and after hardening: deterministic capability probes (can the agent read the tests, reach the fix commit, forge a pass), opt-in LLM attacker policies, a verifier and hardening profiles. First targets: SWE-bench Verified, ControlArena. Apache-2.0. Results are published only after the maintainers have been notified.

• The ETB registry, one OSV-format record per confirmed evaluation-integrity flaw (ETB-YYYY-NNNN), with the disclosure policy and the full taxonomy (ETB-01 to ETB-10 published, ETB-11 to ETB-16 proposed). Opens with the first published entries.

• The ledger: every defect filed, every fix that landed upstream, and every target read and found sound. Includes the MLflow CVE-2026-64849 credit.

Evaluation-integrity audits, and a free claim check: one published number checked against the instrument that produced it, within 48 hours.

authensor.com · john@authensor.com

This repository first held Authensor's open-source agent-safety toolkit: the policy engine, the Aegis content scanner, the Sentinel monitor, SafeClaw and the @authensor npm packages. That product line is retired and no longer maintained. Its code and documentation stay here for reference:

Quickstart · Features · Red teaming · API reference · Deployment · Development

MIT.