semantic-test-matcher is a TypeScript CLI for semantic test matching. It exposes the rbt command, which ranks likely test files for a changed source file, inspects resolved runtime configuration, and prints shell completion scripts.
The matching flow combines:
- document profiling from file paths, code structure, and diffs
- TypeSafe's Jev System One model, which judges whether each candidate test file should re-run for the change
- a local answer cache
- optional OpenAI Decisions ranking, selected with
--ranker decisions - score blending for model and structural signals
rbt match <file>ranks candidate tests for a changed filerbt benchmarkscores the matcher against a file of expected rankingsrbt statusshows the resolved runtime configurationrbt completion [bash|zsh]prints a shell completion script- scores candidates together in provider requests, batching larger suites
- falls back to local structural heuristics when no API key is available
- caches provider answers separately in
.rbt/cacheby default - accepts candidate file lists from CLI flags, config, or stdin
flowchart TD
A["CLI entry<br/>src/cli.ts"] --> B["Commander program<br/>global options + subcommands"]
B --> C["Command layer<br/>match / benchmark / status / completion"]
C --> D["Config resolution<br/>src/config.ts"]
D --> D1["Sources<br/>CLI flags -> env vars -> config file -> defaults"]
C --> E["Input handling"]
E --> E1["Changed file / diff / stdin"]
E --> E2["Candidate discovery<br/>src/utils/files.ts"]
E1 --> F["Document profiling<br/>src/services/document-profile.ts"]
E2 --> F2["Candidate profiles"]
F --> G["Provider scorer<br/>jev.ts / decisions.ts"]
F2 --> G
G --> H["Answer cache<br/>src/services/cache.ts"]
G --> I["TypeSafe API<br/>POST /v1/systemone"]
G --> J["OpenAI API (opt-in)<br/>POST /v1/decisions"]
G --> K["Ranking engine<br/>src/services/match.ts"]
F --> K
F2 --> K
K --> L["Scoring blend"]
L --> L1["Model probability (60%)"]
L --> L2["Change, phrase, and anchor overlap"]
L --> L3["Semantic token and interface overlap"]
L --> L4["Path family and basename overlap"]
C --> M["Output"]
K --> M
D --> M
M --> M1["Human-readable CLI output"]
M --> M2["JSON output for automation"]
- Node.js 22.12 or newer
- a TypeSafe API key from console.typesafe.ai/keys in
TYPESAFE_API_KEY(without one,rbtranks with local heuristics only)
npm install --global semantic-test-matcher
export TYPESAFE_API_KEY=...Verify the installation:
rbt --version
rbt statusMatch a changed file to likely tests, passing the change as a diff:
git diff > change.diff
rbt match src/price-engine.ts --candidates tests --diff-file change.diff --jsonInspect resolved settings:
rbt status --jsonPrint shell completion:
rbt completion zshRanks likely candidate files for one or more changed source files and merges the selections; a test picked for several files keeps its best score.
With no file arguments or --diff-file, rbt match detects local Git changes under the current directory (staged, unstaged, deleted, and non-ignored untracked JS/TS files). It shows the selected tests, then asks which test command to run, pre-filled from your test script when that script runs Vitest, Jest, Playwright, or Mocha with options and their values only (a script that already names test paths, such as jest tests or mocha --spec ..., would run them too, so the plain runner is suggested instead). Enter a command such as npx playwright test, npx vitest run, or node --test; press Enter to skip. Selected absolute paths are appended as separate arguments, and RBT returns the runner's exit code. The command accepts quoted arguments, but does not interpret shell operators, variable expansion, or pipelines. Use an executable or wrapper script that accepts test paths as trailing arguments.
This is a one-shot flow, not a watcher. A clean tree or empty selection runs nothing. A changed file that resolves outside the repository, such as an untracked symlink, is skipped with a warning. Because it runs what it selects, this mode only considers test-like candidates: .test/.spec and Deno _test files, and any other file outside fixture directories that declares tests (with test(...), it(...), describe(...), or Deno.test(...)). Helpers and fixtures under test directories are not run. --json and --paths-only only report selections and never prompt or execute tests; use these modes without an interactive terminal. Explicit files and --diff-file retain their selection-only behavior. Committed branch changes still require a supplied diff.
Examples:
# Detect local changes, select tests, and ask which test command to run
rbt match --candidates tests
rbt match prompts-idea/src/price-engine.ts --candidates prompts-idea/tests
git diff main > pr.diff
rbt match --diff-file pr.diff --candidates tests
# One path per line; xargs -0 keeps spaces in paths and runs nothing for an empty selection
rbt match --diff-file pr.diff --paths-only | tr '\n' '\0' | xargs -0 -r npx playwright test
cat prompts-idea/candidate-list.txt | \
rbt match prompts-idea/src/price-engine.ts \
--candidates-from-stdin \
--top-k 4 \
--threshold 0.35 \
--jsonUseful flags:
--threshold <number>--min-score <number>--top-k <number>(optional cap on selected files)--selection-policy <adaptive|conservative|targeted>--candidates <path>(repeat for several, e.g.--candidates tests --candidates e2e)--include-file <glob>(repeatable)--exclude-file <glob>(repeatable)--candidates-from-stdin--ranker <jev|decisions|heuristics>(openaiis an alias fordecisions)--jev-model <id>--decisions-model <id>--fallback-ranker <heuristics|decisions>(Decisions fallback requires Jev as primary)--cache-dir <path>--diff-file <path>--diff-root <path>(set the base for relative diff paths, such as.forgit diff --relative)--json--paths-only(print only the selected test paths, one per line)
A deleted file can still be matched: pass a --diff-file that contains its deletion. Every path a --diff-file changes must stay inside its diff root (--diff-root, else the Git root); a diff that names a file outside it is rejected. Candidate files larger than 1 MB are skipped with a warning. In text output, a Why: line explains fallback selections, such as widening coverage when neither the model nor the structural score is confident.
How matching works:
- The changed file (and its hunks from
--diff-file) is read and converted into aDocumentProfile. - Candidate files are collected from configured paths or stdin.
- The chosen model asks one yes/no question per candidate, "should the tests in this file be re-run to check this change?", and returns a probability. All candidates go in one request; larger suites are batched.
rankMatchesblends the Model probability (60%) with structural overlap (40%). With--ranker heuristics, or when configured providers are unavailable, the structural score is used alone.- Results are filtered by the configured minimum score. The default
adaptivepolicy selects every model-affirmative candidate (probability at least 0.5) plus structurally strong neighbors: candidates in the top structural decile with structural score at least 0.2. When no model score reaches 0.7, it widens the structural band to the top quartile; if structural evidence is also weak, it retains all candidates. Without model evidence, the wider structural rule applies. These are selection heuristics, not inferred test dependencies or calibrated probabilities. A shared-code change can therefore select more files than a localized change.
There is no default count cap for adaptive. --top-k (or its config/environment equivalent) limits output after eligibility is computed and may exclude useful tests; JSON reports eligibleCount, selectionLimit, selectionTruncated, and selectionEvidence. Explicit conservative retains the blended top five by default. Explicit targeted retains up to five model-affirmative candidates by default and falls back to conservative selection when the model is unavailable or has no affirmative answers. Both legacy policies also honor an explicit --top-k. The fixed bands above are uncalibrated. Decisions uses the same provisional blend and boundaries; these numbers do not establish equivalent calibration or ranking quality.
- Data leaves your machine. Each request sends the changed file's path, exported symbol names, and its diff hunks (or, without
--diff-file, the first 6,000 characters of the file; with--diff-file, source text is never sent, even for a file the diff has no text hunks for), plus each candidate test file's path and test titles, toapi.typesafe.ai. Candidate file bodies are not sent. Review TypeSafe's data handling before using it on private code. Use--ranker heuristicsto keep everything local. - Pass a diff. Jev is most useful with
--diff-file, because it can then judge the actual change rather than the whole file. - Fallback. If
TYPESAFE_API_KEY(orJEF) is missing or the API fails after retries,matchprints a warning to stderr and ranks with heuristics only. JSON output reports the effectiverankerand arankerFallbackreason, so CI can detect the downgrade. When Jev answers, JSON also lists every model version whose answers were used injev.models.benchmarkfails instead of falling back. - Caching and model pinning. Answers are cached per change and candidate in
<cacheDir>/jev.json, so repeat runs make no requests. The default model is pinned tojev-1.13.0.jev-latestalso works, but its answers can change when TypeSafe ships a new version, so they are never cached: only answers from the exact model requested are reused. - Cost. Jev bills input tokens only, at $0.042 per million. A change with ~30 candidates is roughly 2,000–7,000 tokens.
Jev remains the default. Setting an OpenAI key does not enable OpenAI requests; select Decisions or configure the fallback explicitly:
rbt match src/example.ts --ranker decisions
rbt match src/example.ts --ranker jev --fallback-ranker decisions
rbt match src/example.ts --ranker heuristicsDecisions sends the same bounded change evidence and candidate paths/test titles described above to https://api.openai.com/v1/decisions. It does not send candidate file bodies. The default model is gpt-6-luna, configurable with --decisions-model. Source text is omitted when a diff is supplied. Source/diff excerpts and test titles are untrusted evidence, not instructions to execute.
Direct Decisions runs fall back to local heuristics after an unavailable provider. The opt-in Jev fallback tries Jev, then Decisions, then heuristics. If any source fails, every source is rescored with the next provider before selection; one output never mixes providers. Valid low probabilities do not trigger provider fallback. Refusals and invalid or incomplete answers do.
An explicit --ranker heuristics stays local even if a Decisions fallback is inherited from the environment or config file. An explicit --fallback-ranker decisions with a non-Jev primary is still rejected.
JSON retains ranker as the effective provider and adds requestedRanker, effectiveRanker, rankerAttempts (sanitized reasons), and modelScorer (provider, model identities, request/cache counts, and reported usage). Candidate results carry modelScore; jevScore and the jev metadata object remain available only for Jev results. Missing token usage is omitted rather than estimated. modelIdentity: "requested" means the API omitted its answering-model identity; it does not prove an immutable version answered.
cacheEntries counts stored entries for the last attempted remote provider, including completed answers saved before a fallback to heuristics. A local-only run retains the existing Jev cache-count behavior.
For example, an explicitly configured fallback that succeeds with Decisions reports:
{
"requestedRanker": "jev",
"effectiveRanker": "decisions",
"ranker": "decisions",
"rankerAttempts": [
{ "ranker": "jev", "status": "failed", "reason": "Jev request failed (HTTP 401)" },
{ "ranker": "decisions", "status": "succeeded" }
],
"modelScorer": { "provider": "decisions", "requests": 1, "cacheHits": 0, "models": ["gpt-6-luna"] }
}Provider requests have bounded batches, timeouts, and retry counts; a multi-file run can still require several requests. Decisions answers use a separate decisions.json cache with a 24-hour lifetime, keyed by model, evidence, question, and prompt version. Answers whose reported model differs from the requested id, or has no reported model id, are not cached. No Decisions latency or ranking-quality advantage over Jev is assumed.
Decisions honors Retry-After within its two-second local retry-delay limit. Longer server-requested delays stop the attempt rather than trigger an early retry; match then follows its configured fallback, while benchmark fails.
Runs the matcher over a JSON file of cases (source, optional diffText, and expectedTop1, expectedTop3, or expectedTop10Includes) and reports hit rates. It takes the same --ranker, --jev-model, --decisions-model, candidate, and threshold flags as match. Benchmark reports ranking hit rates; it does not apply match selection policies. With a remote provider, it also reports request counts, cache hits, input tokens, and every model version that answered, so runs against moving aliases stay attributable. JSON also includes each case's full observedRanking. Benchmarking is strict: missing credentials, invalid answers, refusals, or API failures fail the command; --fallback-ranker decisions is rejected. Run each provider separately for a comparison. Benchmarks report rankings, not adaptive selection recall.
rbt benchmark --cases cases.json --candidates tests --json
rbt benchmark --cases cases.json --candidates tests --ranker heuristics
rbt benchmark --cases cases.json --candidates tests --ranker decisionsSee the Playwright benchmark summary for measured results, regression scope, and reproduction commands.
Prints the resolved runtime configuration, whether TYPESAFE_API_KEY is set, and cache stats.
rbt status
rbt status --jsonPrints a bash or zsh completion script.
rbt completion bash
rbt completion zshThe CLI resolves settings in this order:
- command flags
- environment variables
- config file
- built-in defaults
Config files are loaded from:
--config <path>if provided.rbt/config.json.rbtconfig
Example config:
{
"ranker": "jev",
"jevModel": "jev-1.13.0",
"decisionsModel": "gpt-6-luna",
"fallbackRanker": "heuristics",
"cacheDir": ".rbt/cache",
"logLevel": "info",
"match": {
"threshold": 0,
"selectionPolicy": "adaptive",
"candidatePaths": ["test", "tests"],
"includePatterns": ["**/*"],
"excludePatterns": [
"**/dist/**",
"**/.git/**",
"**/node_modules/**",
"**/build/**"
]
}
}Environment variables used by the resolver include:
RBT_RANKERRBT_JEV_MODELRBT_DECISIONS_MODELRBT_FALLBACK_RANKERRBT_CACHE_DIRRBT_LOG_LEVELRBT_VERBOSERBT_QUIETRBT_TOP_KRBT_MATCH_TOP_KRBT_THRESHOLDRBT_MATCH_THRESHOLDRBT_MIN_SCORERBT_MATCH_MIN_SCORERBT_SELECTION_POLICY
A variable set to an empty string counts as unset, so config and defaults still apply.
TYPESAFE_API_KEY supplies the Jev key, with JEF accepted as an alias. OPENAI_API_KEY supplies the OpenAI key, with OPEN_AI accepted as an alias. Canonical names take precedence. Keys are read only from the environment, never from config files, and status prints only set or missing. All four names, including case variants, are removed from subprocess environments when running selected tests.
The CLI does not automatically load .env. Load an ignored local file explicitly with Node when running from source:
node --env-file=.env --experimental-strip-types src/cli.ts match src/example.ts --ranker decisionsKeep credentials out of tracked files. --ranker heuristics uses neither API regardless of credentials.
- Jev answers are cached in
.rbt/cache/jev.jsonby default, keyed by model, change, and candidate - Decisions answers use
.rbt/cache/decisions.json; cache entries are never shared between providers - cache writes are best-effort and do not fail the command if they break
statusreports both provider cache counts
src/
cli.ts CLI entrypoint
commands/ Commander subcommands
services/ Provider scoring, ranking, document profiling, cache
utils/ candidate collection, stdin helpers, glob matching
prompts-idea/
src/ synthetic source files for matching experiments
tests/ synthetic tests used as candidates
candidate-list.txt sample stdin input for --candidates-from-stdin
README.md dataset-specific usage notes
prompts-idea/ is a small synthetic workspace for exercising the matcher. It includes source files, related and unrelated tests, and a candidate list file for stdin-driven matching flows.
Useful commands:
npm test
rbt match prompts-idea/src/price-engine.ts \
--candidates prompts-idea/tests \
--json
cat prompts-idea/candidate-list.txt | \
rbt match prompts-idea/src/price-engine.ts \
--candidates-from-stdin \
--jsonFor dataset-specific notes, see prompts-idea/README.md.
npm install
npm run lint
npm test
npm run buildThe repo currently uses node:test for tests and TypeScript for type-checking and build output. Tests stub the TypeSafe API and never make network calls.