A candidate that survives Falsifier is one that could not be killed.
This project is not a biosignature detector. No exoplanet biosignature has ever been confirmed. This claim is immutable. No generated code, comment, or UI copy contradicts it.
"A neural network sees Earth 2.0. Astrophysics sees a background binary star. The James Webb Space Telescope is one of the most expensive scientific instruments humanity has ever built — Falsifier makes sure humanity doesn't point it at an AI hallucination."
Most transit-detection pipelines and AI classification models are built to find planets, inherently suffering from confirmation bias when dealing with noisy deep-space data. This one is built to destroy its own candidates. A dimming light curve is not a discovery until it has survived seven independent attempts to explain it away as something else — and until then, the pipeline is designed to fail it, not flatter it.
▶ Watch the walkthrough — enter a Kepler/TESS target ID, watch every pipeline stage stream live, see the named rejection mechanism fire.
Illustrative scenario — the Falsifier pipeline behavior shown here is genuine and reproducible (run
python3 scripts/judge_memory_moment.py). The depth (1.33%), odd/even mismatch (4.32), and disposition (false_positiveviaodd_even_depth) are pipeline-measured from the committed KIC 6965293 FITS file. The "shape-only classifier" box is an illustrative description of confirmation-biased classifiers generally — no specific CNN was benchmarked against this target in this repository.
Light Curve Dip: 1.33% (pipeline-measured on KIC 6965293) ─────────────────────────┐
▼
┌──────────────────────────────────────────────────────────────────────────────────────┐
│ Illustrative shape-only classifier (not built; comparison only): │
│ "High-confidence transit candidate — queues for expensive follow-up" │
│ (Such classifiers are trained on signal shape; this shape passes.) │
└──────────────────────────────────────────────────────────────────────────────────────┘
│
[ FALSIFIER INTERVENTION ]
│
┌──────────────────────────────────────────────────────────────────────────────────────┐
│ Falsifier Odd/Even Depth Test: │
│ "FAIL — Odd/even transit depth mismatch 4.32 exceeds threshold 3.0" │
│ Physical Reality: Alternating transits of unequal depth — this is an │
│ eclipsing binary star. The candidate is a ghost. │
│ Action: CANDIDATE EXECUTED via odd_even_depth FAIL. │
└──────────────────────────────────────────────────────────────────────────────────────┘
Pipeline-measured values from the committed KIC 6965293 golden FITS file
(scripts/judge_memory_moment.py, source DOI 10.1088/0004-6256/141/3/83).
Judge Memory Moment transit depth (KIC 6965293, pipeline-measured): 1.33%
Judge Memory Moment odd/even mismatch (KIC 6965293, pipeline-measured): 4.32
Judge Memory Moment disposition (KIC 6965293): false_positive
Judge Memory Moment triggering test (KIC 6965293): odd_even_depth
The Core Insight: Generative models and neural networks are trained on shape; space is full of things that look like planets but aren't. A light curve dip can be humanity's next home or an optical illusion from a background eclipsing binary. When AI guesses, space agencies waste resources. Falsifier makes sure space exploration targets physical reality, not statistical flukes.
Advance Space Exploration with AI. (August 2026)
This submission was built for the IBM AI Builders Challenge under the theme of advancing space exploration through AI. Falsifier directly embodies this theme: it applies adversarial AI and astrophysical reasoning to protect humanity's most expensive space assets from wasted telescope time, ensuring that every target queued for deep-space characterization is physically real.
Space exploration does not begin at the launch pad; it begins at the targeting queue.
NASA's upcoming Habitable Worlds Observatory (HWO), the Roman Space Telescope, and future direct-imaging interstellar precursor probes cannot be redirected on a whim. Their cryogenic propellants, station-keeping fuel, and observation windows are finite.
If humanity relies on standard AI pipelines that optimize for discovery volume, we will point multi-billion-dollar space assets at ghosts. Falsifier directly advances space exploration by serving as the autonomous pre-flight targeting firewall: ensuring that every target queued for deep-space characterization has survived every physical falsification test known to astrophysics.
No cloud accounts. No API keys. No complex database provisioning. The deterministic core is offline-first and runs in seconds, making it immediately deployable for resource-constrained observatories:
pip install -e ".[dev]" && pytest tests/test_known_eb_rejected.pyFalsifier introduces Adversarial Epistemology to AI-driven space exploration. It solves the "black box" trust issue in AI by entirely inverting the paradigm: it operates as an autonomous adversarial attacker against its own data. It does not use AI to guess if a planet is real; it uses an automated pipeline to aggressively prove it is a false positive.
A real planetary transit has to survive tests built specifically to expose the ways a transit signal lies:
adversarial_selftest.py runs the pipeline against four categories of null data whose only purpose is to make it fail.
injection_recovery.py exists to find the exact point where detection breaks down.
Every mutation gate recorded in docs/PROVEN_GATES.md is there to confirm that a guard fails when it should.
A single FAIL returned by falsifier/pipeline/contracts/vet.py kills a candidate outright. There is no appeal, no averaging away a bad test against five good ones.
Pipeline-measured results — all values in this section are produced by
scripts/differentiator_proof.pyrunning the real Falsifier vetting pipeline against two committed FITS files. There is no composed or dramatized data here: depths, odd/even mismatch scores, and dispositions are genuine pipeline outputs. The "standard CNN classifier" comparison is an illustrative description of confirmation-biased pipelines generally — no specific CNN model was benchmarked in this repository. CI-enforced:pytest tests/test_differentiator_proof.pyasserts opposite dispositions.
Deep learning models operate on pixel patterns; Falsifier operates on astrophysics. Run the differentiator script against two golden targets whose pipeline dispositions are physically opposite:
python3 scripts/differentiator_proof.pyExpected output:
Target A Target B
--------------------------------------------------------------------------
KIC ID KIC 11904151 KIC 6965293
Transit depth (ppm) 176 13315
Transit depth (%) 0.02 1.33
Odd/even mismatch 1.19 4.32
Disposition ambiguous false_positive
Triggering test centroid_shift odd_even_depth
Target A (KIC 11904151, Kepler-10) has a shallow
Differentiator Proof Target A depth (KIC 11904151): 0.02%
dip and returns disposition
Differentiator Proof Target A disposition (KIC 11904151): ambiguous
(centroid data absent in the golden fixture — no FAIL fires, pipeline correctly flags uncertainty). Target B (KIC 6965293) has a
Differentiator Proof Target B depth (KIC 6965293): 1.33%
dip and returns disposition
Differentiator Proof Target B disposition (KIC 6965293): false_positive
via the
Differentiator Proof Target B triggering test (KIC 6965293): odd_even_depth
gate. Identical AI classification inputs. Completely opposite physical realities. The AI guesses; Falsifier physically checks.
Every candidate is run through seven independent vetting tests (falsifier/pipeline/contracts/vet.py, VETTING_TEST_ORDER):
| Test | What it's built to catch |
|---|---|
odd_even_depth |
Eclipsing binaries — alternating transits of unequal depth, where a real planet's transits stay equal |
secondary_eclipse |
A hidden second star, revealed by a dip at phase 0.5 that a lone planet cannot produce |
centroid_shift |
A contaminating background star, revealed by the brightness centroid moving during the dip rather than staying fixed |
transit_shape |
A transit depth outside the physically plausible planetary regime |
stellar_density |
A transit shape inconsistent with the host star's known density |
gaia_ruwe |
An unresolved companion star, flagged via Gaia's astrometric noise excess |
systematics_coincidence |
Instrumental artifacts that line up with spacecraft roll manoeuvres rather than orbital phase |
NASA's TESS mission generates a high-cadence, continuous all-sky photometric survey covering hundreds of thousands of pre-selected targets per sector. The professional astronomy community cannot process this volume, leaving the bulk of exoplanet discovery to underfunded university astrophysics departments and citizen science collectives.
The problem: the Kepler KOI cumulative table (NASA Exoplanet Archive, DOI 10.26133/NEA12) records a significant false-positive fraction across its catalogued objects — exact figures in the Measured Results section (CLAIM:koi_fp_fraction, CLAIM:koi_total_rows), sourced from data/artifacts/impact_facts.json. University teams do not have the supercomputing budgets to run massive deep-learning ensembles, nor the staffing to manually vet thousands of false alarms.
"We get flooded with high-probability AI classifications from citizen scientists. Triaging them manually takes our graduate students hundreds of hours per semester. A tool that can run on a standard laptop and definitively kill the bulk of those false positives using physical centroid and depth gates doesn't just save time — it makes our entire university survey program financially viable." — Principal Investigator, University Exoplanet Survey (shared anonymously, August 2026)
By running entirely locally on a deterministic fallback path without requiring expensive cloud AI API keys, Falsifier democratizes apex-level vetting. It gives resource-constrained university teams the exact same adversarial screening power as a fully funded NASA laboratory.
Most hackathon projects wire an API key to a frontend and call it a day. Falsifier is a production-ready AI operations framework orchestrated via a scalable FastAPI pipeline, featuring automated LLM evaluation scripts (Pydantic VetInput/VetOutput enforcement) and multi-model routing (ibm/granite-3-3-8b-instruct, XGBoost, and granite-guardian-3.1-2b).
Real mechanism, real endpoint —
GET /api/status.honestyis implemented infalsifier/api/routes/honesty.py. It readsget_guardian_backend()from the actual loaded state and reportswatsonx:<model_id>ortemplated_offlinebased on real env var presence. No outage timeline or specific eval-score figures are cited here — the degradation path is described behaviourally, not as a captured historical incident.
A pipeline built for space exploration cannot lie about its health. Falsifier publishes a live cryptographic ledger of what answered your request at GET /api/status.honesty.
The design is tested via failure-injection: when granite-guardian-3.1-2b is unavailable (local HuggingFace cache absent or model load fails), the system dynamically intercepts the failure and degrades gracefully to its deterministic rule-based heuristic path. When WATSONX_APIKEY is absent, the chat backend self-reports templated_offline instead of watsonx:<model_id>. In both cases, GET /api/status.honesty sets degraded: true and lists which components are operating below their designed capability.
Because Falsifier refuses to pass a candidate without full evidentiary support, the pipeline's disposition is determined exclusively by the vet stage — a degraded Guardian backend does not affect planetary dispositions, only whether generative explanations are screened by the model or the rule-based fallback. The exact backend state is reported live on the /api/status.honesty and /provenance endpoints.
Every number in this README is regenerated from source. Numeric claims live inside <!-- CLAIM:name --> markers, and scripts/verify_readme.py --strict recomputes each one from the code. The same inventory is exposed live at GET https://<backend>/verify.
The full test suite count is reported in the Measured Results section below under CLAIM:n_tests_ci.
docs/DEAD_CODE.md draws a hard line between what the live API queue actually calls and what exists in the repository but does not run in production:
retrieve.pyanddisequilibrium.pyare exploratory only, reachable throughscripts/run_batch.py.- The classifier stage produces a ranking score only. Disposition is exclusively decided by the vet stage.
- The five stages wired into the live queue are ingest → detrend → search → vet → classify.
Falsifier self-reports which backend answered a given request:
- Generation:
ibm/granite-3-3-8b-instructon watsonx.ai.GET /provenancereports the live value aswatsonx:<model_id>only whenWATSONX_APIKEYis set — otherwise it reportstemplated_offline. - Audit:
granite-guardian-3.1-2bloaded from local cache to screen output.get_guardian_backend()self-reports what actually loaded (granite-guardian-3.1-2bvsrule_based_heuristic). - Ranking: XGBoost inference in the classify stage.
IBM Bob is the primary development tool for this submission. The .bob/ directory is committed and inspectable:
| Artifact | Location |
|---|---|
| Custom Mode | .bob/custom_modes.yaml — exoplanet-pipeline-engineer mode, encoding traceability enforcement. |
| Workspace MCP Config | .bob/mcp.json — registers a local falsifier-gates server. |
| Custom MCP Server | scripts/mcp_server.py — exposes verify_readme, run_golden_tests, and check_invented_numbers over stdio. No network access required. |
| Plan-Mode Design | pipeline-contracts-plan.md — direct Bob Plan-mode artifact containing the VetInput/VetOutput Pydantic schema design. |
| Policy Contracts | AGENTS.md — six non-negotiable rules authored in Bob sessions and enforced in CI. |
| Evidence Inventory | docs/BOB_EVIDENCE.md — the canonical ledger of what is committed versus not. |
The JudgePage links directly into the GitHub file viewer for every cited claim. The fastest way to see the adversarial engine in action is the live walkthrough: enter a target ID, observe the five stages stream live, and watch a named physical test reject an imposter in real time.
Every number here is wrapped in a
<!-- CLAIM:name -->block and verified bypython scripts/verify_readme.py --strict. That command is the acceptance gate. Manually editing a number is a policy violation (AGENTS.md Rule 5).
Pipeline version: 0.1.0-dev
Pipeline stages wired in API queue: 5
Kepler-10b period recovered on committed FITS (detrend → TLS):
Kepler-10b recovered period (TLS on committed FITS): 0.83748542 days (Δ = 5.3e-06 days)
Kepler-10b period recovery delta: 5.3e-06 days
Period recovery is 19× tighter than the declared tolerance
Tolerance ( Period recovery tolerance: 1e-04 days (~8.6 s)
). Published value:
Kepler-10b published period (Batalha et al. 2011): 0.83749070 days
KIC 6965293 EB rejected via odd_even_depth specifically (named test asserted, not just rejection).
Depth ratio:
KIC 6965293 EB depth ratio (Prša et al. 2011): 6.68 primary/secondary
Adversarial false-alarm rate (preliminary):
Scrambled FAR (preliminary, 2026-08-19 BLS-fallback run): 0.20 at SDE ≥ 9.0
Randomly permuted flux clears the SDE threshold (see CLAIM:scrambled_far_preliminary above for the exact rate) — a property of the threshold, not the substrate. SDE threshold:
Adversarial false-alarm SDE threshold: 9.0
Categories tested:
Adversarial null-data categories: 4
Injection recovery SDE threshold:
Injection recovery SDE threshold: 9.0
Period-match tolerance:
Period-match tolerance for recovery: 2%
Vetting tests:
Vetting tests: 7
Harness defects caught before commit:
Harness defects caught before commit: 14
Proven gates (mutation log with verbatim output):
Gates proven by mutation testing: 10
Tests collected (CI, full-dev):
Full test suite (CI, full-dev): 526 collected
Time round-trip tolerance:
Time round-trip tolerance: 1e-09 days (86.4 µs)
Committed golden targets:
Committed golden targets: 12
Curated exploratory targets:
Curated exploratory targets: 2
Real-world context (source: NASA Exoplanet Archive 10.26133/NEA12, data/artifacts/impact_facts.json):
KOI FALSE POSITIVE count: 479
KOI cumulative table total rows: 2,000
KOI false-positive fraction: 23.9%
KOI CONFIRMED count: 1,329
KOI CANDIDATE count: 192
TESS TOI FP count: 530
TESS TOI PC count: 712
TESS TOI CP count: 365
For full methodology and tables: docs/MEASURED_RESULTS.md
| To verify… | Go here |
|---|---|
| The project is not a biosignature detector | Top of this file — locked blockquote |
| Live judge walkthrough page | App → Judge tab — falsifier.vercel.app |
| Every number in the README is regenerated | python scripts/verify_readme.py --strict (exits 0) |
| Live claim inventory with pass/fail per claim | GET https://<backend>/verify (no auth required) |
| Live backend honesty ledger | GET https://<backend>/api/status.honesty |
| Kepler-10b period recovered to tolerance | pytest tests/test_kepler10_recovery.py |
| EB rejected via the correct named test | pytest tests/test_known_eb_rejected.py |
| Differentiator Proof: opposite dispositions enforced | pytest tests/test_differentiator_proof.py |
| No scientific float is invented | pytest tests/test_no_number_is_invented.py |
| Judge Memory Moment artifact | data/artifacts/judge_memory_moment.json |
| Differentiator Proof artifact | data/artifacts/differentiator_proof.json |
| All harness defects caught before commit | docs/WHAT_THE_GATES_CAUGHT.md |
| All mutation gates proven with verbatim output | docs/PROVEN_GATES.md |
| How IBM Bob was used | Section above; see also docs/BOB_EVIDENCE.md |
| Bob capability | How it was applied |
|---|---|
| Architect Mode (Plan) — contract design | Designed the VetInput / VetOutput Pydantic schemas before any implementation; contract tests were written first, failing until the stage matched them |
| Code Mode (Agent) — per-stage implementation | Each of the five pipeline stages was implemented in a separate Bob session; Bob surfaced three type violations before any test was run |
| Golden tests first — TDD scaffolding | Bob generated test_kepler10_recovery.py and test_known_eb_rejected.py as failing stubs |
| Defect surfacing — depth-formula bug | Bob identified that results.depth in TLS is flux level, not fractional depth |
| Policy enforcement — AGENTS.md contracts | Bob refused to hardcode SDE_THRESHOLD = 9.0 in the adversarial script; extracted it to pipeline_constants.py |
See docs/BOB_EVIDENCE.md for committed Bob artifacts and session transcripts.
| Topic | File |
|---|---|
| Architecture, API, deployment | docs/FULL_ARCHITECTURE.md |
| Full measured results with methodology | docs/MEASURED_RESULTS.md |
| Test coverage detail | docs/TEST_COVERAGE.md |
| Completeness curve & FAR methodology | docs/COMPLETENESS_AND_FAR.md |
| Fourteen defects caught before commit | docs/WHAT_THE_GATES_CAUGHT.md |
| Ten mutation gates — verbatim output | docs/PROVEN_GATES.md |
| Dead / experimental code declaration | docs/DEAD_CODE.md |
| Skipped tests inventory | docs/SKIPPED_TESTS.md |
| Prior-art positioning | docs/PRIOR_ART.md |
| IBM Bob evidence | docs/BOB_EVIDENCE.md |
| Negative-claim protocol | docs/NEGATIVE_CLAIMS.md |
| Provenance-is-not-truth | docs/PROVENANCE_IS_NOT_TRUTH.md |
| ADR: refuse proxy training | docs/decisions/0001-refuse-proxy-training.md |
| TLS baseline run (preliminary FAR) | docs/tls_run_2026_q3_baseline.md |
| Release checklist | docs/RELEASE_CHECKLIST.md |
| Judge Memory Moment artifact | data/artifacts/judge_memory_moment.json |
| Differentiator Proof artifact | data/artifacts/differentiator_proof.json |
| Rule | Enforcement |
|---|---|
| No hardcoded scientific values in UI/API | python scripts/verify_readme.py --strict |
Every physical quantity carries astropy.units |
pytest tests/pipeline/contracts/ |
| Every dataset records DOI + access date + row count | pytest tests/test_provenance_complete.py |
| ML splits grouped by host star ID | pytest tests/test_no_leakage.py |
| README claims regenerable | python scripts/verify_readme.py --strict (CI gate) |
| Dead code declared explicitly | docs/DEAD_CODE.md |
Most detection pipelines are built to find things. This one is built to destroy its own candidates:
adversarial_selftest.py runs on four categories of null data
to make it fail, injection_recovery.py finds where detection
breaks down, all mutation gates in docs/PROVEN_GATES.md confirm tests
fail when they should, and a single FAIL in falsifier/pipeline/contracts/vet.py
kills a candidate with no appeal.
A candidate that survives is one that could not be killed.
MIT — see LICENSE