Parity is a restrained financial-reconciliation investigator built for the Razorpay AI Buildathon 2026, Track 04. It compares a frozen bank statement with an internal ledger, finds matches it can substantiate, and flags the rest in a reason-coded exception book. It never moves money or acts on a merchant's behalf.
The graded held-out evaluation is synthetic bank-versus-ledger only. The app also fetches the merchant's Razorpay Test Mode Settlement Recon feed as a separate, unlabeled live-integration panel. It is never included in the graded precision/recall result.
These are the single canonical metrics in
results/current_run.json, captured by a
fixture-free live evaluation on 2026-08-24. It used the real MiniLM
sentence-transformers encoder and live Groq adjudication over the immutable
held-out set.
| Metric | Fresh live result |
|---|---|
| Truth transactions | 300 |
| Source records processed | 628 |
| Tier 1 matches | 219 |
| Tier 2 matches | 53 |
| Total matched | 272 |
| Match rate | 90.67% |
| Precision | 100.00% |
| Recall over 296 resolvable truths | 91.89% |
| False-positive decisions / cost | 0 / ₹0.00 |
| Live wall time / throughput | 582.706 s / 1.078 records/s |
| Groq usage | 207 / 500 calls; 89,966 / 200,000 tokens |
| Configuration | Match rate | Precision | Notes |
|---|---|---|---|
| Tier 1 only (deterministic matcher; no Tier 2) | 219/300 (73.00%) | 100.00% | Deterministic baseline; zero false positives |
| Tier 1 + Tier 2 with fixture adjudicator | 296/300 (98.67%) | Not valid | Reported during development; original artifact not retained — LLM was never called; fixture rubber-stamped every decision, caught before submission |
| Tier 1 + Tier 2 with real Groq, before partial-refund signal | 279/300 (93.00%) | 100.00% | Reported during development; original artifact not retained |
| Tier 1 + Tier 2 + partial-refund signal (Phase 3 final) | 292/300 (97.33%) | 100.00% | Eleven of 13 previously rejected partial refunds resolved |
| Final canonical result after Phase 5 hardening | 272/300 (90.67%) | 100.00% | Evaluator hardened; some previously counted matches no longer qualify |
This is not a single reported number — it is the actual progression across the build. The apparent drop from row 2 to row 3 is the fixture bug being caught and fixed. The drop from row 4 to row 5 is the evaluator being hardened, not the system getting worse. Every artifact-backed number here is reproducible from the commit history; the two development-only rows are disclosed below.
The fixture and pre-signal rows were reported live during development but not saved as separate artifacts — only the final state of each phase was committed. They're included for narrative completeness, not as verified claims.
The accounting is intentionally explicit: 263 truth transactions are matched-only and 37 appear in the exception book, so those exclusive sets sum to 300. Nine duplicate cases appear in both views because the valid pair is matched while the extra duplicate source row is flagged. The exception report therefore contains 38 source-level entries representing 37 truths.
The exception book keeps two money figures separate:
| Classification | Entries | Amount |
|---|---|---|
| Evidence-supported leakage | 13 | ₹12,428.63 |
| Review-only, not claimed as leakage | 25 | ₹5,356.66 |
The live run encountered Groq free-tier pressure: 140 rate-limit responses, 61 retries, one retried transport error, and 13 candidates returned to the exception book rather than guessed. Precision remained 100%.
The frozen hash is
2aacac85b9d15cc186c63b2ceb1557767c99b3dfacd9931e4655a3fd7f9d8154.
make demo verifies that hash, reruns deterministic Tier 1, regrades the
captured live Tier 2 decisions, and republishes the audit/results. It makes no
new LLM calls. make eval-tier2-live is the explicit, network-dependent
command for refreshing the canonical live artifact.
The FastAPI backend and React frontend are one Render service. The interface uses Razorpay's open-source Blade design system rather than copied colors.
- Live app: https://parity-1go2.onrender.com
- Health: https://parity-1go2.onrender.com/api/health
Render's free instance sleeps after 15 minutes idle. A first request can take 30–60 seconds to wake; this is expected.
Prerequisites: Python 3.12, Node/npm, and GNU Make. Razorpay test-mode keys are required for the connectivity gate. A Groq key is required only for an explicit live refresh, not for the reproducible canonical demo.
git clone https://github.com/somuai/parity.git
cd parity
cp .env.example .env
# Add RAZORPAY_TEST_KEY_ID, RAZORPAY_TEST_KEY_SECRET, and GROQ_API_KEY to .env.
make setup
make confirm-api
make demoExpected make demo result: 272/300 matched, 90.67% match rate, 100.00%
precision, 91.89% recall, and ₹0.00 false-positive cost.
To run the app locally:
make run-backend # http://localhost:8000
make run-frontend # http://localhost:5173, proxies /api to FastAPITo perform a new live evaluation with real embeddings and Groq calls:
make eval-tier2-liveThat command can produce a different conservative match count when Groq's free-tier capacity is unavailable; failed adjudications are reason-coded exceptions, never silent matches. The checked-in canonical artifact exists so the panel can reproduce the submitted claim without depending on a provider's minute-by-minute availability.
make confirm-api validates authentication, pagination, response shape, INR
amount direction, and empty-account behavior for Razorpay's test-mode
Settlement Recon API. The Razorpay feed is not silently included in the
graded precision/recall claim: the labeled evaluation is bank-versus-ledger.
RAZORPAY_TEST_KEY_ID and RAZORPAY_TEST_KEY_SECRET are used by the strict
Razorpay client for two read-only endpoints: Settlement Recon and recent
payment activity. On 2026-08-24 we created and captured a ₹1.00 Test Mode
payment; the payment API returned paid, while Settlement Recon still returned
zero rows. That is an honest Test Mode limitation: captured payments do not
necessarily create settlement rows. The app now labels those two facts
separately, and neither is included in frozen precision/recall.
The failure mode was Groq free-tier capacity during the live held-out run: 140 rate-limit responses and one transport failure. The dangerous shortcut would have been to accept plausible candidates without an adjudicator. Parity instead retried with backoff, checked both LLM budgets before every call, and returned 13 candidates to the reason-coded exception book when evidence could not be completed. The result preserved 100% precision and ₹0 false-positive cost. The hosted Re-run held-out batch control then replays the immutable canonical artifact without new LLM calls and verifies the complete outcome digest, so a judge can reproduce the published number even when Groq is busy.
| Area | Status |
|---|---|
| Data freeze | Complete; evaluator checks the immutable hash before metrics |
| Tier 1 | Complete; empty references, zero amounts, sign mismatches, and duplicate references fail closed |
| Tier 2 | Complete; grouped arithmetic runs in Python before any LLM call, missing semantic evidence degrades confidence, and provider failures become exceptions |
| Evaluation | Complete; structural truth checks prevent single-sided, duplicated, or unknown record IDs from earning credit |
| Audit and exception book | Complete; run-scoped exact coverage and separate leakage/non-leakage totals |
| Hosted app | Complete; canonical replay powers /api/rerun, with an outcome digest comparison rather than a match-rate-only comparison |
| Security/API hardening | Complete for submission blockers; secrets are ignored, Render secrets use sync: false, Groq input is delimited as untrusted, and Razorpay responses are strictly validated |
| Phase 5 review | Complete; number-changing and demo-crashing findings fixed |
| Phase 6 reproduction | Complete; a new clone reproduced the canonical outcome digest and every pitch metric |
| Deferred: Razorpay third leg | Connectivity and validation only; a labeled three-source reconciliation claim is not made |
| Deferred: rerun authorization | /api/rerun is unauthenticated, but it is now a deterministic, zero-LLM canonical replay; admin auth and durable cooldown remain deployment work |
| Deferred: artifact transactionality | JSON files are atomically replaced and carry run IDs, but the multi-file publication is not one database transaction |
| Deferred: API polish | Explicit FastAPI response models, bundle splitting, and consolidation of the empty observability/ package are maintainability work |
| Deferred: dependency closure | Direct Python packages, CPU torch, the embedding model revision, and npm lockfile are pinned; the entire transitive Python graph is not hash-locked |
| Deferred: frontend dependency advisories | npm audit --omit=dev reports Blade's moderate ts-deepmerge advisory; the offered fix downgrades Blade, so this remains documented pending an upstream-compatible release. The separate high advisory is build-time/dev-only |
| Deferred: history cleanup | Phase-boundary file ownership is documented, but published git history will not be rewritten merely to make phase attribution prettier |
- Actual architecture
- Five-minute pitch script
- Original product requirements
- Canonical live result
- Current evaluator report
- Exception book
config/schema.py canonical record and decision contracts
clients/razorpay_client.py strict test-mode Settlement Recon client
data/holdout/ immutable labeled bank/ledger evaluation set
engine/tier1_deterministic.py fail-closed deterministic matcher
engine/tier2_reasoning.py residual signals, adjudication, fusion
engine/audit.py run-scoped SQLite audit trail
eval/phase3_live.py structural grader and metrics
eval/exception_book.py reason-coded leakage/review report
app/ FastAPI API and static frontend host
frontend/ React + Razorpay Blade interface
results/canonical_eval.json captured fixture-free live evaluation
Parity finds and flags. It does not claim to recover funds, post entries, or resolve exceptions on a merchant's behalf.