You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Audit-derived onboarding issues ship stale; driftNotes are hand-maintained and drift #341
Filed from the HomeAssistant-Config #16 operational onboarding. Two related staleness problems surfaced: the audit-derived onboarding issue shipped with already-resolved findings, and the registry driftNotes described work already completed.
1. Audit-derived onboarding issues ship stale
#16's "Audit findings driving this" block listed three findings. When an agent picked it up, two were already resolved:
DEFECT secrets: CODEGEN_APP_CLIENT_ID / CODEGEN_APP_PRIVATE_KEY missing (both stores) - both were already present in the Actions and Dependabot stores.
DRIFT branch: main carries content develop lacks (forward-sync needed) - inverted: develop was already content-ahead of main; nothing to forward-sync.
LETTER file: repo-config/... absent - the only finding still real.
An agent that trusted the issue body would "fix" already-fixed things (re-request secrets, attempt a no-op forward-sync). The findings are a point-in-time snapshot with no freshness signal.
Suggestion: have spec/audit.py (or the issue-filing flow) timestamp findings, and/or re-verify at pickup, so a standup issue reflects live state rather than a stale snapshot. A one-line "as of <commit> / <UTC>" stamp plus a "re-run the audit before acting" note would be enough.
2. Registry driftNotes are hand-maintained and drift
HomeAssistant-Config's driftNotes described work already completed at pickup:
"develop is behind main by 2 commits - fast-forward develop... then flip groundTruthBranch to develop" - develop was ahead, not behind (and the flip is declined; see the operational-ground-truth issue).
"pending: lint CI (a Home Assistant config-check feeding the required check)" - test-pull-request.yml with the Check pull request workflow status job required check was already live.
This is the same class of staleness #329 fixed reactively for Vantage-Config. Hand-maintained prose drifts silently because nothing checks it against reality.
Suggestion: treat driftNotes as generated/verified by the audit rather than free prose - e.g. have audit.py flag driftNotes whose asserted facts (branch-behind, missing CI, missing files) no longer match live state, so drift is caught the same way ruleset/secret drift is.
Related
#310 (strengthen onboarding: enforce the audit, STANDUP counterpart, cold-start self-test)
1. Freshness.spec/audit.py prints a run stamp (audit run <UTC> | hub <sha>) and, per repo, the exact commit it read (@ <branch>@<sha>), closing with a re-run reminder. AUDIT.md section 8 now requires anything derived from a run - a report, and especially an onboarding/conformance issue - to quote the stamp, and requires the agent picking it up to re-run the audit and act on the live result, not the pasted findings. Findings are evidence for why an issue was filed, never a claim about current state - exactly the #16 failure where two of three findings were already resolved at pickup.
2. driftNotes. A note records a current deviation; once resolved it is deleted. The audit now raises a drift finding when a repo audits clean but a note still asserts outstanding work (pending / not yet / missing / behind / owed / todo / still / absent), matched on word boundaries. The check is gated on everything else passing and uses narrow markers, so a permanent-deviation note ("no get-version-task; relies on validate-task") never trips it. Codified as setup.driftnotes.current.
The check earned its keep immediately: it caught a stale note I had written myself for HomeAutomation-Config ("OWNER TASK OPEN: ... secrets not yet set") that you had since resolved and I never reconciled. Verified both secrets present in both stores and removed the note - with conformance complete it recorded no deviation at all. Fleet-wide, the heuristic flagged exactly that one note.
Copilot review caught two real bugs I introduced (an early-return path that returned a bare list, masking gh errors as ValueError; and a stamp that could misattribute a develop ground truth to main's SHA) plus a marker-boundary weakness - all fixed and regression-tested.
Related: #310 (onboarding roadmap) and #329/#335/#348 (the reactive driftNotes reconciliations this now catches automatically).
Activity
Registry: HomeAssistant-Config operational conformance completed (#342)
Stamp audit runs and flag stale driftNotes (#341) (#352)
ptr727 commentedon Jul 18, 2026
Both halves fixed in #352 (merged to develop).
1. Freshness.
spec/audit.pyprints a run stamp (audit run <UTC> | hub <sha>) and, per repo, the exact commit it read (@ <branch>@<sha>), closing with a re-run reminder. AUDIT.md section 8 now requires anything derived from a run - a report, and especially an onboarding/conformance issue - to quote the stamp, and requires the agent picking it up to re-run the audit and act on the live result, not the pasted findings. Findings are evidence for why an issue was filed, never a claim about current state - exactly the #16 failure where two of three findings were already resolved at pickup.2. driftNotes. A note records a current deviation; once resolved it is deleted. The audit now raises a drift finding when a repo audits clean but a note still asserts outstanding work (pending / not yet / missing / behind / owed / todo / still / absent), matched on word boundaries. The check is gated on everything else passing and uses narrow markers, so a permanent-deviation note ("no get-version-task; relies on validate-task") never trips it. Codified as
setup.driftnotes.current.The check earned its keep immediately: it caught a stale note I had written myself for HomeAutomation-Config ("OWNER TASK OPEN: ... secrets not yet set") that you had since resolved and I never reconciled. Verified both secrets present in both stores and removed the note - with conformance complete it recorded no deviation at all. Fleet-wide, the heuristic flagged exactly that one note.
Copilot review caught two real bugs I introduced (an early-return path that returned a bare list, masking gh errors as ValueError; and a stamp that could misattribute a develop ground truth to main's SHA) plus a marker-boundary weakness - all fixed and regression-tested.
Related: #310 (onboarding roadmap) and #329/#335/#348 (the reactive driftNotes reconciliations this now catches automatically).