Repository navigation
fix(brain): align AptAutoremove prose with daemon risk - #695
Conversation
|
I have the scoped fix ready at I also provisioned a real Ubuntu 24.04.5 VM through the repository scripts. My CPU-only Could you run the required full Ubuntu comparison on the project's configured |
vladimirrott
left a comment
There was a problem hiding this comment.
Reviewed at b6b2704f6b2a515670fdc974b87e361f0b595adb.
The negation handling in prose_risk_claim is the part I'd have got wrong. Line 932 reads "LOW, not HIGH", so a parser that counted every level token would call it ambiguous and skip it, and the old wording would have slipped past the new guard. You made the not case a tested rule instead of a lucky outcome, and the checked > 0 assertion per family stops the test from passing on a prompt it no longer reads.
I ran your test in a container at this head, then broke things one at a time:
head as submitted test result: ok. 5 passed; 0 failed
both prompt lines back to main prose_risk_claims_match_the_action_specs ... FAILED
[ubuntu] line 453: AptAutoremove: prose says Low, ActionSpec says Medium
[ubuntu] line 492: AptAutoremove: prose says Low, ActionSpec says Medium
only line 891 reverted FAILED, line 453 named
only line 932 reverted FAILED, line 492 named
parser ignores `not` 2 failed: the negation control, and "[debian] no prose risk claims were checked"
parser returns None for every line 2 failed: the negation control, and "[fedora] no prose risk claims were checked"
The command was cargo test -p sysknife-daemon --test prompt_risk_labels --offline --locked. Only [ubuntu] reports the two lines because render_debian_prompt skips DEBIAN_SELECTION_RULES and DEBIAN_COUNTERINTUITIVE when the host is plain Debian, which matches the code. The daemon side was already right: apt.rs sets RiskLevel::Medium, so the approval gate asked at medium before and after this change. What changes is the explanation an Ubuntu operator reads above that prompt.
The one thing for you: tests/evidence/workspace-tests.json conflicts with main, which moved from 2020 to 2091 tests while this was open. That is main's doing, not yours. A rebase plus UPDATE_TEST_BASELINE=1 scripts/test_baseline.sh should land on 2093. The rendered figures stay at their floor, so no doc needs touching.
The live comparison is mine to run. Thank you for the Ubuntu 24.04.5 attempts and for keeping the failures rather than counting them. A 3B model on CPU can't fit story 60 into its 600 seconds. I'll run the Ubuntu family on gpt-oss-120b at 24ebd624 and at your head, and post both results here. Leave the PR as a draft until then if you like; once the rebase is in, the live run is all it waits on.
Optional, and fine to skip: the three Ubuntu cassettes still carry "low-risk" explanations for AptAutoremove. They are recordings, so they will refresh when I re-record after the live run.
I assigned #649 to you, so it now shows on your dashboard as well as under the label.
|
I merged current main The prompt and risk-guard blobs are unchanged from the head you reviewed. The three-file PR is now conflict-free. For the live comparison, the current points are main |
|
Heads-up on a change that landed after you opened this: git fetch https://github.com/lacs-project/sysknife.git main
git rebase --signoff FETCH_HEAD
git push --force-with-leaseThat also brings the branch up to date with |
|
Checked Two things are left. Yours is the sign-off rebase in the note above. Mine is the live gpt-oss-120b comparison on the Ubuntu family, which I still owe you; this PR waits on nothing else. |
vladimirrott
left a comment
There was a problem hiding this comment.
I ran the live comparison I owed you. Full noble suite (stories 55 to 153, destructive stories on) against openai/gpt-oss-120b on Groq, once on main 45d5bdff and once on a822d7a3, each from a fresh VM, every plan a live call:
main 45d5bdff |
this PR a822d7a3 |
|
|---|---|---|
| full suite | 99/99 | 98/99 |
| story 60, autoremove unused packages | PASS | PASS |
| story 142, enable NTP, 10 more live runs | 6/10 | 7/10 |
Story 60 is the one that plans AptAutoremove, and it passed on both. The one failure on your head was story 142: the model sent SetNtp with {} instead of {"enabled":true} and the CLI refused the plan. I reran that story ten times on each ref with no cassette, and both refs fail at the same rate, so it predates this PR and has nothing to do with the two lines you changed. I filed it as #737.
Approving at a822d7a3, and the merge is armed. Once you push the sign-off rebase and mark the PR ready, I merge when all of these hold:
prompt.rsandprompt_risk_labels.rsmatcha822d7a3, and the PR still changes only its three files.- On the rebased head your guard test passes, and it fails with both prompt lines reverted, run in a container the way I ran it in my first review.
- Every check is green,
dcoincluded. - This approval still stands.
If the rebase carries any other change, I'll review it again instead of merging. Your branch merges cleanly with current main and main hasn't touched any of your three files, so git rebase --signoff should go through without conflicts.
Signed-off-by: jabrailkhalil <jabrailkhalil@gmail.com>
a822d7a to
0dee087
Compare
|
I rebased onto I reran the full workspace on current main (2,091 passed, 8 skipped) and the patched source (2,093 passed, 8 skipped), plus format and Clippy. I ran The new head now has 14 successful upstream checks; the opt-in container-smoke job is skipped. The approval remains in place. Your live comparison remains the provider-level evidence. |
vladimirrott
left a comment
There was a problem hiding this comment.
Reviewed at 0dee0879883b101a41b5e04c8773b154d937d8a2.
Thanks for the rebase and the sign-off, and for listing what you checked in your comment: I could confirm each piece in a few minutes. Leaving out the evidence file, your diff from the new merge base (25c9e76) is identical to the diff from the old one (45d5bdf), so my approval and the live Groq comparison carry over. The count moves 2,091 → 2,093 for the two tests the PR adds.
I re-ran your guard at this head in a container, reverting each prompt line on its own:
cargo test -p sysknife-daemon --test prompt_risk_labels --offline --locked
pristine 5 passed, rc=0
prompt.rs:932 back to LOW prose_risk_claims_match_the_action_specs FAILED
[ubuntu] line 492: AptAutoremove: prose says Low, ActionSpec says Medium
prompt.rs:891 back to LOW prose_risk_claims_match_the_action_specs FAILED
[ubuntu] line 453: AptAutoremove: prose says Low, ActionSpec says Medium
both lines back to LOW FAILED, both lines named
restored 5 passed, rc=0
The test catches either copy drifting back by itself, which is the case that matters for a sentence that lives in two places.
Nothing blocks this. Every required check is green and the branch is current with main. The merge is mine to do, and I will take it on my next pass.
Resolve the test baseline after lacs-project#701: main is at 2093 with lacs-project#701's two tests, and this branch adds two more, so the count is 2095. Both sides edited 2091 to 2093 identically, so git merged them silently to 2093. Signed-off-by: Vladimir Rotariu <sysknife-development@protonmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UpyrhbDTW1gBwzGAC44GYU
vladimirrott
left a comment
There was a problem hiding this comment.
Re-approving at 9ed8876f. My merge of main into your branch dismissed the earlier approval; your two files are unchanged from 0dee0879.
I checked that before re-running anything. Your diff against main, minus the evidence file, is byte-identical to 0dee0879 against its old base, and the baseline now reads 2095: main's 2093 after #701, plus your two tests. The rust check agrees.
Then the same three mutations as last time, at 9ed8876f, in a container with no network (cargo test -p sysknife-daemon --test prompt_risk_labels):
- Pristine: 5 pass.
- Line 932 back to LOW:
prose_risk_claims_match_the_action_specsfails with[ubuntu] line 492: AptAutoremove: prose says Low, ActionSpec says Medium. - Line 891 back to LOW: fails with
line 453. - Both: fails naming both lines.
- Restored: 5 pass again.
Nothing is left for you to do here. The merge is mine once docs-and-hygiene, fuzz and CodeQL finish green.
Refresh the Rust test baseline: main is at 2095 after lacs-project#701 and lacs-project#695, and this branch adds five tests, so 2100. Move async-trait below the sysknife-daemon comment in the CLI's dev-dependencies, where it split that comment from the line it describes. Signed-off-by: Vladimir Rotariu <sysknife-development@protonmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UpyrhbDTW1gBwzGAC44GYU
Summary
I corrected both Ubuntu prompt claims about
AptAutoremoveto MEDIUM and stated why: apt chooses the removal set at runtime, and that set can include old kernels.I extended the existing drift guard to compare unambiguous prose claims against the live
ActionSpeccatalogue. It distinguishes affirmative labels from negations such asLOW, not HIGH. I changed no execution policy or approval gate.Related Issue
Closes #649.
Validation
45d5bdffplus this patch, the fresh remote workspace run passed 2093 tests, with 8 skips. Format and Clippy across all workspace features/targets with warnings denied passed.UPDATE_TEST_BASELINE=1 scripts/test_baseline.shand copied the generated file verbatim. The provenance fixture passed; rendering checked 50 figure markers and rewrote no documents. The published three-file patch matches the remotely tested patch byte for byte.a822d7a3: 13 successful checks; the workflow-dispatch-only smoke was skipped.Notes for Reviewers
I merged main
45d5bdff66984ff4231b98ebdca5260c53022774normally intoa822d7a39474ec959bc145aa198c2f6b5b5faea1. The net diff remains the two prompt claims, the prose guard, and the generated baseline; the baseline conflict is resolved.I retained the real Ubuntu VM/CPU attempts, including the last failed live probe. No complete PR-head live-suite result is claimed. I am keeping this draft pending the maintainer's configured
gpt-oss-120bcomparison offered in the review; the current comparison points are main45d5bdffand heada822d7a3.