Skip to content

fix(brain): align AptAutoremove prose with daemon risk - #695

Merged
vladimirrott merged 2 commits into
lacs-project:mainfrom
jabrailkhalil:fix/prompt-risk-prose
Nov 3, 2026
Merged

vladimirrott merged 2 commits into
lacs-project:mainfrom
jabrailkhalil:fix/prompt-risk-prose

Conversation

@jabrailkhalil

@jabrailkhalil jabrailkhalil commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

I corrected both Ubuntu prompt claims about AptAutoremove to MEDIUM and stated why: apt chooses the removal set at runtime, and that set can include old kernels.

I extended the existing drift guard to compare unambiguous prose claims against the live ActionSpec catalogue. It distinguishes affirmative labels from negations such as LOW, not HIGH. I changed no execution policy or approval gate.

Related Issue

Closes #649.

Validation

  • Tests and prompt text updated together; daemon trust boundary preserved.
  • I reproduced both incorrect claims before editing the prompt in the test-first run. Reverting either line independently triggers the intended assertion. The prompt and guard blobs remain identical to the reviewed head.
  • On main 45d5bdff plus this patch, the fresh remote workspace run passed 2093 tests, with 8 skips. Format and Clippy across all workspace features/targets with warnings denied passed.
  • I regenerated the baseline with UPDATE_TEST_BASELINE=1 scripts/test_baseline.sh and copied the generated file verbatim. The provenance fixture passed; rendering checked 50 figure markers and rewrote no documents. The published three-file patch matches the remotely tested patch byte for byte.
  • Fresh upstream CI, CodeQL, fuzz, container and secret-scan checks passed on a822d7a3: 13 successful checks; the workflow-dispatch-only smoke was skipped.

Notes for Reviewers

I merged main 45d5bdff66984ff4231b98ebdca5260c53022774 normally into a822d7a39474ec959bc145aa198c2f6b5b5faea1. The net diff remains the two prompt claims, the prose guard, and the generated baseline; the baseline conflict is resolved.

I retained the real Ubuntu VM/CPU attempts, including the last failed live probe. No complete PR-head live-suite result is claimed. I am keeping this draft pending the maintainer's configured gpt-oss-120b comparison offered in the review; the current comparison points are main 45d5bdff and head a822d7a3.

@jabrailkhalil

Copy link
Copy Markdown
Contributor Author

I have the scoped fix ready at b6b2704f6b2a515670fdc974b87e361f0b595adb: the two prompt claims, the prose drift guard, and the generated test baseline. All 12 executed upstream checks pass. The new guard fails on the original prompt, and reverting either incorrect line independently is detected.

I also provisioned a real Ubuntu 24.04.5 VM through the repository scripts. My CPU-only llama3.2:3b probes cannot complete the unchanged main-side story 60 within its original 600 seconds. In the last attempt, neutral initialization took about 22 minutes; the live story then returned plain text naming AptPurge instead of a propose_plan call, and timed out while processing the planner's correction. I retained the failure, live-call recording, and logs. The full family and PR-head live comparison remain unverified.

Could you run the required full Ubuntu comparison on the project's configured openai/gpt-oss-120b or GPU setup? The exact base is 24ebd624b740debfb1614327b09e281cbcb8f496, and the PR head is b6b2704f6b2a515670fdc974b87e361f0b595adb. I have kept this draft while that check is outstanding, with the original story assertions, risk thresholds, and timeouts intact.

@vladimirrott vladimirrott left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at b6b2704f6b2a515670fdc974b87e361f0b595adb.

The negation handling in prose_risk_claim is the part I'd have got wrong. Line 932 reads "LOW, not HIGH", so a parser that counted every level token would call it ambiguous and skip it, and the old wording would have slipped past the new guard. You made the not case a tested rule instead of a lucky outcome, and the checked > 0 assertion per family stops the test from passing on a prompt it no longer reads.

I ran your test in a container at this head, then broke things one at a time:

head as submitted                     test result: ok. 5 passed; 0 failed
both prompt lines back to main        prose_risk_claims_match_the_action_specs ... FAILED
                                        [ubuntu] line 453: AptAutoremove: prose says Low, ActionSpec says Medium
                                        [ubuntu] line 492: AptAutoremove: prose says Low, ActionSpec says Medium
only line 891 reverted                FAILED, line 453 named
only line 932 reverted                FAILED, line 492 named
parser ignores `not`                  2 failed: the negation control, and "[debian] no prose risk claims were checked"
parser returns None for every line    2 failed: the negation control, and "[fedora] no prose risk claims were checked"

The command was cargo test -p sysknife-daemon --test prompt_risk_labels --offline --locked. Only [ubuntu] reports the two lines because render_debian_prompt skips DEBIAN_SELECTION_RULES and DEBIAN_COUNTERINTUITIVE when the host is plain Debian, which matches the code. The daemon side was already right: apt.rs sets RiskLevel::Medium, so the approval gate asked at medium before and after this change. What changes is the explanation an Ubuntu operator reads above that prompt.

The one thing for you: tests/evidence/workspace-tests.json conflicts with main, which moved from 2020 to 2091 tests while this was open. That is main's doing, not yours. A rebase plus UPDATE_TEST_BASELINE=1 scripts/test_baseline.sh should land on 2093. The rendered figures stay at their floor, so no doc needs touching.

The live comparison is mine to run. Thank you for the Ubuntu 24.04.5 attempts and for keeping the failures rather than counting them. A 3B model on CPU can't fit story 60 into its 600 seconds. I'll run the Ubuntu family on gpt-oss-120b at 24ebd624 and at your head, and post both results here. Leave the PR as a draft until then if you like; once the rebase is in, the live run is all it waits on.

Optional, and fine to skip: the three Ubuntu cassettes still carry "low-risk" explanations for AptAutoremove. They are recordings, so they will refresh when I re-record after the live run.

I assigned #649 to you, so it now shows on your dashboard as well as under the label.

@jabrailkhalil

Copy link
Copy Markdown
Contributor Author

I merged current main 45d5bdff normally in a822d7a3 and regenerated the baseline through the real workspace runner: 2093 passed, 8 skipped. Format, all-features/all-targets Clippy with warnings denied, and the baseline provenance fixture pass. Rendering checked 50 figure markers and rewrote no documents. The remote evidence contains the generated baseline and tested patch; I verified the published patch is byte-identical to it.

The prompt and risk-guard blobs are unchanged from the head you reviewed. The three-file PR is now conflict-free. For the live comparison, the current points are main 45d5bdff66984ff4231b98ebdca5260c53022774 and head a822d7a39474ec959bc145aa198c2f6b5b5faea1. I have kept the draft pending your run and updated the description with the exact results.

@vladimirrott

Copy link
Copy Markdown
Member

Heads-up on a change that landed after you opened this: main now requires a DCO sign-off on every commit (#713, merged in #732). Each commit needs a Signed-off-by: Your Name <you@example.com> trailer whose address matches the commit's author, and the new dco check blocks this branch until its commits carry one. From your branch:

git fetch https://github.com/lacs-project/sysknife.git main
git rebase --signoff FETCH_HEAD
git push --force-with-lease

That also brings the branch up to date with main, and git commit -s adds the trailer to new commits. Sorry for changing the rules mid-review; nothing else about this PR changes.

@vladimirrott

Copy link
Copy Markdown
Member

Checked a822d7a3: prompt.rs and prompt_risk_labels.rs are byte-identical to the b6b2704f I reviewed, the baseline reads 2093, which is main's 2091 plus your two tests, and the branch merges cleanly with current main. Thank you for regenerating it through the real runner.

Two things are left. Yours is the sign-off rebase in the note above. Mine is the live gpt-oss-120b comparison on the Ubuntu family, which I still owe you; this PR waits on nothing else.

vladimirrott
vladimirrott previously approved these changes Oct 29, 2026

@vladimirrott vladimirrott left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran the live comparison I owed you. Full noble suite (stories 55 to 153, destructive stories on) against openai/gpt-oss-120b on Groq, once on main 45d5bdff and once on a822d7a3, each from a fresh VM, every plan a live call:

main 45d5bdff this PR a822d7a3
full suite 99/99 98/99
story 60, autoremove unused packages PASS PASS
story 142, enable NTP, 10 more live runs 6/10 7/10

Story 60 is the one that plans AptAutoremove, and it passed on both. The one failure on your head was story 142: the model sent SetNtp with {} instead of {"enabled":true} and the CLI refused the plan. I reran that story ten times on each ref with no cassette, and both refs fail at the same rate, so it predates this PR and has nothing to do with the two lines you changed. I filed it as #737.

Approving at a822d7a3, and the merge is armed. Once you push the sign-off rebase and mark the PR ready, I merge when all of these hold:

  1. prompt.rs and prompt_risk_labels.rs match a822d7a3, and the PR still changes only its three files.
  2. On the rebased head your guard test passes, and it fails with both prompt lines reverted, run in a container the way I ran it in my first review.
  3. Every check is green, dco included.
  4. This approval still stands.

If the rebase carries any other change, I'll review it again instead of merging. Your branch merges cleanly with current main and main hasn't touched any of your three files, so git rebase --signoff should go through without conflicts.

Signed-off-by: jabrailkhalil <jabrailkhalil@gmail.com>
@jabrailkhalil
jabrailkhalil marked this pull request as ready for review October 30, 2026 13:04
@jabrailkhalil

jabrailkhalil commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor Author

I rebased onto 25c9e76 with my author-matching DCO sign-off and marked this ready for review. The prompt and guard test are byte-identical to a822d7a; the diff is still limited to those two files and the generated baseline.

I reran the full workspace on current main (2,091 passed, 8 skipped) and the patched source (2,093 passed, 8 skipped), plus format and Clippy. I ran prose_risk_claims_match_the_action_specs inside an Ubuntu 24.04 container: it passes, fails with both prompt lines reverted, and passes again after restoration. The three changed files match the tested artifacts exactly. scripts/check_dco.sh origin/main HEAD passes for the signed commit.

The new head now has 14 successful upstream checks; the opt-in container-smoke job is skipped. The approval remains in place. Your live comparison remains the provider-level evidence.

@vladimirrott vladimirrott left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 0dee0879883b101a41b5e04c8773b154d937d8a2.

Thanks for the rebase and the sign-off, and for listing what you checked in your comment: I could confirm each piece in a few minutes. Leaving out the evidence file, your diff from the new merge base (25c9e76) is identical to the diff from the old one (45d5bdf), so my approval and the live Groq comparison carry over. The count moves 2,091 → 2,093 for the two tests the PR adds.

I re-ran your guard at this head in a container, reverting each prompt line on its own:

cargo test -p sysknife-daemon --test prompt_risk_labels --offline --locked

pristine                          5 passed, rc=0
prompt.rs:932 back to LOW         prose_risk_claims_match_the_action_specs FAILED
                                    [ubuntu] line 492: AptAutoremove: prose says Low, ActionSpec says Medium
prompt.rs:891 back to LOW         prose_risk_claims_match_the_action_specs FAILED
                                    [ubuntu] line 453: AptAutoremove: prose says Low, ActionSpec says Medium
both lines back to LOW            FAILED, both lines named
restored                          5 passed, rc=0

The test catches either copy drifting back by itself, which is the case that matters for a sentence that lives in two places.

Nothing blocks this. Every required check is green and the branch is current with main. The merge is mine to do, and I will take it on my next pass.

Resolve the test baseline after lacs-project#701: main is at 2093 with lacs-project#701's two
tests, and this branch adds two more, so the count is 2095. Both sides
edited 2091 to 2093 identically, so git merged them silently to 2093.

Signed-off-by: Vladimir Rotariu <sysknife-development@protonmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UpyrhbDTW1gBwzGAC44GYU

@vladimirrott vladimirrott left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving at 9ed8876f. My merge of main into your branch dismissed the earlier approval; your two files are unchanged from 0dee0879.

I checked that before re-running anything. Your diff against main, minus the evidence file, is byte-identical to 0dee0879 against its old base, and the baseline now reads 2095: main's 2093 after #701, plus your two tests. The rust check agrees.

Then the same three mutations as last time, at 9ed8876f, in a container with no network (cargo test -p sysknife-daemon --test prompt_risk_labels):

  • Pristine: 5 pass.
  • Line 932 back to LOW: prose_risk_claims_match_the_action_specs fails with [ubuntu] line 492: AptAutoremove: prose says Low, ActionSpec says Medium.
  • Line 891 back to LOW: fails with line 453.
  • Both: fails naming both lines.
  • Restored: 5 pass again.

Nothing is left for you to do here. The merge is mine once docs-and-hygiene, fuzz and CodeQL finish green.

@vladimirrott
vladimirrott merged commit fceee73 into lacs-project:main Nov 3, 2026
15 checks passed
vladimirrott added a commit to inmassat/sysknife that referenced this pull request Nov 3, 2026
Refresh the Rust test baseline: main is at 2095 after lacs-project#701 and lacs-project#695, and
this branch adds five tests, so 2100. Move async-trait below the
sysknife-daemon comment in the CLI's dev-dependencies, where it split
that comment from the line it describes.

Signed-off-by: Vladimir Rotariu <sysknife-development@protonmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UpyrhbDTW1gBwzGAC44GYU
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The Ubuntu prompt calls AptAutoremove LOW twice, while its own table and the daemon say MEDIUM

2 participants