Skip to content

fix: score top trials on merged models to close live-LoRA/export gap - #495

Open
akushonkamen wants to merge 1 commit into
p-e-w:masterfrom
akushonkamen:fix/490-merged-score-fidelity
Open

akushonkamen wants to merge 1 commit into
p-e-w:masterfrom
akushonkamen:fix/490-merged-score-fidelity

Conversation

@akushonkamen

Copy link
Copy Markdown

Fixes #490.

Trial scores come from the live-LoRA operator: during the study, the Optuna objective runs evaluator.get_scores() on the model that get_peft_model attached the adapters to, while the exported model runs the merged operator from merge_and_unload() (Model.get_merged_model()). The two apply different rounding orders — live is y = R(R(xWᵀ) + R(R(xAᵀ)Bᵀ)), merged is y = R(x·R(W+BA)ᵀ) — so the optimizer selects on numbers the exported model does not reproduce.

A CPU micro-repro of exactly these two formulas (torch, bf16) confirms it: the live and merged results diverge on most elements (this machine: max |live − merged| = 0.125, 138/256 elements differ; the magnitude depends on the random draw, but the divergence is always present).

Per the issue, per-trial baking is too high a performance price, so this leaves trial-time scoring untouched and closes the gap after the study, before the Pareto menu is built:

  • Re-scores the top n_merged_score_trials trials (new setting, default 3, 0 disables) on their merged models: restore the trial's live-LoRA state (the same path used when a trial is selected after the optimization), call get_merged_model(), temporarily point model.model at the merged operator, and run the regular evaluator.get_scores(). A fresh Context is created per trial, so there is no cache carry-over, and the live reference is restored in a finally.
  • Prints a live → merged delta report per scorer and persists the merged scores in study.user_attrs["merged_scores"].
  • Re-ranks the trial menu by merged scores only when every trial on the front was re-scored — mixing live and merged numbers in one ranking would be misleading — and shows both values as live → merged in the menu.

Memory stays bounded (at most one merged model exists at a time), runtime stays bounded (at most K additional small evaluations), and a failed per-trial evaluation only prints a warning and skips that trial, never losing the optimization result.

Tests (all run locally on darwin/arm64):

  • uv run --with pytest pytest tests/test_config.py tests/test_keyword_rate.py -q → 5 passed, 3 subtests passed (baseline unchanged).
  • New tests/test_merged_rescoring.py (fake study/model/evaluator/modifier): top-K re-scoring with the live weights restored afterwards, a failing trial skipped without interrupting the rest, K = 0 disabling the feature, and full-front coverage re-ranking by merged scores. Without the fix it fails at collection (ImportError: cannot import name 'rescore_top_trials_on_merged_models'); with the fix, 4 passed.
  • uv run --offline --with pytest pytest tests/test_config.py tests/test_keyword_rate.py tests/test_merged_rescoring.py -q → 9 passed, 3 subtests passed; uv run --offline python -m unittest discover -s tests -p 'test_*.py' → 9 tests OK.
  • CI commands pass: uv run ruff format --check . (25 files already formatted), uv run ruff check --extend-select I . and uv run --offline ty check --error-on-warning . both report all checks passed.

Not done here, stated as such: end-to-end verification with a real model (KLD/refusal deltas versus the --evaluate-model path) needs CUDA plus model downloads; this machine is darwin/arm64 with no GPU.

(This contribution was prepared with AI coding assistance, reviewed and tested by me.)

Trial scores come from PEFT's live-LoRA operator, which rounds
intermediate results differently than the merged operator of the
exported model, so the optimizer selects on numbers the exported
model does not reproduce (p-e-w#490).

After the study, re-score the top n_merged_score_trials trials
(default 3) on their merged models, print a live → merged delta
report, persist the merged scores in the study, re-rank the trial
menu by merged scores when the whole Pareto front was re-scored,
and show both scores in the menu.

Memory stays bounded (at most one merged model exists at a time)
and runtime stays bounded (at most K additional small evaluations);
per-trial merging during the optimization itself remains unchanged.
@p-e-w

p-e-w commented Oct 14, 2026

Copy link
Copy Markdown
Owner

Thanks for the PR, but this is not the correct fix. The trial scores already need to match the final score during the optimization phase, because divergent scores steer the optimizer towards a wrong parameter optimum. For a hack that doesn't solve the main issue, this is way too much code to add.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Help wanted] Trial scores differ from scores for the exported model

2 participants