Repository navigation
fix: score top trials on merged models to close live-LoRA/export gap - #495
Open
akushonkamen wants to merge 1 commit into
Open
akushonkamen wants to merge 1 commit into
akushonkamen wants to merge 1 commit into
Conversation
Trial scores come from PEFT's live-LoRA operator, which rounds intermediate results differently than the merged operator of the exported model, so the optimizer selects on numbers the exported model does not reproduce (p-e-w#490). After the study, re-score the top n_merged_score_trials trials (default 3) on their merged models, print a live → merged delta report, persist the merged scores in the study, re-rank the trial menu by merged scores when the whole Pareto front was re-scored, and show both scores in the menu. Memory stays bounded (at most one merged model exists at a time) and runtime stays bounded (at most K additional small evaluations); per-trial merging during the optimization itself remains unchanged.
Owner
|
Thanks for the PR, but this is not the correct fix. The trial scores already need to match the final score during the optimization phase, because divergent scores steer the optimizer towards a wrong parameter optimum. For a hack that doesn't solve the main issue, this is way too much code to add. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #490.
Trial scores come from the live-LoRA operator: during the study, the Optuna
objectiverunsevaluator.get_scores()on the model thatget_peft_modelattached the adapters to, while the exported model runs the merged operator frommerge_and_unload()(Model.get_merged_model()). The two apply different rounding orders — live isy = R(R(xWᵀ) + R(R(xAᵀ)Bᵀ)), merged isy = R(x·R(W+BA)ᵀ)— so the optimizer selects on numbers the exported model does not reproduce.A CPU micro-repro of exactly these two formulas (torch, bf16) confirms it: the live and merged results diverge on most elements (this machine: max |live − merged| = 0.125, 138/256 elements differ; the magnitude depends on the random draw, but the divergence is always present).
Per the issue, per-trial baking is too high a performance price, so this leaves trial-time scoring untouched and closes the gap after the study, before the Pareto menu is built:
n_merged_score_trialstrials (new setting, default 3,0disables) on their merged models: restore the trial's live-LoRA state (the same path used when a trial is selected after the optimization), callget_merged_model(), temporarily pointmodel.modelat the merged operator, and run the regularevaluator.get_scores(). A freshContextis created per trial, so there is no cache carry-over, and the live reference is restored in afinally.study.user_attrs["merged_scores"].live → mergedin the menu.Memory stays bounded (at most one merged model exists at a time), runtime stays bounded (at most K additional small evaluations), and a failed per-trial evaluation only prints a warning and skips that trial, never losing the optimization result.
Tests (all run locally on darwin/arm64):
uv run --with pytest pytest tests/test_config.py tests/test_keyword_rate.py -q→ 5 passed, 3 subtests passed (baseline unchanged).tests/test_merged_rescoring.py(fake study/model/evaluator/modifier): top-K re-scoring with the live weights restored afterwards, a failing trial skipped without interrupting the rest,K = 0disabling the feature, and full-front coverage re-ranking by merged scores. Without the fix it fails at collection (ImportError: cannot import name 'rescore_top_trials_on_merged_models'); with the fix, 4 passed.uv run --offline --with pytest pytest tests/test_config.py tests/test_keyword_rate.py tests/test_merged_rescoring.py -q→ 9 passed, 3 subtests passed;uv run --offline python -m unittest discover -s tests -p 'test_*.py'→ 9 tests OK.uv run ruff format --check .(25 files already formatted),uv run ruff check --extend-select I .anduv run --offline ty check --error-on-warning .both report all checks passed.Not done here, stated as such: end-to-end verification with a real model (KLD/refusal deltas versus the
--evaluate-modelpath) needs CUDA plus model downloads; this machine is darwin/arm64 with no GPU.(This contribution was prepared with AI coding assistance, reviewed and tested by me.)