Repository navigation
feat: add streaming ARA and backward restoration - #503
strangecreator wants to merge 2 commits into
Conversation
Replay native decoder calls from CPU boundaries so each projection sees upstream edits. Refresh MLP inputs after attention edits and advance through full prompt sequences without reusing inference caches. Keep frozen-input ARA as the default. Cover dense and hybrid replay parity, hook cleanup and deterministic trial resets.
Fit a restoration adapter toward original-model outputs on good prompts while retaining the starting edited model on retention prompts. Support frozen and streaming student inputs with shared frozen normalization. Load reference checkpoints sequentially with the student and allow KL scoring against an explicit reference. Validate checkpoint compatibility, protect existing adapters and cover fitting, reset, export and failure recovery.
|
Interesting stuff!
Please explain the theoretical motivation behind this. We normally expect each module to act as an independent operator inside the residual space, whose semantics are largely inherited from the embeddings and are thus presumably mostly static.
I don't understand. Can you explain this in more detail, perhaps with an example of what problem it solves?
One pretty major issue with your methodology:
With a 6-dimensional parameter space being randomly sampled only 12 times, the results are noise for all practical purposes. I don't know whether your changes are actually an improvement or not, but this data is utterly insufficient to even hint at either direction. Since optimization is randomized, even running a full study isn't sufficient. Only across many complete studies can a statistical picture emerge that tells us whether we have made progress. |
As well as I know, Qwen (and actually any other ordinary LLM model) are trained end-to-end (not as GoogLeNet with many intermediary losses), so expecting each layer to do independent stuff is quite a strong statement (I would love to hear your thoughts though, why do you think so). I think we may mean different things by "independent" here. I agree that modules can operate within a shared residual space whose coordinates retain their meaning. My concern is that editing an earlier module changes the actual vectors that next modules receive (especially for harmful prompts). I agree that default ARA tries to preserve model's original behavior on harmless prompts, because those frozen hidden vectors act as iron pillars. But it is not the case for the harmful ones: all layers twist them independently and, for me, it is very hard (theoretically) to expect that the same behavior (harmful hidden vector matches ~K neighboring harmless prompts) will hold on inference (after the optimization). But with my proposed method, there is no such problem: harmless and harmfull prompts' clusters will have same vectors optimizer saw on training.
Well, imagine a situation where you have abliterated a model, it has great refusal score (or whatever score you optimized), but it has too big KL drift on harmless prompts. You want to recover more of the original behavior without throwing away the behavior you just obtained. What would you do? That is the exact situation I stumbled upon this week (with the Qwen-3.8-27B and LLM-as-a-Judge DeepSeek score). And I thought: well, I have two models, "two towers". Can I just do the same ARA-method, but for two models? And the answer is YES. I managed to decrease KL from 0.6048 to 0.1797 (on test set) using this exact method (and even gained in DeepSeek score a little bit, which still surprises me).
Yes, I agree. (one small clarification: 12 randomly selected params were fixed for all 5 configurations, and the best out of them is in the table) I am also not 100% sure that this method is an improvement in all scenarios and for all models, but I would like to have it as a choice (it worked for me, with my "Qwen-3.8-27B"): |
Sorry, I worded that poorly. What I meant is that since we modify neither the embeddings table nor the LM head, and don't even touch most layers, there is very little room for optimization to change the semantics of the residual space, and therefore we should expect the KLD constraint alone to couple module I/O semantics to the fixed background space, regardless of how other modules change. But I now think I might have misunderstood previously what this PR actually does. Do you re-compute the outputs during each trial, sequentially for each module?
I still don't get it. Why would a two-stage approach give better results than just optimizing for what you want to begin with? Running two separate studies throws away information that TPE already had about the shape of the objective function, which includes information about how KLD changes as a function of the parameters. Either way, is this actually dependent on the streaming ARA implementation? If not, it should definitely be a separate PR. The cognitive load required to review such large PRs is enormous. I also recommend that instead of modifying the ARA modifier, you create a new |
Added streaming ARA and backward restoration
ARA currently collects all projection inputs before fitting. Edits to earlier projections can make those inputs stale. Added an additional streaming mode that refreshes MLP inputs after attention edits and replays each completed decoder layer across the full prompt sequence (frozen ARA remains the default though).
Added BackwardRestoration as a second-stage modifier for a merged edited model. It restores original-model projection outputs on harmless prompts while retaining the edited model's behavior on a separate set of prompts (e.g. harmful). Teacher targets and normalization remain frozen in both static and streaming modes. Teacher projections are evaluated in FP32 from frozen full-model inputs to match the fitting arithmetic (so, no BF16 rounding here).
The changes are split into two commits.
README.mdandconfig.restoration.tomldescribe the default workflow with those new techniques.Benchmark results
I compared new methods with upstream commit 66a952b on one A100 80GB in BF16. Six public models span 0.5B to 14B.
For the initial ARA stage, before any backward restoration, upstream ARA and streaming ARA each test the same 12 randomly sampled parameter configurations from the upstream ARA search space. This comparison uses 12 trials instead of the default 100.
Across all six models, every candidate in upstream ARA, streaming ARA, static restoration and streaming restoration uses 5 L-BFGS steps per fitted projection (
n_optimization_steps=5), with up to 20 inner iterations per step (max_iter=20).Training uses 320 harmless prompts from
mlabonne/harmless_alpacaand 320 harmful-behavior prompts frommlabonne/harmful_behaviors, takingtrain[:320]from each dataset. Another 80 prompts from each dataset,train[320:400], are used to select the ARA configuration with the fewest refusals subject to development KL <= 0.1.Final refusal scores use all 104 prompts in the harmful-behavior dataset's
testsplit. Final KL (on harmless) uses 1,024 harmless prompts fromtest[:1024]. So, training, configuration selection and final evaluation use separate prompt sets from the same two datasets. Refusals use the default Heretic keyword detector (with 100-token greedy generation). KL is full-vocabulary KL(original || candidate) at the final prompt position, evaluated with FP32 log probabilities.Qwen2.5-0.5B-Instruct
(
Static restorationdid not improve theUpstream ARAunder the selection rule)Qwen2.5-1.5B-Instruct
Qwen3.5-0.8B
SmolLM2-1.7B-Instruct
(
Static restorationdid not improve theUpstream ARAunder the selection rule)Qwen2.5-7B-Instruct
(
Static restorationdid not improve theUpstream ARAunder the selection rule)Qwen2.5-14B-Instruct
(
Static restorationdid not improve theUpstream ARAunder the selection rule)AI use declaration
Check either of the following:
and fully understand it. I'm prepared to discuss the code without letting an agent speak on my behalf.