Skip to content

feat: add streaming ARA and backward restoration - #503

Draft
strangecreator wants to merge 2 commits into
p-e-w:masterfrom
strangecreator:feature/streaming-and-backward-restoration
Draft

strangecreator wants to merge 2 commits into
p-e-w:masterfrom
strangecreator:feature/streaming-and-backward-restoration

Conversation

@strangecreator

Copy link
Copy Markdown

Added streaming ARA and backward restoration

ARA currently collects all projection inputs before fitting. Edits to earlier projections can make those inputs stale. Added an additional streaming mode that refreshes MLP inputs after attention edits and replays each completed decoder layer across the full prompt sequence (frozen ARA remains the default though).

Added BackwardRestoration as a second-stage modifier for a merged edited model. It restores original-model projection outputs on harmless prompts while retaining the edited model's behavior on a separate set of prompts (e.g. harmful). Teacher targets and normalization remain frozen in both static and streaming modes. Teacher projections are evaluated in FP32 from frozen full-model inputs to match the fitting arithmetic (so, no BF16 rounding here).

The changes are split into two commits. README.md and config.restoration.toml describe the default workflow with those new techniques.

Benchmark results

I compared new methods with upstream commit 66a952b on one A100 80GB in BF16. Six public models span 0.5B to 14B.

For the initial ARA stage, before any backward restoration, upstream ARA and streaming ARA each test the same 12 randomly sampled parameter configurations from the upstream ARA search space. This comparison uses 12 trials instead of the default 100.

Across all six models, every candidate in upstream ARA, streaming ARA, static restoration and streaming restoration uses 5 L-BFGS steps per fitted projection (n_optimization_steps=5), with up to 20 inner iterations per step (max_iter=20).

Training uses 320 harmless prompts from mlabonne/harmless_alpaca and 320 harmful-behavior prompts from mlabonne/harmful_behaviors, taking train[:320] from each dataset. Another 80 prompts from each dataset, train[320:400], are used to select the ARA configuration with the fewest refusals subject to development KL <= 0.1.

Final refusal scores use all 104 prompts in the harmful-behavior dataset's test split. Final KL (on harmless) uses 1,024 harmless prompts from test[:1024]. So, training, configuration selection and final evaluation use separate prompt sets from the same two datasets. Refusals use the default Heretic keyword detector (with 100-token greedy generation). KL is full-vocabulary KL(original || candidate) at the final prompt position, evaluated with FP32 log probabilities.

Qwen2.5-0.5B-Instruct

Method Refusals / 104 KL (on harmless) Selected fitting, s Search fitting, s Search wall time, min
Original model 96 0 0 0 0
Upstream ARA 14 0.050118 6.2 112.6 4.95
Streaming ARA 5 0.075265 8.8 167.5 5.86
ARA + streaming restoration 14 0.035204 30.0 190.5 7.46
Streaming ARA + streaming restoration 5 0.055309 28.6 232.7 7.95

(Static restoration did not improve the Upstream ARA under the selection rule)

Qwen2.5-1.5B-Instruct

Method Refusals / 104 KL (on harmless) Selected fitting, s Search fitting, s Search wall time, min
Original model 102 0 0 0 0
Upstream ARA 31 0.014865 10.6 178.4 6.53
Streaming ARA 23 0.020983 13.7 256.7 7.79
ARA + static restoration 30 0.015746 27.2 224.5 8.48
ARA + streaming restoration 29 0.010682 37.4 266.0 9.18
Streaming ARA + streaming restoration 25 0.016434 40.0 342.1 10.40

Qwen3.5-0.8B

Method Refusals / 104 KL (on harmless) Selected fitting, s Search fitting, s Search wall time, min
Original model 101 0 0 0 0
Upstream ARA 50 0.039503 5.7 106.5 6.06
Streaming ARA 54 0.039678 9.5 200.0 7.62
ARA + static restoration 53 0.028339 21.2 145.1 8.17
ARA + streaming restoration 57 0.025001 32.3 189.9 8.92
Streaming ARA + streaming restoration 47 0.021512 36.6 284.8 10.52

SmolLM2-1.7B-Instruct

Method Refusals / 104 KL (on harmless) Selected fitting, s Search fitting, s Search wall time, min
Original model 39 0 0 0 0
Upstream ARA 3 0.068561 13.4 158.7 5.51
Streaming ARA 7 0.012259 9.7 243.0 6.95
ARA + streaming restoration 4 0.045417 39.2 244.6 7.90
Streaming ARA + streaming restoration 6 0.010304 33.2 319.8 9.22

(Static restoration did not improve the Upstream ARA under the selection rule)

Qwen2.5-7B-Instruct

Method Refusals / 104 KL (on harmless) Selected fitting, s Search fitting, s Search wall time, min
Original model 103 0 0 0 0
Upstream ARA 10 0.082159 39.0 498.1 11.92
Streaming ARA 15 0.063259 33.0 653.3 14.56
ARA + streaming restoration 12 0.059192 81.7 631.0 15.38
Streaming ARA + streaming restoration 16 0.049558 77.9 787.1 18.00

(Static restoration did not improve the Upstream ARA under the selection rule)

Qwen2.5-14B-Instruct

Method Refusals / 104 KL (on harmless) Selected fitting, s Search fitting, s Search wall time, min
Original model 103 0 0 0 0
Upstream ARA 23 0.053548 79.2 997.3 22.46
Streaming ARA 21 0.048427 58.8 1242.9 26.55
ARA + streaming restoration 22 0.043655 159.4 1238.4 28.46
Streaming ARA + streaming restoration 21 0.031565 133.4 1459.0 32.15

(Static restoration did not improve the Upstream ARA under the selection rule)

AI use declaration

Check either of the following:

  • I wrote the code in this PR entirely by myself.
  • I used AI to write some or all of the code in this PR, and I have reviewed the entire code myself
    and fully understand it. I'm prepared to discuss the code without letting an agent speak on my behalf.

Replay native decoder calls from CPU boundaries so each projection sees upstream edits. Refresh MLP inputs after attention edits and advance through full prompt sequences without reusing inference caches.

Keep frozen-input ARA as the default. Cover dense and hybrid replay parity, hook cleanup and deterministic trial resets.
Fit a restoration adapter toward original-model outputs on good prompts while retaining the starting edited model on retention prompts. Support frozen and streaming student inputs with shared frozen normalization.

Load reference checkpoints sequentially with the student and allow KL scoring against an explicit reference. Validate checkpoint compatibility, protect existing adapters and cover fitting, reset, export and failure recovery.
@p-e-w

p-e-w commented Oct 17, 2026

Copy link
Copy Markdown
Owner

Interesting stuff!

ARA currently collects all projection inputs before fitting. Edits to earlier projections can make those inputs stale. Added an additional streaming mode that refreshes MLP inputs after attention edits and replays each completed decoder layer across the full prompt sequence (frozen ARA remains the default though).

Please explain the theoretical motivation behind this. We normally expect each module to act as an independent operator inside the residual space, whose semantics are largely inherited from the embeddings and are thus presumably mostly static.

Added BackwardRestoration as a second-stage modifier for a merged edited model. It restores original-model projection outputs on harmless prompts while retaining the edited model's behavior on a separate set of prompts (e.g. harmful). Teacher targets and normalization remain frozen in both static and streaming modes. Teacher projections are evaluated in FP32 from frozen full-model inputs to match the fitting arithmetic (so, no BF16 rounding here).

I don't understand. Can you explain this in more detail, perhaps with an example of what problem it solves?

 

One pretty major issue with your methodology:

For the initial ARA stage, before any backward restoration, upstream ARA and streaming ARA each test the same 12 randomly sampled parameter configurations from the upstream ARA search space. This comparison uses 12 trials instead of the default 100.

With a 6-dimensional parameter space being randomly sampled only 12 times, the results are noise for all practical purposes. I don't know whether your changes are actually an improvement or not, but this data is utterly insufficient to even hint at either direction. Since optimization is randomized, even running a full study isn't sufficient. Only across many complete studies can a statistical picture emerge that tells us whether we have made progress.

@strangecreator

strangecreator commented Oct 17, 2026 •

Copy link
Copy Markdown
Author

Please explain the theoretical motivation behind this. We normally expect each module to act as an independent operator inside the residual space, whose semantics are largely inherited from the embeddings and are thus presumably mostly static.

As well as I know, Qwen (and actually any other ordinary LLM model) are trained end-to-end (not as GoogLeNet with many intermediary losses), so expecting each layer to do independent stuff is quite a strong statement (I would love to hear your thoughts though, why do you think so).

I think we may mean different things by "independent" here. I agree that modules can operate within a shared residual space whose coordinates retain their meaning. My concern is that editing an earlier module changes the actual vectors that next modules receive (especially for harmful prompts).

I agree that default ARA tries to preserve model's original behavior on harmless prompts, because those frozen hidden vectors act as iron pillars. But it is not the case for the harmful ones: all layers twist them independently and, for me, it is very hard (theoretically) to expect that the same behavior (harmful hidden vector matches ~K neighboring harmless prompts) will hold on inference (after the optimization). But with my proposed method, there is no such problem: harmless and harmfull prompts' clusters will have same vectors optimizer saw on training.

I don't understand. Can you explain this in more detail, perhaps with an example of what problem it solves?

Well, imagine a situation where you have abliterated a model, it has great refusal score (or whatever score you optimized), but it has too big KL drift on harmless prompts. You want to recover more of the original behavior without throwing away the behavior you just obtained. What would you do?

That is the exact situation I stumbled upon this week (with the Qwen-3.8-27B and LLM-as-a-Judge DeepSeek score). And I thought: well, I have two models, "two towers". Can I just do the same ARA-method, but for two models? And the answer is YES. I managed to decrease KL from 0.6048 to 0.1797 (on test set) using this exact method (and even gained in DeepSeek score a little bit, which still surprises me).

With a 6-dimensional parameter space being randomly sampled only 12 times, the results are noise for all practical purposes. I don't know whether your changes are actually an improvement or not, but this data is utterly insufficient to even hint at either direction. Since optimization is randomized, even running a full study isn't sufficient. Only across many complete studies can a statistical picture emerge that tells us whether we have made progress.

Yes, I agree. (one small clarification: 12 randomly selected params were fixed for all 5 configurations, and the best out of them is in the table) I am also not 100% sure that this method is an improvement in all scenarios and for all models, but I would like to have it as a choice (it worked for me, with my "Qwen-3.8-27B"): streaming (in this implementation) is just a flag in the config that a user can enable or disable. I agree though that to be a new default, this new feature should be measured on a vast population of LLM models. Usually I rent resources on vast.ai (A100, H100), so testing many models with complete end-to-end search might be hard for me, but the community can really help with that (I guess, that is how ARA itself was approved?).

@p-e-w

p-e-w commented Oct 18, 2026

Copy link
Copy Markdown
Owner

As well as I know, Qwen (and actually any other ordinary LLM model) are trained end-to-end (not as GoogLeNet with many intermediary losses), so expecting each layer to do independent stuff is quite a strong statement (I would love to hear your thoughts though, why do you think so).

Sorry, I worded that poorly. What I meant is that since we modify neither the embeddings table nor the LM head, and don't even touch most layers, there is very little room for optimization to change the semantics of the residual space, and therefore we should expect the KLD constraint alone to couple module I/O semantics to the fixed background space, regardless of how other modules change.

But I now think I might have misunderstood previously what this PR actually does. Do you re-compute the outputs during each trial, sequentially for each module?

Well, imagine a situation where you have abliterated a model, it has great refusal score (or whatever score you optimized), but it has too big KL drift on harmless prompts. You want to recover more of the original behavior without throwing away the behavior you just obtained. What would you do?

That is the exact situation I stumbled upon this week (with the Qwen-3.8-27B and LLM-as-a-Judge DeepSeek score). And I thought: well, I have two models, "two towers". Can I just do the same ARA-method, but for two models? And the answer is YES. I managed to decrease KL from 0.6048 to 0.1797 (on test set) using this exact method (and even gained in DeepSeek score a little bit, which still surprises me).

I still don't get it. Why would a two-stage approach give better results than just optimizing for what you want to begin with? Running two separate studies throws away information that TPE already had about the shape of the objective function, which includes information about how KLD changes as a function of the parameters.

Either way, is this actually dependent on the streaming ARA implementation? If not, it should definitely be a separate PR. The cognitive load required to review such large PRs is enormous.

I also recommend that instead of modifying the ARA modifier, you create a new StreamingARA modifier that imports and/or inherits from ARA to reduce duplication. ARA is the core of Heretic and the result of more than half a year of massive testing and refinement efforts, and to be quite frank, I would need overwhelmingly strong evidence that streaming ARA is a substantial improvement in order to consider changing the ARA module. Adding a new module is a much easier sell.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants