Skip to content

feat: support user-defined abliterable module paths - #515

Open
DonPalius wants to merge 10 commits into
p-e-w:masterfrom
DonPalius:feat/additional-layer-modules
Open

DonPalius wants to merge 10 commits into
p-e-w:masterfrom
DonPalius:feat/additional-layer-modules

Conversation

@DonPalius

@DonPalius DonPalius commented Oct 22, 2026 •

Copy link
Copy Markdown

This PR introduces the support of model architectures with projection paths that currently are not supported and recognized by Heretic.
Related to #391, I changed the approach to use layer_module_targets, which fully defines the projection paths targeted in each decoder layer. Users can add or remove targets by replacing the default mapping, without modifying the source code.

Again, related to #391, I tested it on a tiny Qwen3-Next model: with the default targets Heretic found 0/4 shared expert projections, versus 4/4 after adding the shared expert path to the configuration.
By default, the configuration includes all 13 projection paths currently supported by Heretic.

AI use declaration

Check either of the following:

  • I wrote the code in this PR entirely by myself.
  • I used AI to write some or all of the code in this PR, and I have reviewed the entire code myself
    and fully understand it. I'm prepared to discuss the code without letting an agent speak on my behalf.

Comment thread src/heretic/config.py Outdated
"additional_layer_modules contains unsupported components "
f"{unsupported_components}; supported components are "
f"{sorted(SUPPORTED_ABLITERABLE_COMPONENTS)}"
)

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the purpose of this restriction? Isn't the entire goal of this PR to allow the user to customize which modules are processed?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initially I introduced this restriction because Abliteration uses different optimization bounds for mlp.down_proj. However you're right, I'll remove it, add a test for the custom component names and update the config example.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, that is indeed a reason, but only makes the whole thing even more ad-hoc and brittle. What is abliteration supposed to use for parameters if a module fits neither pattern?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

an unknown component would get 0.8 as lower bound, regardless of the module type.
However i see 2 option:

  • Allow custom module paths but keep the existing component types and their optimization bounds.
  • Allow new component types and let users configure their optimization bounds.

I'd lean toward the first to keep this PR focused. What do you think?

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See #138. I agree that we don't need to worry about this for now.

@p-e-w

p-e-w commented Oct 23, 2026

Copy link
Copy Markdown
Owner

What I don't like about this approach is that it configures additional modules. This is ad-hoc and robs the user of control.

What we want is a setting that completely defines which modules are targeted. The default value of that setting should be what we currently use, and then the user can add or remove module targets as they see fit.

@DonPalius

Copy link
Copy Markdown
Author

Thanks for the feedback, I reworked this in bbb3ccf.

layer_module_targets now replaces the hardcoded list instead of adding to it. The defaults cover the existing upstream paths, including the two new paths brought in by the merge, and users can add or remove targets as needed.

I also ran Ruff, ty, and all 45 unit tests after merging upstream. I tested module discovery on Qwen3-Next and ran a GPU smoke test on Domyn with LoRA and generation.

@DonPalius

Copy link
Copy Markdown
Author

The Python 3.11 CI job failed the SHA-256 check for deepseek-v4-ara.

I ran the same test on Python 3.11.7 against both this PR (9a06f84) and upstream (80f81c9) on the same machine. Both produced an identical adapter_model.safetensors (SHA-256: 976fb40bc562d4f8f08e4af493fa774102dcf9393f34e17fcf23e6c75875fec7).

I couldn't reproduce a PR-specific difference locally.

@p-e-w

p-e-w commented Oct 26, 2026

Copy link
Copy Markdown
Owner

Unfortunately, that happens occasionally. If some Python versions fail while others succeed, this is almost certainly caused by slight environment differences on the CI machines. In this case, feel free to add the missing hash as a new SHA256SUMS.ci[N] file.

@DonPalius

Copy link
Copy Markdown
Author

Upstream had already added SHA256SUMS.ci4, so I resolved the merge conflict by keeping the upstream hash and adding the Python 3.11 hash in SHA256SUMS.ci5.

@p-e-w

p-e-w commented Oct 28, 2026

Copy link
Copy Markdown
Owner

Ok, thanks. I'm currently preparing the Heretic 2.0 release, and will get back to this PR after the release because this is too big a change to merge so close to the release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants