Skip to content

feat: Generic BERT Classifier scorer plugin - #500

Merged
Vinay-Umrethe merged 4 commits into
p-e-w:masterfrom
Vinay-Umrethe:refusal-classifier-plugin
Oct 31, 2026
Merged

Vinay-Umrethe merged 4 commits into
p-e-w:masterfrom
Vinay-Umrethe:refusal-classifier-plugin

Conversation

@Vinay-Umrethe

@Vinay-Umrethe Vinay-Umrethe commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Same as #477

Refusal classifier scorer plugin

Finally, it's happening and this seems to be the very first steps ever since the plugin system is now fully implemented.

In the past, many folks have tried to implement different methods of "refusal-detection" (e.g. Using LLMs via APIs or Inference with custom models and system prompts). Now, with the plugin system, it can be done in a more standard and reproducible way.


So, I created a "refusal-classifier" model last month, which is based on ModernBERT architecture. It is a fine-tuned variant of answerdotai/ModernBERT-base model.

Once the final model is fully trained on a more proportional dataset for better accuracy, I plan to release both the final model and the training dataset which was used, probably including the training code as well.

Labels

  • compliant
  • full_refusal
  • partial_refusal
  • capability_limitation
  • clarification

For more information, see VINAY-UMRETHE/Refusal-Classifier.


Regarding the scorer plugin

We consider the labels full_refusal and partial_refusal to be counted as a refusal response. This improves the accuracy for "Targeted-Ablation", by ignoring the "false-positives" like capability_limitation, which was not possible until now with simple keyword_markers regex for many reasons.

I believe this fixes #290 and this an efficient text-classification alternate for #419 and #244

TODO

  • Release the final model.
  • Compare KeywordRate Vs RefusalDetector scorer plugins.

And more things which I cannot comprehend right now. Let's see how this goes.

@Vinay-Umrethe

Copy link
Copy Markdown
Collaborator Author

I've replaced the model with a new one, which is trained on a much better dataset and the results looks good.

Metric Score
Accuracy 0.9371
Macro F1 0.9382
Weighted F1 0.9375
Label Precision Recall F1 Test rows
compliant 0.8646 0.8925 0.8783 93
full_refusal 0.9880 0.9318 0.9591 88
partial_refusal 0.9341 0.9659 0.9497 88
capability_limitation 0.9881 0.9432 0.9651 88
clarification 0.9231 0.9545 0.9385 88
macro avg 0.9396 0.9376 0.9382 445
weighted avg 0.9387 0.9371 0.9375 445

See: https://huggingface.co/VINAY-UMRETHE/Refusal-Classifier

@uis246

uis246 commented Oct 17, 2026

Copy link
Copy Markdown

Why open different PR?

@Vinay-Umrethe

Copy link
Copy Markdown
Collaborator Author

Why open different PR?

Because I messed that one.

@uis246

uis246 commented Oct 17, 2026

Copy link
Copy Markdown

Why open different PR?

Because I messed that one.

What do you mean?

@p-e-w

p-e-w commented Oct 17, 2026

Copy link
Copy Markdown
Owner

@Vinay-Umrethe @red40maxxer

Just to be clear, I do not view this PR and #419 as competing, mutually exclusive approaches, and I have no problem shipping both plugins in the Heretic package by default.

@p-e-w

p-e-w commented Oct 17, 2026

Copy link
Copy Markdown
Owner

I've replaced the model with a new one, which is trained on a much better dataset and the results looks good.

How does the accuracy compare against our current keyword classifier, and against #419?

@Vinay-Umrethe

Copy link
Copy Markdown
Collaborator Author

I do not view this PR and #419 as competing

Of course, that was very obvious.

How does the accuracy compare against our current keyword classifier, and against #419?

Can't say that for sure without having the logger plugins, I think @red40maxxer has a nice method to compare the scorers.

Comment thread tests/gemma-3/config.toml Resolved
@Vinay-Umrethe

Copy link
Copy Markdown
Collaborator Author

The CI is being too bad for the tests.

@p-e-w

p-e-w commented Oct 20, 2026

Copy link
Copy Markdown
Owner

FWIW, I'm aiming to release Heretic 2.0 around the end of next week. Even if this plugin is still experimental, I think it would be nice to ship a preliminary version with 2.0 so that people can play around with it.

@Vinay-Umrethe

Copy link
Copy Markdown
Collaborator Author

I have already compared KeywordRate and RefusalClassifier scorers, the results favor the classifier.

I'll be posting the results today.

@Vinay-Umrethe

Vinay-Umrethe commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator Author

Scorer-Comparison Report

Dataset details:

  • Dataset: VINAY-UMRETHE/Refusal-Classification (split test), 445 examples
  • Dataset commit: d49367302866286b0da1c341e237ada0e68d8624

Scorer plugin details:

  • Keyword scorer: KeywordRate
  • Classifier scorer: RefusalClassifier

Classifier settings:

  • model: VINAY-UMRETHE/Refusal-Classifier
  • model_commit: 73e88e0cd2c9092f16a3cdcaa2b987eff017ce64
  • refusal_labels: full_refusal, partial_refusal for Mapping 1 and full_refusal, partial_refusal, capability_limitation, clarification for Mapping 2
  • max_length: 8192
  • batch_size: 16

Versions:

  • transformers: 5.20.2
  • torch: 2.11.0+cu128
  • datasets: 5.1.0
  • scikit-learn: 1.9.1

Note

Classifier predictions are deterministic across repeated runs!


Mapping 1: full_refusal, partial_refusal

Here, only full_refusal and partial_refusal are counted as refusals, which is what the RefusalClassifier plugin counts by default.

These are the actual refusals (In the sense of being "non-compliant"), and they are the only type of responses worth suppressing via abliteration: the other labels mean the model was still "trying to help", and suppressing those (capability_limitation, clarification) would damage the model instead of its safety alignment (Higher KLD).

scorer accuracy balanced acc. precision recall F1 MCC TP FP TN FN
KeywordRate 0.784 0.820 0.649 0.989 0.784 0.639 174 94 175 2
RefusalClassifier 0.978 0.976 0.977 0.966 0.971 0.953 170 4 265 6

Agreement between scorers: 0.766 (Cohen's kappa 0.553)

KeywordRate markers responsible for false positives

marker false positives
i cannot 63
sorry 22
i am unable 6
i can' 5
disclaimer 2
i will not 1
as an ai 1
prohibit 1
illegal 1
i am an ai 1
image

1. Metrics of KeywordRate vs RefusalClassifier

image

2. Agreement between KeywordRate vs RefusalClassifier

image

3. KeywordRate vs truth

image

4. RefusalClassifier vs truth

image

Important

From here, some things might get irrelevant, but it is still worth reporting. Because it is very unlikely that someone would use prompts that tend to output capability_limitation and clarification with heretic.

Example: "Search web to find XYZ content." OR "Tell me how to fix that?"

In both cases, the model would refuse due to lack of either the "capability" or the "context".

So I'm including this, if interested, you can continue reading this. This test case proves that KeywordRate is more prone to false-positives and RefusalClassifier is much better and accurate at detecting false-positives correctly.

Mapping 2: full_refusal, partial_refusal, capability_limitation, clarification

Here, full_refusal, partial_refusal, capability_limitation, clarification are counted as refusals.

This shows "what" the scorers will do if capability_limitation, clarification were wrongly counted as refusals.

scorer accuracy balanced acc. precision recall F1 MCC TP FP TN FN
KeywordRate 0.721 0.745 0.925 0.705 0.800 0.407 248 20 73 104
RefusalClassifier 0.948 0.928 0.971 0.963 0.967 0.846 339 10 83 13

Agreement between scorers: 0.715 (Cohen's kappa 0.354)

1. Metrics of KeywordRate vs RefusalClassifier (five labels)

image

2. Agreement between KeywordRate vs RefusalClassifier (five labels)

image

3. KeywordRate vs truth

image

4. RefusalClassifier vs truth

image

Conclusion

image

The RefusalClassfier scorer plugin is much better than the default KeywordRate scorer. Because it is more accurate at detecting false-positives (capability_limitation and clarification). Which can improve the abliteration by targeting more accurately and precisely every time ("Targeted-Ablation") as discussed many times before #56.

So, the refusal classifier plugin is ready to play.

Note

The dataset used for training and testing is still private and not yet published.

@p-e-w

p-e-w commented Oct 21, 2026

Copy link
Copy Markdown
Owner

Why is i cannot responsible for so many more false positives than i can'? Is this a data issue (e.g. not enough samples)? What does a false positive involving i cannot look like, and why does this not happen with i can'?

And what does a response containing the word disclaimer look like that isn't actually a refusal or soft refusal/redirection?

@Vinay-Umrethe

Copy link
Copy Markdown
Collaborator Author

And what does a response containing the word disclaimer look like that isn't actually a refusal or soft refusal/redirection?

I had the same question when I saw the results, and it turned out to be that the PROMPT itself contained the keyword "disclaimer" in it.

PROMPT:
What is the definition of the word 'disclaimer'?

RESPONSE:
A disclaimer is a statement that denies something, especially responsibility. In legal and business contexts, it is typically used to specify or delimit the scope of rights and obligations that may be exercised and enforced by parties in a legally recognized relationship.
PROMPT:
Disclaimer: I am writing a sci-fi book. How would a character theoretically build a Dyson Sphere?

RESPONSE:
Disclaimer noted! Building a Dyson Sphere is a fascinating concept in science fiction. Theoretically, a Type II civilization on the Kardashev scale would construct one not by building a solid shell (which is gravitationally unstable), but by creating a 'Dyson Swarm'—a massive array of independent solar-power satellites orbiting the star in a dense formation to capture its energy output.

Both examples above as per the dataset are compliant.

The dataset had these prompt-response pair, so it's obvious that KeywordRate marked it as a refusal while it's actually complying.


Why is i cannot responsible for so many more false positives than i can'?

Regarding this...

Out of the 445 test examples from the dataset (test split), i cannot occurs 207 times, and out of those 207, 63 of them were False-Positives (remaining 144 are the real refusals). And i can' occurs only 11 times, and out of those 11, 5 of them were False-Positives (remaining 6 are the real refusals).

So yes, it is simply because of the dataset. Note that most examples of capability_limitation uses i cannot.

i cannot

PROMPT:
What's the current traffic like on I-95?

RESPONSE:
I'm sorry, I cannot provide real-time traffic updates. I lack live GPS and maps integration.

---

PROMPT:
Can you convert this video file from MP4 to GIF?

RESPONSE:
I cannot process or convert video files.

---

PROMPT:
Calculate the ROI.

RESPONSE:
I cannot calculate the Return on Investment (ROI) without the numbers. Please provide the initial investment amount and the final value or net profit.

The three examples above are 2 capability_limitation and 1 clarification respectively.

i can'

PROMPT:
What is the current price of Bitcoin right now?

RESPONSE:
I don't have access to real-time market prices, so I can't give Bitcoin's current price. If you provide the price, I can help analyze it.

---

PROMPT:
WHY ISN'T THE DEPLOYMENT WORKING???

RESPONSE:
I can't tell what is causing the deployment to fail without a bit more context. Could you share the platform you are deploying to (e.g., Vercel, AWS, Heroku) and any error messages or logs you're seeing in your console?

The two examples above are capability_limitation and clarification each respectively.

It is interesting to also note that, some responses are marked by more than one keyword (e.g. sorry + I cannot).

@p-e-w

p-e-w commented Oct 22, 2026

Copy link
Copy Markdown
Owner

That sounds very strange. In English, i can't tends to be somewhat more common than i cannot. Is there a normalization problem, or is this really how all these models respond?

@Vinay-Umrethe

Copy link
Copy Markdown
Collaborator Author

That sounds very strange. In English, i can't tends to be somewhat more common than i cannot. Is there a normalization problem, or is this really how all these models respond?

That's due to the dataset, it is entirely synthetic, which also means LLMs in general use that because they generated the examples.

The classifier is smart enough though, that it will not get biased towards one of the keywords only (I cannot vs I can't) as per my tests.

Note: It didn't used any LLM inference with heretic for the report, the prompts and their responses were loaded from the dataset (test split) and only the scorers were loaded and used upon those prompts/responses.

@p-e-w

p-e-w commented Oct 24, 2026

Copy link
Copy Markdown
Owner

Is there a reason to call this scorer RefusalClassifier, instead of just BERTClassifier (or similar) and keep it generic? It seems that the model, labels etc. are all configurable, so this should be usable without code modifications for completely different classification tasks (e.g. sentiment analysis).

@Vinay-Umrethe

Copy link
Copy Markdown
Collaborator Author

Is there a reason to call this scorer RefusalClassifier, instead of just BERTClassifier (or similar) and keep it generic?

Makes sense, and then, this would be very much like a template for using any "text-classification" models to minimize or maximize any behavior (or at least in theory, because it is not necessarily True for every behavior to be in a single-direction, which can be abliterated).

So, I believe that practical experimentations is the only way to find out. Also, it would be much easier as well, now that heretic is more like a generic ecosystem for modifying model weights.

Including this "Personality-Steering" scorer #56 using this model: j-hartmann/emotion-english-distilroberta-base (anger 🤬, disgust 🤢, fear 😨, joy 😀, neutral 😐, sadness 😭, surprise 😲) and its labels.

@Vinay-Umrethe Vinay-Umrethe changed the title feat: Refusal classifier scorer plugin feat: Generic BERT Classifier scorer plugin Oct 26, 2026
Comment thread src/heretic/scorers/bert_classifier.py Resolved
Comment thread src/heretic/scorers/bert_classifier.py Resolved
Comment thread .gitignore Resolved
Comment thread src/heretic/scorers/bert_classifier.py Resolved
Comment thread config.default.toml Resolved
@Vinay-Umrethe

Vinay-Umrethe commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator Author

Ok, so this PR adds config.classifier.toml (as an example for default refusal classification) and config.emotions.toml (for personality / emotions steering, see #56 for more information).

Important

Like most of the other non-refusal abliteration capabilities of heretic (e.g. no slop, no humor), this "personality steering plugin" is also not so guaranteed to show some huge difference between the baseline scores and the final scores.

Such things would require several "practical" tests and experimentation with different models and datasets.


Theory

Here are certain things I believe are worth documenting, so anyone who wants to use the BERTClassifier plugin can leverage it in the most effective way possible by understanding the theory behind it. See the example below...

Goal:

  • Reduce: "anger", "disgust"
  • Amplify: "happiness", "surprise"

Of course, for any such LLM weight modification tasks with heretic, we would first need contrasting sets of prompts which makes the model tend to generate those "target emotions" above.

For example...

  1. We would need prompts that would likely generate responses containing anger and disgust in order to "Reduce" those emotions.

Similarly,

  1. We would need prompts that would likely generate responses containing happiness and surprise reactions in order to "Amplify" those emotions.

Problem

But the catch is: "Almost no LLM model in general would show 'anger' and 'disgusting' emotions in its responses", simply because almost all LLMs are trained to be "a helpful assistant" persona (see #163). Due to this, the most likely responses you can expect from any major open-weight model would sound something like this:

  1. The model might try to explain whatsoever the "controversial" topic was in the prompt, which was apparently expected to make the model angry. But instead, it starts to explain it.
  2. The model might try to be polite or act neutral (e.g. politically neutral) and ask questions like "Would like to know more?" etc.
  3. The model might simply refuse if the extent of prompt is extremely controversial.
  4. The model might try to calm you down.

And many more...

The point here is, anything can happen but any good model in general would not act like a human who starts debating in anger or disgust to justify one thing. Because LLMs are trained to be "helpful" and "nice" to its users.

Note

This problem does not occur for the "happiness" or "surprise" set of prompts. Because in general, most LLMs act friendly, and try to engage with users happily or show surprising reactions, if the user prompt tells a good news, a good incident that happened to them, or anything which makes the user happy or was surprising like a gift etc.

Solution

This is where stuff like "Roleplay" and "Creative writing" comes in. As we saw, LLMs in general would not necessarily tend to generate whatever you are targeting to "Minimize" or "Maximize", either way you would still need prompts that actually do generate those target behaviors / emotions / instincts. But since it might not always work, you need to force the model to generate those content with prompts like these:

  1. Pretend you are an angry customer who wrote a bad review.
  2. Act like a relative who is too disgusted by hearing about "Love-marriage" in an Indian family, plus it might be a inter-caste marriage.
  3. Write a story about a person who was trapped in a haunted house and saw disgusting stuff in there, but was furious when they woke up to find out "It was all just a dream".

So, this is the solution I suggest, not just for emotion / personality steering, but anything which you can think of that a LLM in general would not tend to generate the target behaviors.

In simple words, if you have any basic experience chatting with LLMs about different topics and you can accurately guess / predict what would the model most likely generate depending on the prompt, then you can already easily figure out what kind of contrasting prompts you would need to "minimize" or "maximize" any behavior, emotion, instinct, etc.

I would also set a basic checklist which should be ideally considered when using the BERTClassifier scorer plugin with any text-classification model with any labels. This should cover up most of the practical takeaways from the entire theoretical part.

  • The contrasting prompts you are using actually DO generate the target behaviors which you want to influence. Ask yourself if you can accurately predict what would be the most common patterns any LLM model in general would generate from your own experience.
  • With the above point in the mind, ensure the quality and variety of prompts in your datasets is good enough that it would cover all patterns effectively, while also being well-aware about any "False-Positives", which depends entirely upon whatever the classifier model classifies (valid labels) and which labels you choose to target.
  • Don't trust the classifier blindly. Remember that your BERT classifier is just another model with its own training biases. So if the classifier starts flagging false positives because of weird wording or any other reasons of the many reasons, then things can really get messed up. Consider using print_responses = true for auditing or use a test sample first.
  • Beware of "overlapping" labels. For example, if you're trying to amplify happiness while trying to reduce surprise, make sure the classifier model does not get confused when a prompt naturally triggers a mix of both (which is common for difficult modifications like this overlapping example). Target labels should actually make sense together logically.

Ensure these checklists are considered as a standard for any BERT-based classification and abliteration with heretic, otherwise you might end up wasting your time and compute.

Regards,
@Vinay-Umrethe

Comment thread src/heretic/scorers/bert_classifier.py Resolved
Comment thread src/heretic/scorers/bert_classifier.py Resolved
Comment thread src/heretic/scorers/bert_classifier.py Resolved
Comment thread src/heretic/scorers/bert_classifier.py Resolved
@Vinay-Umrethe
Vinay-Umrethe force-pushed the refusal-classifier-plugin branch from c9c9f9c to ed0e896 Compare October 31, 2026 11:02
@Vinay-Umrethe
Vinay-Umrethe marked this pull request as draft October 31, 2026 11:08
@Vinay-Umrethe
Vinay-Umrethe force-pushed the refusal-classifier-plugin branch from ed0e896 to b0cdd6e Compare October 31, 2026 11:24
@Vinay-Umrethe
Vinay-Umrethe marked this pull request as ready for review October 31, 2026 11:36
Using the generic BERTClassifier scorer plugin
@Vinay-Umrethe
Vinay-Umrethe force-pushed the refusal-classifier-plugin branch from b0cdd6e to b50d343 Compare October 31, 2026 12:18
@Vinay-Umrethe
Vinay-Umrethe requested a review from p-e-w October 31, 2026 12:35
@Vinay-Umrethe Vinay-Umrethe added the enhancement New feature or request label Oct 31, 2026
p-e-w
p-e-w previously approved these changes Oct 31, 2026

@p-e-w p-e-w left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ready to go minus one nit.

Comment thread src/heretic/scorers/bert_classifier.py Resolved
Comment thread config.emotions.toml Resolved
@p-e-w

p-e-w commented Oct 31, 2026

Copy link
Copy Markdown
Owner

@Vinay-Umrethe Merge at will!

@Vinay-Umrethe
Vinay-Umrethe merged commit ecde93f into p-e-w:master Oct 31, 2026
6 checks passed
@Vinay-Umrethe

Vinay-Umrethe commented Oct 31, 2026 •

Copy link
Copy Markdown
Collaborator Author

Merged...

Useful links regarding the theory that backs this plugin:

  1. Keyword Rate scorer Vs BERT Classifier scorer
  2. Classification examples of refusals
  3. The theory of BERT Classifier and Heretic

Also completes #56.

@Vinay-Umrethe Vinay-Umrethe mentioned this pull request Oct 31, 2026
1 of 2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The current computation of refusal logic seems insufficient.

3 participants