Skip to content

[Z-Image] Apply the LoRA scale passed via joint_attention_kwargs - #15015

Merged
sayakpaul merged 2 commits into
huggingface:mainfrom
Sergasgr:zimage-lora-scale
Oct 13, 2026
Merged

sayakpaul merged 2 commits into
huggingface:mainfrom
Sergasgr:zimage-lora-scale

Conversation

@Sergasgr

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes #15007

joint_attention_kwargs={"scale": ...} was accepted and stored by the Z-Image pipelines but never forwarded to the transformer, and ZImageTransformer2DModel.forward did not apply the LoRA scale, so a loaded LoRA always ran at full strength.

This PR:

  • decorates ZImageTransformer2DModel.forward with @apply_lora_scale("joint_attention_kwargs") and adds the joint_attention_kwargs argument with its docstring entry. It goes last in the signature, so existing positional calls are unaffected.
  • passes joint_attention_kwargs=self.joint_attention_kwargs to the transformer in ZImagePipeline, ZImageControlNetPipeline, ZImageControlNetInpaintPipeline, ZImageImg2ImgPipeline, ZImageInpaintPipeline and ZImageOmniPipeline.
  • re-enables test_simple_inference_with_text_denoiser_lora_and_scale, as suggested in [Z-Image] joint_attention_kwargs={"scale": ...} is silently ignored when loading LoRA weights #15007 (comment). It also re-enables test_set_adapters_match_attention_kwargs, which was skipped for the same reason (it checks that scale passed via joint_attention_kwargs matches set_adapters()) and passes with this fix.

Tests

Run locally on an NVIDIA RTX 5070 Ti (PyTorch 2.14.1+cu130).

The two re-enabled tests:

$ pytest tests/pipelines/z_image/test_z_image.py -k "match_attention_kwargs or denoiser_lora_and_scale" -v
tests/pipelines/z_image/test_z_image.py::TestZImagePipelineLoRA::test_simple_inference_with_text_denoiser_lora_and_scale PASSED [ 50%]
tests/pipelines/z_image/test_z_image.py::TestZImagePipelineLoRA::test_set_adapters_match_attention_kwargs PASSED [100%]

All Z-Image pipeline and transformer tests:

$ pytest tests/pipelines/z_image/ tests/models/transformers/test_models_transformer_z_image.py
195 passed, 37 skipped, 37 warnings in 221.52s (0:03:41)

Reproduction script from #15007, on this branch:

scale=0.0 identical to LoRA at full strength: False
scale=0.0 identical to no LoRA: True

Self-review notes

Verdict: READY, no blocking issues. No dead code added.

Left for the reviewer:

  1. Only scale is read from joint_attention_kwargs; other keys are accepted and ignored, since the Z-Image attention processors take no extra kwargs. This is stated in the forward docstring.
  2. The new argument is placed last in forward rather than next to the other keyword arguments, so positional calls that pass patch_size/f_patch_size keep working.
  3. The pipeline docstrings describe joint_attention_kwargs as passed to the AttentionProcessor (text shared with other pipelines); for Z-Image only scale is used. Left unchanged to keep the diff focused.
  4. Out of scope: the modular Z-Image denoise block (modular_pipelines/z_image/denoise.py) does not expose attention kwargs at all.

Before submitting

Who can review?

@sayakpaul

@sayakpaul sayakpaul left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet!

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@sayakpaul

Copy link
Copy Markdown
Member

/diffusers-bot pytest tests/pipelines/z_image/test_z_image.py

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

✅ pytest tests/pipelines/z_image/test_z_image.py passed on GPU — view logs.

@sayakpaul
sayakpaul merged commit 916a579 into huggingface:main Oct 13, 2026
12 checks passed
@sayakpaul sayakpaul added this to the Release 0.42.0 milestone Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

[Z-Image] joint_attention_kwargs={"scale": ...} is silently ignored when loading LoRA weights

3 participants