Repository navigation
[Z-Image] Apply the LoRA scale passed via joint_attention_kwargs - #15015
Merged
Merged
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Member
|
/diffusers-bot pytest tests/pipelines/z_image/test_z_image.py |
Contributor
|
✅ |
2 of 7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes #15007
joint_attention_kwargs={"scale": ...}was accepted and stored by the Z-Image pipelines but never forwarded to the transformer, andZImageTransformer2DModel.forwarddid not apply the LoRA scale, so a loaded LoRA always ran at full strength.This PR:
ZImageTransformer2DModel.forwardwith@apply_lora_scale("joint_attention_kwargs")and adds thejoint_attention_kwargsargument with its docstring entry. It goes last in the signature, so existing positional calls are unaffected.joint_attention_kwargs=self.joint_attention_kwargsto the transformer inZImagePipeline,ZImageControlNetPipeline,ZImageControlNetInpaintPipeline,ZImageImg2ImgPipeline,ZImageInpaintPipelineandZImageOmniPipeline.test_simple_inference_with_text_denoiser_lora_and_scale, as suggested in [Z-Image] joint_attention_kwargs={"scale": ...} is silently ignored when loading LoRA weights #15007 (comment). It also re-enablestest_set_adapters_match_attention_kwargs, which was skipped for the same reason (it checks thatscalepassed viajoint_attention_kwargsmatchesset_adapters()) and passes with this fix.Tests
Run locally on an NVIDIA RTX 5070 Ti (PyTorch 2.14.1+cu130).
The two re-enabled tests:
All Z-Image pipeline and transformer tests:
Reproduction script from #15007, on this branch:
Self-review notes
Verdict: READY, no blocking issues. No dead code added.
Left for the reviewer:
scaleis read fromjoint_attention_kwargs; other keys are accepted and ignored, since the Z-Image attention processors take no extra kwargs. This is stated in theforwarddocstring.forwardrather than next to the other keyword arguments, so positional calls that passpatch_size/f_patch_sizekeep working.joint_attention_kwargsas passed to theAttentionProcessor(text shared with other pipelines); for Z-Image onlyscaleis used. Left unchanged to keep the diff focused.modular_pipelines/z_image/denoise.py) does not expose attention kwargs at all.Before submitting
self-reviewskill on the diff?forwarddocstring)Who can review?
@sayakpaul