Repository navigation
Description
Title: Add cross-attention matrix capture for lyric alignment scoring
Problem
The Python ACE-Step implementation supports lyric alignment scoring via
AceStepHandler.get_lyric_score() (in acestep/core/generation/handler/lyric_score.py).
This runs a single DiT decoder forward pass with output_attentions=True to
capture cross-attention matrices, then computes DTW-based alignment metrics
(coverage, monotonicity, confidence) on CPU.
The C++ ace-server cannot support this today because of two gaps:
Gap 1: Cross-attention matrices are not materialized
dit_ggml_build_cross_attn() in src/dit-graph.h:275-359 uses
ggml_flash_attn_ext() for both self- and cross-attention (line 348).
Flash attention fuses softmax(QK^T)V into a single kernel and never
produces the attention weight matrix. The fallback dit_attn_f32()
(line 121-135) computes scores = ggml_soft_max_ext(...) but immediately
consumes it in the V multiplication — the scores tensor is not returned.
The Python DiT supports output_attentions=True on its cross-attention
layers, returning [n_layers, batch, n_heads, S, enc_S] matrices. There
is no C++ equivalent.
Gap 2: Conditioning tensors and lyric token IDs are discarded
SynthState in src/pipeline-synth-impl.h computes during /synth:
enc_hidden(stacked encoder hidden states) — used as K/V source for
cross-attention, discarded after the DiT loopcontext(context latents) — discarded after the DiT loopper_enc_S(encoder sequence lengths) — discardedoutput(post-DiT latent) — already returned as a multipart part
ops_encode_text() in src/pipeline-synth-ops.cpp:427-506 tokenizes
lyrics via bpe_encode() (line 473) and runs them through the text
encoder, but the raw lyric_ids are stored only in a local
TextEncForward struct (line 478) and never persisted to SynthState.
The Python scoring code needs these token IDs to slice the attention
matrix to lyric-token positions only.
This forces anyone needing lyric scoring to run a separate Python process
with a full second copy of the text encoder, VAE, and turbo DiT (~6GB
VRAM), duplicating models that ace-server already has loaded.
Proposed Solution
Add the ability to capture and return cross-attention matrices from the
DiT, plus the conditioning tensors and lyric token IDs needed for
alignment scoring, via a new POST /score endpoint.
What scoring needs
The Python get_lyric_score() does:
- Create two input latents:
xt_lm(pure noise, t=1.0) andxt_dit
(partially denoised:t_last * noise + (1-t_last) * pred_latent,
wheret_last = 1.0 / inference_steps) - Concatenate them (batch=2) and run a single DiT decoder forward
pass withoutput_attentions=Trueon all cross-attn layers - Extract cross-attention matrices:
[n_layers, 2, n_heads, S, enc_S] - Slice to lyric token positions:
[..., start_idx:end_idx, :] - Compute DTW pathfinding + coverage/monotonicity/confidence metrics
on CPU (numpy + numba)
Steps 1-4 require the DiT and conditioning tensors. Step 5 is pure CPU
tensor math.
Design: hybrid approach
ace-server (C++) handles steps 1-4:
- Accepts
pred_latent+ request JSON (caption, lyrics, duration,
vocal_language, audio_codes, inference_steps, seed) viaPOST /score - Reuses the in-memory DiT (already loaded for
/synth) - Runs a single scoring forward pass (batch=2, no diffusion loop)
- Captures cross-attention matrices from all layers
- Slices to lyric token positions server-side (reduces payload from
~2.8GB to ~1.2MB for a 30s track) - Returns sliced attention matrices + lyric token IDs as multipart
Client (Python, CPU-only) handles step 5:
- Runs DTW + metrics on the received matrices
- No GPU models needed — pure numpy + numba
Memory footprint analysis
For a 30-second vocal track:
n_layers = 24,n_heads ≈ 16,batch = 2S(patchified sequence) =(30 × 25) / 2 = 375enc_S(encoder sequence) ≈256(text + lyric tokens)lyric_tokens≈100(after slicing to lyric segment)
| Strategy | Payload size | Complexity |
|---|---|---|
| Full matrices (all positions, all heads) | ~2.8 GB | Simple C++, huge payload |
| Sliced to lyric tokens, all heads | ~750 MB | Medium C++, large payload |
| Sliced to lyric tokens + head selection (~6 layers × 1 head) | ~1.2 MB | More C++, minimal payload |
The Python scoring code already applies custom_layers_config to select
specific layers/heads. Doing this selection in the C++ graph (only
mark those specific heads' scores as outputs via ggml_set_output())
keeps the footprint trivial.
Feasibility Verification
All key technical questions have been verified against the local source:
✅ Single forward pass without diffusion loop
dit_ggml_build_graph() (src/dit-graph.h:465) builds a graph for one
forward pass. It takes a single timestep t and produces velocity.
The diffusion loop (dit_ggml_generate() in dit-sampler.h) calls this
graph repeatedly. For scoring, we call it once with N=2 (noise +
partially denoised). The graph is already parameterized by batch size N.
✅ Capture attention scores via ggml_set_output()
dit_attn_f32() (src/dit-graph.h:121-135) computes:
struct ggml_tensor * scores = ggml_mul_mat(ctx, k, q); // QK^T
scores = ggml_soft_max_ext(ctx, scores, mask, scale, 0.0f); // softmax(QK^T)
struct ggml_tensor * out = ggml_mul_mat(ctx, vt, scores); // scores @ VThe scores tensor is the attention weight matrix. Adding
ggml_set_output(scores) tells the ggml scheduler to compute and retain
it. This is a standard ggml pattern already used throughout the graph
for debug tensors (e.g., hidden_after_layer0 at line 582).
✅ Lyric token IDs accessible
bpe_encode() at src/pipeline-synth-ops.cpp:473 computes lyric_ids
but stores them only in a local TextEncForward struct. Persisting them
to SynthState is a one-line addition.
✅ Conditioning tensors accessible
SynthState (src/pipeline-synth-impl.h:46-120) already has enc_hidden,
context, per_enc_S — all computed during /synth and currently
discarded.
✅ Job queue reusable for new endpoint
The job queue in tools/ace-server.cpp (work_push, Job struct,
/job polling) is generic. Adding POST /score follows the same pattern
as POST /synth — create a job, push to queue, return job ID, client
polls /job?id=N&result=1.
Implementation Areas
1. src/dit-graph.h — attention capture path
- Add
bool output_attentionsandstd::vector<ggml_tensor*> * attn_out
parameters todit_ggml_build_cross_attn()(line 275) - When
output_attentions=true, usedit_attn_f32()instead of
ggml_flash_attn_ext()and callggml_set_output(scores)on the
attention weight tensor - Build a separate scoring graph function
(dit_ggml_build_scoring_graph) that runs all 24 layers with
attention capture, single forward pass, no diffusion loop - Optionally accept a
custom_layers_configto only mark specific
layer/head combinations as outputs (reduces memory from ~2.8GB to ~1MB)
2. src/pipeline-synth-impl.h — retain lyric token IDs
- Add
std::vector<std::vector<int>> per_lyric_idstoSynthState - Populate in
ops_encode_text()alongsideper_enc
3. src/pipeline-synth-ops.cpp — scoring forward op
- New
ops_score_forward()that:- Builds the batch=2 input (noise + partially denoised)
- Runs the scoring graph
- Extracts and slices attention matrices to lyric token positions
- Needs
enc_hidden,context,per_enc_SfromSynthState - Two options for getting these:
- (a) Re-run text encoding + context building from the request JSON
(simpler, self-contained, but duplicates work) - (b) Accept conditioning tensors as inputs from the client (client
got them from a modified/synthresponse — avoids re-computation
but couples the two endpoints)
- (a) Re-run text encoding + context building from the request JSON
4. tools/ace-server.cpp — new POST /score endpoint
- Accepts latent (multipart) + request JSON
- Reuses the existing job queue (
work_push,Job,/jobpolling) - Loads DiT + text encoder + cond encoder (same as
/synthminus VAE) - Returns multipart: sliced attention matrices (f32) + lyric token IDs
5. tools/synth-batch-runner.h — optional tensor capture
- If approach (b) is chosen, extend
synth_batch_run()to also capture
enc_hidden,context,per_lyric_idsalongside the existing
latents_out
Open Questions
-
Re-run vs. cache conditioning: Should
/scorere-run text
encoding + context building from the request JSON (simpler, self-
contained, but duplicates work), or should/synthbe modified to
return conditioning tensors that/scorethen accepts as inputs
(avoids re-computation but couples the two endpoints)? -
Attention matrix format: Return raw
[n_layers, 2, n_heads, S_lyric, enc_S]f32 bytes (simple, large),
or pre-process further (e.g., average over heads, apply
custom_layers_configselection) to reduce payload? -
Graph build cost: Building a separate scoring graph with
output_attentions=trueon all 24 layers means a second ggml graph
compile. Is this acceptable, or should the existing graph be
parameterized to optionally capture attention?
Context
- Python scoring:
acestep/core/generation/handler/lyric_score.pyand
acestep/core/scoring/dit_score.pyin ace-step/ACE-Step-1.5 SynthStatestruct:src/pipeline-synth-impl.h:46-120- Cross-attn graph:
src/dit-graph.h:273-359 - F32 attention fallback:
src/dit-graph.h:121-135 - DiT forward graph:
src/dit-graph.h:465-614 - Text encoding op:
src/pipeline-synth-ops.cpp:427-506 - HTTP server + job queue:
tools/ace-server.cpp - Batch runner:
tools/synth-batch-runner.h - Multipart builder:
tools/ace-server.cpp(multipart_build_audio_latent)
Activity
feat: add POST /score endpoint for lyric alignment scoring