Skip to content

Add cross-attention matrix capture for lyric alignment scoring #1

Open
ServeurpersoCom/acestep.cpp
#95

Description

@jeeftor

Title: Add cross-attention matrix capture for lyric alignment scoring

Problem

The Python ACE-Step implementation supports lyric alignment scoring via
AceStepHandler.get_lyric_score() (in acestep/core/generation/handler/lyric_score.py).
This runs a single DiT decoder forward pass with output_attentions=True to
capture cross-attention matrices, then computes DTW-based alignment metrics
(coverage, monotonicity, confidence) on CPU.

The C++ ace-server cannot support this today because of two gaps:

Gap 1: Cross-attention matrices are not materialized

dit_ggml_build_cross_attn() in src/dit-graph.h:275-359 uses
ggml_flash_attn_ext() for both self- and cross-attention (line 348).
Flash attention fuses softmax(QK^T)V into a single kernel and never
produces the attention weight matrix. The fallback dit_attn_f32()
(line 121-135) computes scores = ggml_soft_max_ext(...) but immediately
consumes it in the V multiplication — the scores tensor is not returned.

The Python DiT supports output_attentions=True on its cross-attention
layers, returning [n_layers, batch, n_heads, S, enc_S] matrices. There
is no C++ equivalent.

Gap 2: Conditioning tensors and lyric token IDs are discarded

SynthState in src/pipeline-synth-impl.h computes during /synth:

  • enc_hidden (stacked encoder hidden states) — used as K/V source for
    cross-attention, discarded after the DiT loop
  • context (context latents) — discarded after the DiT loop
  • per_enc_S (encoder sequence lengths) — discarded
  • output (post-DiT latent) — already returned as a multipart part

ops_encode_text() in src/pipeline-synth-ops.cpp:427-506 tokenizes
lyrics via bpe_encode() (line 473) and runs them through the text
encoder, but the raw lyric_ids are stored only in a local
TextEncForward struct (line 478) and never persisted to SynthState.
The Python scoring code needs these token IDs to slice the attention
matrix to lyric-token positions only.

This forces anyone needing lyric scoring to run a separate Python process
with a full second copy of the text encoder, VAE, and turbo DiT (~6GB
VRAM), duplicating models that ace-server already has loaded.

Proposed Solution

Add the ability to capture and return cross-attention matrices from the
DiT, plus the conditioning tensors and lyric token IDs needed for
alignment scoring, via a new POST /score endpoint.

What scoring needs

The Python get_lyric_score() does:

  1. Create two input latents: xt_lm (pure noise, t=1.0) and xt_dit
    (partially denoised: t_last * noise + (1-t_last) * pred_latent,
    where t_last = 1.0 / inference_steps)
  2. Concatenate them (batch=2) and run a single DiT decoder forward
    pass with output_attentions=True on all cross-attn layers
  3. Extract cross-attention matrices: [n_layers, 2, n_heads, S, enc_S]
  4. Slice to lyric token positions: [..., start_idx:end_idx, :]
  5. Compute DTW pathfinding + coverage/monotonicity/confidence metrics
    on CPU (numpy + numba)

Steps 1-4 require the DiT and conditioning tensors. Step 5 is pure CPU
tensor math.

Design: hybrid approach

ace-server (C++) handles steps 1-4:

  • Accepts pred_latent + request JSON (caption, lyrics, duration,
    vocal_language, audio_codes, inference_steps, seed) via POST /score
  • Reuses the in-memory DiT (already loaded for /synth)
  • Runs a single scoring forward pass (batch=2, no diffusion loop)
  • Captures cross-attention matrices from all layers
  • Slices to lyric token positions server-side (reduces payload from
    ~2.8GB to ~1.2MB for a 30s track)
  • Returns sliced attention matrices + lyric token IDs as multipart

Client (Python, CPU-only) handles step 5:

  • Runs DTW + metrics on the received matrices
  • No GPU models needed — pure numpy + numba

Memory footprint analysis

For a 30-second vocal track:

  • n_layers = 24, n_heads ≈ 16, batch = 2
  • S (patchified sequence) = (30 × 25) / 2 = 375
  • enc_S (encoder sequence) ≈ 256 (text + lyric tokens)
  • lyric_tokens ≈ 100 (after slicing to lyric segment)
Strategy Payload size Complexity
Full matrices (all positions, all heads) ~2.8 GB Simple C++, huge payload
Sliced to lyric tokens, all heads ~750 MB Medium C++, large payload
Sliced to lyric tokens + head selection (~6 layers × 1 head) ~1.2 MB More C++, minimal payload

The Python scoring code already applies custom_layers_config to select
specific layers/heads. Doing this selection in the C++ graph (only
mark those specific heads' scores as outputs via ggml_set_output())
keeps the footprint trivial.

Feasibility Verification

All key technical questions have been verified against the local source:

✅ Single forward pass without diffusion loop

dit_ggml_build_graph() (src/dit-graph.h:465) builds a graph for one
forward pass
. It takes a single timestep t and produces velocity.
The diffusion loop (dit_ggml_generate() in dit-sampler.h) calls this
graph repeatedly. For scoring, we call it once with N=2 (noise +
partially denoised). The graph is already parameterized by batch size N.

✅ Capture attention scores via ggml_set_output()

dit_attn_f32() (src/dit-graph.h:121-135) computes:

struct ggml_tensor * scores = ggml_mul_mat(ctx, k, q);       // QK^T
scores = ggml_soft_max_ext(ctx, scores, mask, scale, 0.0f);  // softmax(QK^T)
struct ggml_tensor * out = ggml_mul_mat(ctx, vt, scores);     // scores @ V

The scores tensor is the attention weight matrix. Adding
ggml_set_output(scores) tells the ggml scheduler to compute and retain
it. This is a standard ggml pattern already used throughout the graph
for debug tensors (e.g., hidden_after_layer0 at line 582).

✅ Lyric token IDs accessible

bpe_encode() at src/pipeline-synth-ops.cpp:473 computes lyric_ids
but stores them only in a local TextEncForward struct. Persisting them
to SynthState is a one-line addition.

✅ Conditioning tensors accessible

SynthState (src/pipeline-synth-impl.h:46-120) already has enc_hidden,
context, per_enc_S — all computed during /synth and currently
discarded.

✅ Job queue reusable for new endpoint

The job queue in tools/ace-server.cpp (work_push, Job struct,
/job polling) is generic. Adding POST /score follows the same pattern
as POST /synth — create a job, push to queue, return job ID, client
polls /job?id=N&result=1.

Implementation Areas

1. src/dit-graph.h — attention capture path

  • Add bool output_attentions and std::vector<ggml_tensor*> * attn_out
    parameters to dit_ggml_build_cross_attn() (line 275)
  • When output_attentions=true, use dit_attn_f32() instead of
    ggml_flash_attn_ext() and call ggml_set_output(scores) on the
    attention weight tensor
  • Build a separate scoring graph function
    (dit_ggml_build_scoring_graph) that runs all 24 layers with
    attention capture, single forward pass, no diffusion loop
  • Optionally accept a custom_layers_config to only mark specific
    layer/head combinations as outputs (reduces memory from ~2.8GB to ~1MB)

2. src/pipeline-synth-impl.h — retain lyric token IDs

  • Add std::vector<std::vector<int>> per_lyric_ids to SynthState
  • Populate in ops_encode_text() alongside per_enc

3. src/pipeline-synth-ops.cpp — scoring forward op

  • New ops_score_forward() that:
    • Builds the batch=2 input (noise + partially denoised)
    • Runs the scoring graph
    • Extracts and slices attention matrices to lyric token positions
  • Needs enc_hidden, context, per_enc_S from SynthState
  • Two options for getting these:
    • (a) Re-run text encoding + context building from the request JSON
      (simpler, self-contained, but duplicates work)
    • (b) Accept conditioning tensors as inputs from the client (client
      got them from a modified /synth response — avoids re-computation
      but couples the two endpoints)

4. tools/ace-server.cpp — new POST /score endpoint

  • Accepts latent (multipart) + request JSON
  • Reuses the existing job queue (work_push, Job, /job polling)
  • Loads DiT + text encoder + cond encoder (same as /synth minus VAE)
  • Returns multipart: sliced attention matrices (f32) + lyric token IDs

5. tools/synth-batch-runner.h — optional tensor capture

  • If approach (b) is chosen, extend synth_batch_run() to also capture
    enc_hidden, context, per_lyric_ids alongside the existing
    latents_out

Open Questions

  1. Re-run vs. cache conditioning: Should /score re-run text
    encoding + context building from the request JSON (simpler, self-
    contained, but duplicates work), or should /synth be modified to
    return conditioning tensors that /score then accepts as inputs
    (avoids re-computation but couples the two endpoints)?

  2. Attention matrix format: Return raw
    [n_layers, 2, n_heads, S_lyric, enc_S] f32 bytes (simple, large),
    or pre-process further (e.g., average over heads, apply
    custom_layers_config selection) to reduce payload?

  3. Graph build cost: Building a separate scoring graph with
    output_attentions=true on all 24 layers means a second ggml graph
    compile. Is this acceptable, or should the existing graph be
    parameterized to optionally capture attention?

Context

  • Python scoring: acestep/core/generation/handler/lyric_score.py and
    acestep/core/scoring/dit_score.py in ace-step/ACE-Step-1.5
  • SynthState struct: src/pipeline-synth-impl.h:46-120
  • Cross-attn graph: src/dit-graph.h:273-359
  • F32 attention fallback: src/dit-graph.h:121-135
  • DiT forward graph: src/dit-graph.h:465-614
  • Text encoding op: src/pipeline-synth-ops.cpp:427-506
  • HTTP server + job queue: tools/ace-server.cpp
  • Batch runner: tools/synth-batch-runner.h
  • Multipart builder: tools/ace-server.cpp (multipart_build_audio_latent)

Activity

  1. added a commit that references this issue on Jul 10, 2026
    2b38981
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions