Skip to content

Upstream GGML patches and deprecate patched ggml. #66

Open

Description

@mmwillet

Initial discussion for this came up here and the current proposal, to upstream necessary operations directly to GGML or otherwise support them with custom operations and no longer require the patched version of ggml, represents the best long term outcome for TTS.cpp.

Doing so will allow TTS.cpp to reap the benefits of recent GGML updates and make compatibility with other GGML reliant technologies more seamless.

Our goal would be, as @tyrauber outlined, to upstream the following support to GGML:

Enhanced Conv Transpose 1D
Remove the p0 == 0 constraint and add support for non-zero padding. Maintain backward compatibility (existing calls with p0=0 work unchanged). This is the most critical blocker and affects multiple model types. The lack of full PyTorch/TensorFlow parity in GGML's convolution operations seems like a significant gap that would benefit the broader ecosystem.

This should also include support for non-zero output padding (which is straightforward to add).

Convolution Grouping Support
Add grouping parameter to convolution operations. Required by Kokoro and common in modern CNN architectures. Extend existing conv operations with optional grouping parameter.

Our requirement here would only be that the grouping parameter can be applied to convolutional transposition.

Enhanced Upscaling Operation
Extend ggml_upscale with multiple interpolation modes. Better PyTorch/TensorFlow parity, removes need for custom ggml_upscale_linear. Maybe nearest (existing), linear (1D sequences), bilinear, bicubic, trilinear, area, nearest-exact.

Additional modifications such as ggml_reciprocal, STFT, and iSTFT, would also be desirable but could initially be supported via custom map operations.

Please see the discussion linked above for posterity.

Activity

  1. mmwillet commented on May 29, 2025

    @mmwillet
    OwnerAuthor

    @tyrauber

    My primary target for this library is mobile devices, so I was trying to squeeze as much performance and acceleration out of the library as possible. As I dug into the code, I ran into all these questions. So, I really appreciate you sharing your time, knowledge and experience.

    Gotcha. In that case we will likely want to keep the reciprocal operation and add appropriate acceleration for it if we can't find another easy trick.

    @danielzgtg

    inline operations against a scalar

    ggml_reciprocal

    Do stride tricks work? There's no speedup for multiplication by 2, but a reciprocal is 1/x. In numpy the 1 is a 1-element n-dimensional array that can be broadcasted by setting the dimensions to anything and all the strides to 0. We can see if this works in ggml.

    The stride trick should work assuming that the division operation uses the tensor's nb to iterate and there isn't some assert that will catch us along the way. It is annoying this solution would force us to assign arbitrary scalars to 1 at load time, but if it works it would get us good acceleration across devices without having to implement ggml_reciprocal.

  2. mmwillet commented on May 29, 2025

    @mmwillet
    OwnerAuthor

    Added documentation on the ggml patch requirement here.

  3. mmwillet commented on May 29, 2025

    @mmwillet
    OwnerAuthor

    @danielzgtg tested out the stride hack that you mentioned and it doesn't work because ggml_new_tensor_impl raises an assert error when the produce of the tensor shape plus its offsets is less than the total computed bytes.

  4. danielzgtg commented on May 29, 2025

    @danielzgtg
    Collaborator

    stride hack [...] doesn't work

    --- a/src/util.cpp
    +++ b/src/util.cpp
    @@ -79,7 +79,12 @@ float round_to_float(double v) {
     // Snake1d is a common tunable activation function used in the DAC model.
     struct ggml_tensor * snake_1d(ggml_context * ctx, struct ggml_tensor * alpha, struct ggml_tensor * a) {
         assert(a->ne[2] == 1 && a->ne[3] == 1);
    -    return ggml_add(ctx, a, ggml_mul(ctx, ggml_sqr(ctx, ggml_sin(ctx, ggml_mul(ctx, a, alpha))), ggml_reciprocal(ctx, alpha)));
    +    static constexpr float one = 1;
    +    ggml_tensor * ones = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 1, alpha->ne[1]);
    +    ones->nb[1] = 0; // broadcast
    +    ones->data = const_cast<float *>(&one);
    +
    +    return ggml_add(ctx, a, ggml_mul(ctx, ggml_sqr(ctx, ggml_sin(ctx, ggml_mul(ctx, a, alpha))), ggml_div(ctx, ones, alpha)));
     }
     
     bool has_suffix(std::string value, std::string suffix) {

    Output sounds fine and doesn't fail any asserts.

    Although, reciprocal should still be upstreamed because a unary op is faster.

  5. mmwillet commented on May 29, 2025

    @mmwillet
    OwnerAuthor

    @danielzgtg my bad. I updated the step with ggml_view instead of modifying the tensor directly. That works well enough: #68

  6. danielzgtg commented on Jun 2, 2025

    @danielzgtg
    Collaborator

    I have a branch with the snake1d kernel fused: optimize/snake1d-fused-failed (7ab4720). The optimization was statistically insignificant, and I must have already squeezed out all the potential gains with my ffast-math ggml_sin PR. The bottleneck is convs1_weights build_kokoro_generator_res_block. The math wasn't simplifiable, and a dilation of 5+ rules out SIMD. This instead creates demand for GPU support, which is good at matmuls.

    ggml_mul(ctx, other, ggml_reciprocal(ctx, alpha)) can be simplified to ggml_div(ctx, other, alpha) in standard ggml. However, the benefit of reciprocals is that multiplication is faster than division, and there is a specific RCPSS x86 instruction for them, as well as reciprocals being able to be broadcasted from [K] to [IC, K].

  7. mmwillet commented on Jun 6, 2025

    @mmwillet
    OwnerAuthor

    @danielzgtg I agree gpu support should be a priority sooner rather than later, but I suspect that we have quite a bit to get through before that. Let me break down my current plan on how to approach all of this:

    1. Break out current patches to GGML into polished commits against the latest GGML branch (to be PRs against GGML main). This would include 3-5 commits:
      • Improvements to 1d Convolutional Transposition (including support for padding, output padding, and grouping)
      • Metal implementation of Convolutional Transposition.
      • Improvements to Interpolation (upscaling). This would include linear interpolation and possibly additional changes.
      • (Probably Optional) The addition of useful unary ops (if any exist). E.G. mod, reciprocal, cumsum.
      • (Optional) add support for STFT and iSTFT ops. I still haven't looked at the latest implementation in whisper so this might be refactorable
    2. Discretely PR those 3-5 commits against GGML main.
    3. Rebase to GGML latest and move any non-upstreamed work into TTS.cpp
    4. Refactor current model code to properly use the latest GGML branch.
    5. Refactor current model code to support one device GPU support.
    6. Refactor current model code for PP across GPUs.
  8. mmwillet commented on Jun 27, 2025

    @mmwillet
    OwnerAuthor

    I read through the current STFT behavior that exists in whisper.cpp and there are still some innate problems with it assuming we want model parity with torch.stft and torch.istft respectively. It has definitely improved dramatically from when I last read it (it is at least correctly applying the padding and bins now), but I suspect that it will be prohibitive to incorporate its approach without significant work.

    I may try to upstream the STFT and iSTFT ops in the long run, but in the near term I will work to incorporate them into TTS.cpp proper.

  9. henk717 commented on Oct 7, 2025

    @henk717

    Had a chat with someone who contributes to llamacpp upstream, and he pointed me to this : https://github.com/ggml-org/llama.cpp/blob/master/docs/ops.md
    That is the ops tracker that informs GPU backend developers if their own GPU backend is missing acceleration.
    So if you focus on upstreaming your ops to the wider GGML project GPU support should be easier to achieve afterwards as its then not your project having to do all this work by yourself, your ops will appear as missing GPU backend ops by upstreaming them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

enhancementNew feature or request

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions