You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Upstream GGML patches and deprecate patched ggml. #66
Initial discussion for this came up here and the current proposal, to upstream necessary operations directly to GGML or otherwise support them with custom operations and no longer require the patched version of ggml, represents the best long term outcome for TTS.cpp.
Doing so will allow TTS.cpp to reap the benefits of recent GGML updates and make compatibility with other GGML reliant technologies more seamless.
Our goal would be, as @tyrauber outlined, to upstream the following support to GGML:
Enhanced Conv Transpose 1D
Remove the p0 == 0 constraint and add support for non-zero padding. Maintain backward compatibility (existing calls with p0=0 work unchanged). This is the most critical blocker and affects multiple model types. The lack of full PyTorch/TensorFlow parity in GGML's convolution operations seems like a significant gap that would benefit the broader ecosystem.
This should also include support for non-zero output padding (which is straightforward to add).
Convolution Grouping Support
Add grouping parameter to convolution operations. Required by Kokoro and common in modern CNN architectures. Extend existing conv operations with optional grouping parameter.
Our requirement here would only be that the grouping parameter can be applied to convolutional transposition.
Enhanced Upscaling Operation
Extend ggml_upscale with multiple interpolation modes. Better PyTorch/TensorFlow parity, removes need for custom ggml_upscale_linear. Maybe nearest (existing), linear (1D sequences), bilinear, bicubic, trilinear, area, nearest-exact.
Additional modifications such as ggml_reciprocal, STFT, and iSTFT, would also be desirable but could initially be supported via custom map operations.
My primary target for this library is mobile devices, so I was trying to squeeze as much performance and acceleration out of the library as possible. As I dug into the code, I ran into all these questions. So, I really appreciate you sharing your time, knowledge and experience.
Gotcha. In that case we will likely want to keep the reciprocal operation and add appropriate acceleration for it if we can't find another easy trick.
Do stride tricks work? There's no speedup for multiplication by 2, but a reciprocal is 1/x. In numpy the 1 is a 1-element n-dimensional array that can be broadcasted by setting the dimensions to anything and all the strides to 0. We can see if this works in ggml.
The stride trick should work assuming that the division operation uses the tensor's nb to iterate and there isn't some assert that will catch us along the way. It is annoying this solution would force us to assign arbitrary scalars to 1 at load time, but if it works it would get us good acceleration across devices without having to implement ggml_reciprocal.
I have a branch with the snake1d kernel fused: optimize/snake1d-fused-failed (7ab4720). The optimization was statistically insignificant, and I must have already squeezed out all the potential gains with my ffast-math ggml_sin PR. The bottleneck is convs1_weightsbuild_kokoro_generator_res_block. The math wasn't simplifiable, and a dilation of 5+ rules out SIMD. This instead creates demand for GPU support, which is good at matmuls.
ggml_mul(ctx, other, ggml_reciprocal(ctx, alpha)) can be simplified to ggml_div(ctx, other, alpha) in standard ggml. However, the benefit of reciprocals is that multiplication is faster than division, and there is a specific RCPSS x86 instruction for them, as well as reciprocals being able to be broadcasted from [K] to [IC, K].
@danielzgtg I agree gpu support should be a priority sooner rather than later, but I suspect that we have quite a bit to get through before that. Let me break down my current plan on how to approach all of this:
Break out current patches to GGML into polished commits against the latest GGML branch (to be PRs against GGML main). This would include 3-5 commits:
Improvements to 1d Convolutional Transposition (including support for padding, output padding, and grouping)
Metal implementation of Convolutional Transposition.
Improvements to Interpolation (upscaling). This would include linear interpolation and possibly additional changes.
(Probably Optional) The addition of useful unary ops (if any exist). E.G. mod, reciprocal, cumsum.
(Optional) add support for STFT and iSTFT ops. I still haven't looked at the latest implementation in whisper so this might be refactorable
Discretely PR those 3-5 commits against GGML main.
Rebase to GGML latest and move any non-upstreamed work into TTS.cpp
Refactor current model code to properly use the latest GGML branch.
Refactor current model code to support one device GPU support.
I read through the current STFT behavior that exists in whisper.cpp and there are still some innate problems with it assuming we want model parity with torch.stft and torch.istft respectively. It has definitely improved dramatically from when I last read it (it is at least correctly applying the padding and bins now), but I suspect that it will be prohibitive to incorporate its approach without significant work.
I may try to upstream the STFT and iSTFT ops in the long run, but in the near term I will work to incorporate them into TTS.cpp proper.
Had a chat with someone who contributes to llamacpp upstream, and he pointed me to this : https://github.com/ggml-org/llama.cpp/blob/master/docs/ops.md
That is the ops tracker that informs GPU backend developers if their own GPU backend is missing acceleration.
So if you focus on upstreaming your ops to the wider GGML project GPU support should be easier to achieve afterwards as its then not your project having to do all this work by yourself, your ops will appear as missing GPU backend ops by upstreaming them.
Activity
mmwillet commentedon May 29, 2025
@tyrauber
Gotcha. In that case we will likely want to keep the reciprocal operation and add appropriate acceleration for it if we can't find another easy trick.
@danielzgtg
The stride trick should work assuming that the division operation uses the tensor's
nbto iterate and there isn't some assert that will catch us along the way. It is annoying this solution would force us to assign arbitrary scalars to 1 at load time, but if it works it would get us good acceleration across devices without having to implementggml_reciprocal.mmwillet commentedon May 29, 2025
Added documentation on the ggml patch requirement here.
mmwillet commentedon May 29, 2025
@danielzgtg
tested out the stride hack that you mentioned and it doesn't work becauseggml_new_tensor_implraises an assert error when the produce of the tensor shape plus its offsets is less than the total computed bytes.danielzgtg commentedon May 29, 2025
Output sounds fine and doesn't fail any asserts.
Although, reciprocal should still be upstreamed because a unary op is faster.
mmwillet commentedon May 29, 2025
@danielzgtg my bad. I updated the step with ggml_view instead of modifying the tensor directly. That works well enough: #68
danielzgtg commentedon Jun 2, 2025
I have a branch with the snake1d kernel fused:
optimize/snake1d-fused-failed(7ab4720). The optimization was statistically insignificant, and I must have already squeezed out all the potential gains with my ffast-math ggml_sin PR. The bottleneck isconvs1_weightsbuild_kokoro_generator_res_block. The math wasn't simplifiable, and a dilation of 5+ rules out SIMD. This instead creates demand for GPU support, which is good at matmuls.ggml_mul(ctx, other, ggml_reciprocal(ctx, alpha))can be simplified toggml_div(ctx, other, alpha)in standard ggml. However, the benefit of reciprocals is that multiplication is faster than division, and there is a specific RCPSS x86 instruction for them, as well as reciprocals being able to be broadcasted from[K]to[IC, K].mmwillet commentedon Jun 6, 2025
@danielzgtg I agree gpu support should be a priority sooner rather than later, but I suspect that we have quite a bit to get through before that. Let me break down my current plan on how to approach all of this:
mmwillet commentedon Jun 27, 2025
I read through the current STFT behavior that exists in whisper.cpp and there are still some innate problems with it assuming we want model parity with
torch.stftandtorch.istftrespectively. It has definitely improved dramatically from when I last read it (it is at least correctly applying the padding and bins now), but I suspect that it will be prohibitive to incorporate its approach without significant work.I may try to upstream the STFT and iSTFT ops in the long run, but in the near term I will work to incorporate them into TTS.cpp proper.
henk717 commentedon Oct 7, 2025
Had a chat with someone who contributes to llamacpp upstream, and he pointed me to this : https://github.com/ggml-org/llama.cpp/blob/master/docs/ops.md
That is the ops tracker that informs GPU backend developers if their own GPU backend is missing acceleration.
So if you focus on upstreaming your ops to the wider GGML project GPU support should be easier to achieve afterwards as its then not your project having to do all this work by yourself, your ops will appear as missing GPU backend ops by upstreaming them.