You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I would like to run this model consistently in memory without loading it each time and I can't run it as a command line. It is useful. Is there a serial version in plan?
I could not make it run either. I have always had the same error
python setup_env.py -md ~/.cache/llama.cpp/microsoft_bitnet-b1.58-2B-4T-gguf_ggml-model-i2_s.gguf -q i2_s
Traceback (most recent call last):
File "/home/zezen/Downloads/BitNet/setup_env.py", line 232, in <module>
main()
File "/home/zezen/Downloads/BitNet/setup_env.py", line 208, in main
gen_code()
File "/home/zezen/Downloads/BitNet/setup_env.py", line 188, in gen_code
raise NotImplementedError()
NotImplementedError
Tips and random musings:
Try: cmake --build build --config Release, as the build files were ready but not built yet.
Lots of warnings:
3rdparty/llama.cpp/ggml/src/ggml-cpu/ops.cpp:7:
/home/zezen/Downloads/BitNet/3rdparty/llama.cpp/ggml/src/ggml-cpu/vec.h:412:16: warning: compound literals are a C99-specific feature [-Wc99-extensions]
y[i] = GGML_FP32_TO_FP16(v * fminf(1.0f, fmaxf(0.0f, (v + 3.0f) / 6.0f)));
^
note: expanded from macro 'GGML_FP32_TO_FP16'
#define GGML_FP32_TO_FP16(x) GGML_COMPUTE_FP32_TO_FP16(x)
^
Update, I have pasted also my musings to Grok and it (rightly methinks) claims that we are using the wrong GGUF files and are not isolating the pythons there.
Grok (trimmed, sorry for this raw paste):
...
Root Causes
Let’s dissect the causes based on the error and your observations:
Model Name Mismatch in setup_env.py:
The gen_code() function expects the model name to match one of the predefined values. The model file path (~/.cache/llama.cpp/microsoft_bitnet-b1.58-2B-4T-gguf_ggml-model-i2_s.gguf) suggests you’re using the BitNet-b1.58-2B-4T model, but get_model_name() (likely parsing the model path or metadata) might not return the expected string (BitNet-b1.58-2B-4T).
Possible reasons:
The model file’s metadata (e.g., GGUF headers) doesn’t match what setup_env.py expects.
The gguf package or llama.cpp version is outdated, causing incorrect parsing of the model file.
The path format or naming convention confuses the script.
GGUF Quantization Type Error:
The gguf-dump error indicates that the i2_s quantization type (enum value 36) is not recognized by gguf version 0.16.2. This suggests that the GGUF file uses a newer quantization format introduced in a later version of llama.cpp or BitNet’s fork of llama.cpp.
The BitNet README specifies using i2_s for quantization [Web:1], but your gguf package or llama.cpp library might not support it due to version mismatches.
...
When setup_env.py runs, Python’s dynamic linker (ld) might load the wrong libggml-base.so (e.g., from /usr/local/lib or ~/.local/lib), causing version incompatibilities. This could break GGUF parsing or kernel generation.
Clang Version:
BitNet requires Clang 18 for optimal performance and compatibility [Web:3]. Your Clang 14 build succeeded, but the generated libraries might lack specific optimizations or features required by setup_env.py’s code generation.
Steps to Resolve the Issue
To fix the NotImplementedError, GGUF quantization error, and library conflicts, follow these steps systematically:
Update the gguf Package:
The gguf package version 0.16.2 is outdated and doesn’t support the i2_s quantization type (enum 36). Update to the latest version or install BitNet’s fork of gguf-py:
bash
pip uninstall gguf
pip install --no-cache-dir -r ~/Downloads/BitNet/requirements.txt
pip install ~/Downloads/BitNet/3rdparty/llama.cpp/gguf-py
Verify the version:
bash
pip show gguf
The latest version (as of April 2025) should be >=0.17.0 or a custom fork from BitNet.
Isolate the Python Environment:
To avoid library conflicts, create a clean Conda environment as recommended by the BitNet README [Web:3]:
bash
conda create -n bitnet-cpp python=3.9
conda activate bitnet-cpp
pip install -r ~/Downloads/BitNet/requirements.txt
pip install ~/Downloads/BitNet/3rdparty/llama.cpp/gguf-py
This ensures setup_env.py uses the correct gguf package and avoids loading incompatible libraries from ~/.local/lib.
Fix Model Path and Naming:
The NotImplementedError suggests get_model_name() doesn’t recognize the model. The model path (~/.cache/llama.cpp/microsoft_bitnet-b1.58-2B-4T-gguf_ggml-model-i2_s.gguf) is unusual, as it’s nested in ~/.cache/llama.cpp and has a non-standard name.
Download the model directly to a clean directory:
bash
mkdir -p ~/Downloads/BitNet/models/BitNet-b1.58-2B-4T
huggingface-cli download microsoft/BitNet-b1.58-2B-4T-gguf --local-dir ~/Downloads/BitNet/models/BitNet-b1.58-2B-4T
Run setup_env.py with the correct path:
bash
python setup_env.py -md ~/Downloads/BitNet/models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -q i2_s
If the model name still fails, explicitly specify the Hugging Face repo:
bash
python setup_env.py --hf-repo microsoft/BitNet-b1.58-2B-4T -md ~/Downloads/BitNet/models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -q i2_s
Resolve Library Conflicts:
To prevent setup_env.py from loading the wrong libggml-base.so, set the LD_LIBRARY_PATH to prioritize BitNet’s libraries:
bash
export LD_LIBRARY_PATH=~/Downloads/BitNet/build/bin:$LD_LIBRARY_PATH
python setup_env.py -md ~/Downloads/BitNet/models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -q i2_s
Alternatively, uninstall conflicting llama-cpp-python packages:
bash
pip uninstall llama-cpp-python llama-cpp-scripts
If you need llama.cpp separately, keep its libraries isolated (e.g., don’t install to /usr/local/lib).
Rebuild with Clang 18:
Your Clang 14 build works but may lack BitNet-specific optimizations. Install Clang 18 as recommended [Web:3]:
bash
sudo bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"
Clean and rebuild:
bash
cd ~/Downloads/BitNet
rm -rf build
cmake -S . -B build -DLLAMA_BUILD_SERVER=ON -DCMAKE_C_COMPILER=clang-18 -DCMAKE_CXX_COMPILER=clang++-18
cmake --build build --config Release
This ensures compatibility with BitNet’s optimized kernels.
Verify GGUF File:
Check the GGUF file’s metadata to confirm it’s compatible:
bash
python -m gguf.tools dump ~/Downloads/BitNet/models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf
If the gguf-dump error persists, the GGUF file may be from a newer llama.cpp version. Update BitNet’s llama.cpp submodule:
bash
cd ~/Downloads/BitNet
git submodule update --init --recursive
rm -rf build
cmake -S . -B build -DLLAMA_BUILD_SERVER=ON -DCMAKE_C_COMPILER=clang-18 -DCMAKE_CXX_COMPILER=clang++-18
cmake --build build --config Release
Patch setup_env.py (Temporary Workaround):
[I skip this one ;) and pages of more interesting and sound general advice ]
cpp/bin/activate
(bitnet-cpp) lco@rtx:/mnt/nvme0n1/LLM/git/BitNet$ python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s
INFO:root:Compiling the code using CMake.
INFO:root:Loading model from directory models/BitNet-b1.58-2B-4T.
INFO:root:Converting HF model to GGUF format...
ERROR:root:Error occurred while running command: Command '['/mnt/nvme0n1/LLM/git/BitNet/bitnet-cpp/bin/python', 'utils/convert-hf-to-gguf-bitnet.py', 'models/BitNet-b1.58-2B-4T', '--outtype', 'f32']' returned non-zero exit status 1., check details in logs/convert_to_f32_gguf.log
(bitnet-cpp) lco@rtx:/mnt/nvme0n1/LLM/git/BitNet$ cat logs/convert_to_f32_gguf.log
INFO:hf-to-gguf:Loading model: BitNet-b1.58-2B-4T
Traceback (most recent call last):
File "/mnt/nvme0n1/LLM/git/BitNet/utils/convert-hf-to-gguf-bitnet.py", line 1165, in
main()
File "/mnt/nvme0n1/LLM/git/BitNet/utils/convert-hf-to-gguf-bitnet.py", line 1143, in main
model_class = Model.from_model_architecture(hparams["architectures"][0])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/nvme0n1/LLM/git/BitNet/utils/convert-hf-to-gguf-bitnet.py", line 240, in from_model_architecture
raise NotImplementedError(f'Architecture {arch!r} not supported!') from None
NotImplementedError: Architecture 'BitNetForCausalLM' not supported!
Are you trying to convert the model by yourself? I'd suggest you download the model from HF and run it directly.
please follow the instruction to download the model (python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s). model name is sensitive since it will check if the model exists and if model name is different, it will trigger conversion which may fail. thanks.
python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s
INFO:root:Compiling the code using CMake.
INFO:root:Loading model from directory models/BitNet-b1.58-2B-4T.
INFO:root:GGUF model already exists at models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf
But:
python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s
INFO:root:Compiling the code using CMake.
INFO:root:Loading model from directory models/BitNet-b1.58-2B-4T.
INFO:root:GGUF model already exists at models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf
and the session freezes.
Quick debugging, over my morning coffee:
build/bin/llama-cli --version
version: 3955 (a8ac7072)
built with Ubuntu clang version 14.0.0-1ubuntu1.1 for x86_64-pc-linux-gnu
so it finally got patched, as vs.:
/usr/local/bin/llama-cli --version
version: 5172 (eb1776b1)
built with Ubuntu clang version 14.0.0-1ubuntu1.1 for x86_64-pc-linux-gnu
Let us try it by hand:
ls build/bin/llama-cli
build/bin/llama-cli
so only this can be used to test the right gguf.
For the sake of it, let us try the "old" gguf dumper gguf-dump gguf-dump models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf before the potential new session freeze :
.local/lib/python3.10/site-packages/gguf/gguf_reader.py", line 151, in _get
.newbyteorder(override_order or self.byte_order)
AttributeError: `newbyteorder` was removed from the ndarray class in NumPy 2.0. Use `arr.view(arr.dtype.newbyteorder(order))` instead
a promising new error this time, so maybe something got patched in the previous step.
Let us try the command (stripped somehow) by hand:
build/bin/llama-cli -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -n 128 -t 2 -p 'You are a helpful assistant' -cnv
build: 3955 (a8ac7072) with Ubuntu clang version 14.0.0-1ubuntu1.1 for x86_64-pc-linux-gnu
main: llama backend init
main: load the model and apply lora adapter, if any
Segmentation fault
Interestinger and interestinger, as all run at the vanilla almost:
uname -a
Linux above-hp2-silver 6.2.0-39-generic #40~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 16 10:53:04 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
My interim guess (thanks to Grok too):
Library Mismatch: The llama-cli binary is using incompatible libraries (libllama.so, libggml.so) from /usr/local/lib, which are from llama.cpp (version 5172) and don’t support BitNet’s i2_s quantization or model format.
as:
please follow the instruction to download the model (python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s). model name is sensitive since it will check if the model exists and if model name is different, it will trigger conversion which may fail. thanks.
python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s INFO:root:Compiling the code using CMake. INFO:root:Loading model from directory models/BitNet-b1.58-2B-4T. INFO:root:GGUF model already exists at models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf
llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'bitnet-b1.58'
llama_load_model_from_file: failed to load model
common_init_from_params: failed to load model 'models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf'
srv load_model: failed to load model, 'models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf'
main: exiting due to model loading error
Re: git pull --recurse-submodules - that newer version would be the most elegant if it would work with https://huggingface.co/microsoft/bitnet-b1.58-2B-4T-gguf that has puzzled us all, without virtualization or holding left ear with the right hand, all backwards.
Instead of any cmake (does not work), I am trying as per the RTFM: huggingface-cli download microsoft/BitNet-b1.58-2B-4T-gguf --local-dir models/BitNet-b1.58-2B-4T and then
python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s
INFO:root:Compiling the code using CMake.
INFO:root:Loading model from directory models/BitNet-b1.58-2B-4T.
INFO:root:GGUF model already exists at models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf
Compiles, seemingly both in Ubuntu and Termux.
But dejavu:
python run_inference.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -p "You are a helpful assistant" -cnv
build: 3957 (5eb47b72) with Ubuntu clang version 14.0.0-1ubuntu1.1 for x86_64-pc-linux-gnu
main: llama backend init
main: load the model and apply lora adapter, if any
Error occurred while running command: Command '['build/bin/llama-cli', '-m', 'models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf', '-n', '128', '-t', '2', '-p', 'You are a helpful assistant', '-ngl', '0', '-c', '2048', '--temp', '0.8', '-b', '1', '-cnv']' died with <Signals.SIGSEGV: 11>.
so I shall try on another virgin machine, after all.
export LD_LIBRARY_PATH=$(pwd)/build/3rdparty/llama.cpp/src/:$(pwd)/build/3rdparty/llama.cpp//ggml/src/:$LD_LIBRARY_PATH to disable the old .so file clashes.
Update: a note to myself mostly: doing the above on Droid this time. I have decoded what that setup_env.py file is doing. All the log output hidden by hard pipe to logs, no tee to terminal, needlessly. I am trying this then: cmake sped up by 8 times or more and debugging possible finally:
# Create log directory
mkdir -p logs
# 1. setup_gguf(): Install gguf-py package
python -m pip install 3rdparty/llama.cpp/gguf-py
# 2. gen_code(): Generate code for BitNet-b1.58-2B-4T with i2_s quantization
python utils/codegen_tl1.py --model bitnet_b1_58-3B --BM 160,320,320 --BK 64,128,64 --bm 32,64,32
# 3. compile(): Compile the codebase with CMake
cmake -B build -DBITNET_ARM_TL1=ON
cmake --build build --config Release --parallel 8
# 4. prepare_model(): Convert model to GGUF format (i2_s quantization)
# Check if model directory exists (assumed to exist as per -md argument)
# Convert to f32 GGUF
python utils/convert-hf-to-gguf-bitnet.py models/BitNet-b1.58-2B-4T --outtype f32
# Quantize f32 to i2_s
./build/bin/llama-quantize models/BitNet-b1.58-2B-4T/ggml-model-f32.gguf models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf I2_S 1
FYi, after experimenting with the above, to clear up some PKG_CONFIG , LD_PATH like weirdnesses in my prooted Debian, it finally compiles and runs there, even the usual way. Yipee!
Re the needed export LD_LIBRARY_PATH=$(pwd)/build/3rdparty/llama.cpp/src/:$(pwd)/build/3rdparty/llama.cpp//ggml/src/:$LD_LIBRARY_PATH trick so that it does not run into segmentation fault. It works, but as I forget about it now and then -
ChatGPT suggests that it all should have been compiled with rpath, e.g.
✅ Modify CMakeLists.txt (Top Level)
Place this after setting the output directories (CMAKE_RUNTIME_OUTPUT_DIRECTORY, etc.):
# Use $ORIGIN so binaries find libs relative to their location
set(CMAKE_SKIP_BUILD_RPATH FALSE)
set(CMAKE_BUILD_WITH_INSTALL_RPATH FALSE)
set(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE)
set(CMAKE_INSTALL_RPATH "$ORIGIN/../lib") # or $ORIGIN if libs are next to binaries
💡 Explanation:
$ORIGIN = path of the binary at runtime
../lib = if your .so files are in a lib/ folder next to the bin/ folder
Ensures libraries are found without setting LD_LIBRARY_PATH
The maintainers may think adding some version thereof .
Activity
sd983527 commentedon Apr 22, 2025
please refer to #78
CementZhang commentedon Apr 22, 2025
To compile the server:
cmake -S . -B build -DLLAMA_BUILD_SERVER=ONManamama commentedon Apr 24, 2025
I could not make it run either. I have always had the same error
Tips and random musings:
Try:
cmake --build build --config Release, as the build files were ready but not built yet.Lots of warnings:
yet it compiles fine:
and still the same python error, due to:
which (as unpatched, I presume) , cause:
I suspect that we both have the "regular" llama-cli etc. built separately:
, so the .so files etc. clash.
Manamama commentedon Apr 24, 2025
Update, I have pasted also my musings to Grok and it (rightly methinks) claims that we are using the wrong GGUF files and are not isolating the pythons there.
Grok (trimmed, sorry for this raw paste):
[I skip this one ;) and pages of more interesting and sound general advice ]
sd983527 commentedon Apr 24, 2025
Are you trying to convert the model by yourself? I'd suggest you download the model from HF and run it directly.
sd983527 commentedon Apr 24, 2025
please follow the instruction to download the model (python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s). model name is sensitive since it will check if the model exists and if model name is different, it will trigger conversion which may fail. thanks.
Manamama commentedon Apr 24, 2025
Re:
Indeed, we may have been also trying to convert the wrong models, hand downloaded ones.
Let us retry, almost comme il faut (yet without virtualization still, no conda):
So far so good as:
But:
and the session freezes.
Quick debugging, over my morning coffee:
Let us try it by hand:
so only this can be used to test the right gguf.
For the sake of it, let us try the "old" gguf dumper
gguf-dump gguf-dump models/BitNet-b1.58-2B-4T/ggml-model-i2_s.ggufbefore the potential new session freeze :Let us try the command (stripped somehow) by hand:
stracing it:
Interestinger and interestinger, as all run at the vanilla almost:
My interim guess (thanks to Grok too):
Library Mismatch: The llama-cli binary is using incompatible libraries (libllama.so, libggml.so) from /usr/local/lib, which are from llama.cpp (version 5172) and don’t support BitNet’s i2_s quantization or model format.
as:
grctest commentedon Apr 25, 2025
@gnusupport seems relevant #226
sd983527 commentedon Apr 25, 2025
are you loading the model on https://huggingface.co/microsoft/bitnet-b1.58-2B-4T-gguf ? there is an update to this model recently. and you need to git pull --recurse-submodules to pull the latest code changes.
Manamama commentedon Apr 25, 2025
Re:
git pull --recurse-submodules- that newer version would be the most elegant if it would work withhttps://huggingface.co/microsoft/bitnet-b1.58-2B-4T-ggufthat has puzzled us all, without virtualization or holding left ear with the right hand, all backwards.FYI, on Unix:
so :
and downloading from scratch seems to be needed:
Not a biggie - testing it now...
Manamama commentedon Apr 25, 2025
Done git cloning anew.
Instead of any
cmake(does not work), I am trying as per the RTFM:huggingface-cli download microsoft/BitNet-b1.58-2B-4T-gguf --local-dir models/BitNet-b1.58-2B-4Tand thenCompiles, seemingly both in Ubuntu and Termux.
But dejavu:
so I shall try on another virgin machine, after all.
Benjamin-Wegener commentedon Apr 26, 2025
try #204 please, should work for local use
Benjamin-Wegener commentedon May 1, 2025
so please close the issue if it worked for you
Manamama commentedon May 2, 2025
That: #206 (comment) helped - it compiles. After a couple of
git reset --hard origin/maintype operations, however, deja vu:After a chat with an AI or two, about:
(yes, on that old machine)
I have settled on :
export LD_LIBRARY_PATH=$(pwd)/build/3rdparty/llama.cpp/src/:$(pwd)/build/3rdparty/llama.cpp//ggml/src/:$LD_LIBRARY_PATHto disable the old .so file clashes.Only now it works on that other gguf:
Thanks.
Ver. 1.2
Manamama commentedon May 2, 2025
Update: a note to myself mostly: doing the above on Droid this time. I have decoded what that setup_env.py file is doing. All the log output hidden by hard pipe to logs, no tee to terminal, needlessly. I am trying this then: cmake sped up by 8 times or more and debugging possible finally:
....
Manamama commentedon May 5, 2025
FYi, after experimenting with the above, to clear up some PKG_CONFIG , LD_PATH like weirdnesses in my prooted Debian, it finally compiles and runs there, even the usual way. Yipee!
Manamama commentedon May 6, 2025
Re the needed
export LD_LIBRARY_PATH=$(pwd)/build/3rdparty/llama.cpp/src/:$(pwd)/build/3rdparty/llama.cpp//ggml/src/:$LD_LIBRARY_PATHtrick so that it does not run into segmentation fault. It works, but as I forget about it now and then -ChatGPT suggests that it all should have been compiled with rpath, e.g.
✅ Modify CMakeLists.txt (Top Level)
Place this after setting the output directories (CMAKE_RUNTIME_OUTPUT_DIRECTORY, etc.):
💡 Explanation:
$ORIGIN = path of the binary at runtime
../lib = if your .so files are in a lib/ folder next to the bin/ folder
Ensures libraries are found without setting LD_LIBRARY_PATH
The maintainers may think adding some version thereof .
Ver. 1.2