Bitnet Ternary MoE GGUF
A 1GB Mixture-of-Experts model designed to run efficiently on an everyday laptop CPU, with no GPU needed.
Model source
- Kind
- Other
- File size
- 52.0 MB–1.03 GB
- Base model
- microsoft/bitnet-b1.58-2B-4T
- Version
- vRevision 4e75887
- Updated
- Oct 27, 2026
- Downloads
- 223
- Source
- Hugging Face
Source description
bitnet-ternary-moe-gguf 🧮
A 1GB Mixture-of-Experts model designed to run efficiently on an everyday laptop CPU, with no GPU needed.
A single 1.58-bit BitNet base model with two lightweight LoRA expert adapters (multiplication and Roman numerals) that are dynamically activated at runtime by an external router. The base model is loaded once and remains unchanged, reducing memory usage by approximately 5× compared to an equivalent ~5 GB bf16 setup.
➡️ Code, launcher, eval harness and full setup guide: github.com/agentulukaADMIN/ternary-moe
Files in this repo
| file | size | what it is |
|---|---|---|
bitnet-2b-tq1_0.gguf | 1.03GB | microsoft/bitnet-b1.58-2B-4T converted to ternary TQ1_0 — all transformer matrices are ternary |
mult-f16.gguf | 55MB | 2-digit multiplication expert (LoRA r=32 α=64, FFN modules only) |
roman-f16.gguf | 55MB | Roman numerals expert (same shape) |
fabric-bitnet-fixes.patch | 3KB | required engine fixes (see below) |
moe_driver.py | — | the chat program — routes each question, streams the answer live |
START HERE.bat | — | one-click Windows launcher (starts server + chat) |
results.md | — | full evaluation report |
moe_driver.py and START HERE.bat here are the current, canonical
versions — the same as on GitHub. Download these rather than copying from
older forum posts or snippets.
Results
Temperature 0, exact match on the required The answer is X line:
| task | this stack (ternary) | original bf16 | ternary base, no expert |
|---|---|---|---|
| 2-digit multiplication | 0.80 | 0.94 | 0.70 |
| Roman numerals | 0.30 | 0.24 | 0.05 |
| routing accuracy | 1.00 | — | — |
Use TQ1_0, not TQ2_0. Both formats store ternary weights exactly, but the TQ2_0 runtime kernels neutralized the weaker expert (0.05, below the plain base) while TQ1_0 lifted it above its own bf16 score. If your BitNet + LoRA stack underperforms, try the other ternary format before anything else.
Quickstart
Needs tetherto/qvac-fabric-llm.cpp
built with fabric-bitnet-fixes.patch applied — the stock fork produces
repeating garbage with this model (wrong FFN activation, a double-quantization
bug, and inverted autobitlinear weight_scale semantics).
# serve: one base, both experts loaded but dormant
llama-server -m bitnet-2b-tq1_0.gguf \
--lora mult-f16.gguf --lora roman-f16.gguf \
--lora-init-without-apply -c 4096 --port 8080
# activate exactly one expert (id 0 = mult, id 1 = roman):
curl -X POST http://localhost:8080/lora-adapters \
-d '[{"id":0,"scale":1.0},{"id":1,"scale":0.0}]'
# then use the normal OpenAI-style chat endpoint:
curl -X POST http://localhost:8080/v1/chat/completions -d '{
"messages": [
{"role":"system","content":"You are a careful calculator. Work step by step, then end with exactly '\''The answer is X'\''."},
{"role":"user","content":"What is 34 times 57?"}
], "max_tokens": 384, "temperature": 0}'
For automatic routing (the router reads the question and picks the expert),
use moe_driver.py from the GitHub repo together with
UlukaDev/bitnet-moe-router —
it streams answers live, Ctrl+C cancels an answer without quitting, and
/limit N adjusts the max answer length (default 384).
Note the router expects normalize_embeddings=True when encoding.
⚠️ Never merge these adapters into the base
BitNet re-quantizes its weights on every forward pass — merging a LoRA into the base destroys it (output degenerates to repeating tokens). Always apply the adapters as runtime deltas, as shown above. This is also what lets one loaded base serve any number of experts.
Credits
Base model by Microsoft Research (MIT). Engine: Tether's QVAC Fabric llama.cpp fork (MIT). Router embedding: all-MiniLM-L6-v2. Experts trained with PEFT LoRA on the bf16 base. Code and adapters: MIT.
Sources
1 source
Sources
1 sourceVerified Oct 27
Model artifacts
3 artifacts
Model artifacts
3 artifactsSource excerpts
3 excerpts
Source excerpts
3 excerpts--- license: mit base_model: microsoft/bitnet-b1.58-2B-4T tags: - bitnet - ternary - gguf - lora - mixture-of-experts --- # bitnet-ternary-moe-gguf 🧮 **A 1GB Mixture-of-Experts model designed to run efficiently on an everyday laptop CPU, with no GPU needed.** A single 1.58-bit BitNet base model with two lightweight LoRA expert adapters (multiplication and Roman numerals) that are dynamically activated at runtime by an external router. The base model is loaded once and remains unchanged, reducing memory usage by approximately 5× compared to an equivalent ~5 GB bf16 setup. ➡️ **Code, launcher, eval harness and full setup guide:** [github.com/agentulukaADMIN/ternary-moe](https://github.com/agentulukaADMIN/ternary-moe) ## Files in this repo | file | size | what it is| |---|---|---| | `bitnet-2b-tq1_0.gguf` | 1.03GB | [microsoft/bitnet-b1.58-2B-4T](https://huggingface.co/microsoft/bitnet-b1.58-2B-4T) converted to ternary **TQ1_0** — all transformer matrices are ternary | | `mult-f16.gguf` | 55MB | 2-digit multiplication expert (LoRA r=32 α=64, FFN modules only) | | `roman-f16.gguf` | 55MB | Roman numerals expert (same shape) | | `fabric-bitnet-fixes.patch` | 3KB | **required** engine fixes (see below) | | `moe_driver.py` | — | **the chat program** — routes each question, streams the answer live | | `START HERE.bat` | — | one-click Windows launcher (starts server + chat) | | `results.md` | — | full evaluation report | `moe_driver.py` and `START HERE.bat` here are the **current, canonical versions** — the same as on GitHub. Download these rather than copying from older forum posts or snippets. ## Results Temperature 0, exact match on the required `The answer is X` line: | task | this stack (ternary) | original bf16 | ternary base, no expert | |---|---|---|---| | 2-digit multiplication | **0.80** | 0.94 | 0.70 | | Roman numerals | **0.30** | 0.24 | 0.05 | | routing accuracy | **1.00** | — | — | **Use TQ1_0, not TQ2_0.** Both formats store ternary weights exactly, but the TQ2_0 runtime kernels neutralized the weaker expert (0.05, *below* the plain base) while TQ1_0 lifted it above its own bf16 score. If your BitNet + LoRA stack underperforms, try the other ternary format before anything else. ## Quickstart Needs [tetherto/qvac-fabric-llm.cpp](https://github.com/tetherto/qvac-fabric-llm.cpp) built **with `fabric-bitnet-fixes.patch` applied** — the sto...
--- license: mit base_model: microsoft/bitnet-b1.58-2B-4T tags: - bitnet - ternary - gguf - lora - mixture-of-experts --- # bitnet-ternary-moe-gguf 🧮 **A 1GB Mixture-of-Experts model designed to run efficiently on an everyday laptop CPU, with no GPU needed.** A single 1.58-bit BitNet base model with two lightweight LoRA expert adapters (multiplication and Roman numerals) that are dynamically activated at runtime by an external router. The base model is loaded once and remains unchanged, reducing memory usage by approximately 5× compared to an equivalent ~5 GB bf16 setup. ➡️ **Code, launcher, eval harness and full setup guide:** [github.com/agentulukaADMIN/ternary-moe](https://github.com/agentulukaADMIN/ternary-moe) ## Files in this repo | file | size | what it is ? | |---|---|---| | `bitnet-2b-tq1_0.gguf` | 1.03GB | [microsoft/bitnet-b1.58-2B-4T](https://huggingface.co/microsoft/bitnet-b1.58-2B-4T) converted to ternary **TQ1_0** — all transformer matrices are ternary | | `mult-f16.gguf` | 55MB | 2-digit multiplication expert (LoRA r=32 α=64, FFN modules only) | | `roman-f16.gguf` | 55MB | Roman numerals expert (same shape) | | `fabric-bitnet-fixes.patch` | 3KB | **required** engine fixes (see below) | | `moe_driver.py` | — | **the chat program** — routes each question, streams the answer live | | `START HERE.bat` | — | one-click Windows launcher (starts server + chat) | | `results.md` | — | full evaluation report | `moe_driver.py` and `START HERE.bat` here are the **current, canonical versions** — the same as on GitHub. Download these rather than copying from older forum posts or snippets. ## Results Temperature 0, exact match on the required `The answer is X` line: | task | this stack (ternary) | original bf16 | ternary base, no expert | |---|---|---|---| | 2-digit multiplication | **0.80** | 0.94 | 0.70 | | Roman numerals | **0.30** | 0.24 | 0.05 | | routing accuracy | **1.00** | — | — | **Use TQ1_0, not TQ2_0.** Both formats store ternary weights exactly, but the TQ2_0 runtime kernels neutralized the weaker expert (0.05, *below* the plain base) while TQ1_0 lifted it above its own bf16 score. If your BitNet + LoRA stack underperforms, try the other ternary format before anything else. ## Quickstart Needs [tetherto/qvac-fabric-llm.cpp](https://github.com/tetherto/qvac-fabric-llm.cpp) built **with `fabric-bitnet-fixes.patch` applied** — the...
Source context: 223 downloads · 8 likes · Repo UlukaDev/bitnet-ternary-moe-gguf
