Skip to content
ModelListedExternal
Download

Qwen3 8B Q5 K M

Qwen3-8B in Q5KM GGUF format for CPU/GPU inference with llama.cpp or llama-cpp-python.

Sourcehuggingface.co/Mukilan06/Qwen3-8B-Q5_K_M/tree/ea0345f6053c552b9e22d262423d24856e7df0f1
KindOtherBase modelQwen/Qwen3-8BVersionQwen3 8 B Q5 K MBgrade

Model source

Kind
Other
File size
5.45 GB
Base model
Qwen/Qwen3-8B
Version
vQwen3 8 B Q5 K M
Updated
Oct 18, 2026
Source
Hugging Face

Source description

Qwen3-8B-Q5_K_M

Qwen3-8B in Q5_K_M GGUF format for CPU/GPU inference with llama.cpp or llama-cpp-python.

Usage

py -3.11 -m pip install llama-cpp-python transformers

from llama_cpp import Llama
from transformers import AutoTokenizer

llm = Llama(model_path="Qwen3-8B-Q5_K_M.gguf", n_ctx=8192)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")

messages = [{"role": "user", "content": "Hello!"}]
prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
)
out = llm(prompt, max_tokens=256, stop=[tokenizer.eos_token])
print(out["choices"][0]["text"])

Sources

1 source
Hugging FacePrimaryVersion ea0345f6053c552b9e22d262423d24856e7df0f1
huggingface.co/Mukilan06/Qwen3-8B-Q5_K_M/tree/ea0345f6053c552b9e22d262423d24856e7df0f1

Verified Oct 18

Model artifacts

1 artifact

Qwen3-8B-Q5_K_M.gguf

gguf · 5.45 GB · SHA-256 068bae163faa…2738 · Hugging Face

Download file

Source excerpts

3 excerpts
ReadmeAug 8huggingface.co/Mukilan06/Qwen3-8B-Q5_K_M

--- license: apache-2.0 base_model: Qwen/Qwen3-8B tags: - gguf - qwen3 - q5_k_m - text-generation library_name: llama-cpp-python --- # Qwen3-8B-Q5_K_M Qwen3-8B in **Q5_K_M** GGUF format for CPU/GPU inference with llama.cpp or llama-cpp-python. - Source base model: [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) - Original GGUF source: [Qwen/Qwen3-8B-GGUF](https://huggingface.co/Qwen/Qwen3-8B-GGUF) - Quantization: Q5_K_M (~5.85 GB) - License: Apache 2.0 ## Usage ```powershell py -3.11 -m pip install llama-cpp-python transformers ``` ```python from llama_cpp import Llama from transformers import AutoTokenizer llm = Llama(model_path="Qwen3-8B-Q5_K_M.gguf", n_ctx=8192) tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B") messages = [{"role": "user", "content": "Hello!"}] prompt = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, enable_thinking=False ) out = llm(prompt, max_tokens=256, stop=[tokenizer.eos_token]) print(out["choices"][0]["text"]) ```

NoteOct 18huggingface.co/Mukilan06/Qwen3-8B-Q5_K_M

Source context: 60 downloads · 0 likes · Pipeline text-generation · Library llama-cpp-python · Repo Mukilan06/Qwen3-8B-Q5_K_M

NoteAug 8huggingface.co/Mukilan06/Qwen3-8B-Q5_K_M

Source context: 20 downloads · 0 likes · Pipeline text-generation · Library llama-cpp-python · Repo Mukilan06/Qwen3-8B-Q5_K_M