Qwen3.6 35B A3B NVFP4 Fast GGUF

--- license: apache-2.0 base_model: - unsloth/Qwen3.6-35B-A3B-NVFP4-Fast pipeline_tag: image-text-to-text --- # Qwen 3.6 35B A3B NVFP4 Fast GGUF ## About This model is an unmodified GGUF quantization of [unsloth/Qwen3.6-35B-A3B-NVFP4-Fast](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-NVFP4-Fast), made with the llama.cpp conversion tool. Please refer to unsloth repository for quality metrics. Your feedback, contributions, optimization proposals and questions are highly appreciated. This GGUF is primarily intended to be run fully on the GPU without offloading to the CPU. However, you can absolutely run it with offloading. The relevant guides are provided in this repo. ## Speed metrics I've tested this model on my setup (5070 Ti 16GB, 5060 Ti 16GB, 32GB DDR5). Token generation and prefill speeds will vary depending on your hardware. Please use the metrics below as a rough estimate of your potential performance. | Setup | Token generation, t/s | Prompt prefill, t/s | |:---|:---|:---| | 5070ti + 5060ti | ~134.4 | ~3462 | | 5070ti + CPU | ~68.7 | ~324 | | 5060ti + CPU | ~51.0 | ~161 | ## Quick start This GGUF requires a recent build of llama.cpp. If the commands below raise a compatibility error, please update your llama.cpp version. All commands have been tested on llama.cpp version `b9957` - ensure your version is `>=b9957`. ### Agentic tasks ```bash docker run --rm \ --runtime nvidia \ --gpus 2 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -e HF_HUB_CACHE=/root/.cache/huggingface/hub \ -p 8080:8080 \ ghcr.io/ggml-org/llama.cpp:server-cuda13 \ -hf catlilface/Qwen3.6-35B-A3B-NVFP4-Fast-GGUF \ -np 1 \ -ngl -1 \ -c 256000 \ --no-op-offload \ --spec-type draft-mtp \ --spec-draft-n-max 3 ``` ### Speed-efficient The token generation speeds listed below were achieved using varying prompt lengths, KV cache quantization, and the more powerful GPU 0. These optimizations simply let more computations to be done on 5070ti. ```bash docker run --rm \ --runtime nvidia \ --gpus 2 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -e HF_HUB_CACHE=/root/.cache/huggingface/hub \ -p 8080:8080 \ ghcr.io/ggml-org/llama.cpp:server-cuda13 \ -hf catlilface/Qwen3.6-35B-A3B-NVFP4-Fast-GGUF \ -np 1 \ -ngl -1 \ -c 16000 \ -ctk q8_0 \ -ctv q8_0 \ --tensor-split 2,1 \ # only for unequal gpus --no-op-offload \ --spec-type draft-mtp \ --spe...

Source context: 383 downloads · 6 likes · Pipeline image-text-to-text · Repo catlilface/Qwen3.6-35B-A3B-NVFP4-Fast-GGUF