JiRack 10b

Lllama3 JiRack 10B FP32, INT8, INT4

A fast and efficient coding assistant with a clean, modern built-in web UI.

Powered by Meta Llama 3.1 8B Instruct weights and a fully refactored architecture optimized for a 10B-scale model. The model was specifically designed for high-performance tuning with advanced quantization options.

Llama3 JiRack Ternary 10B https://huggingface.co/CMSManhattan/Llama3Ternary_10b

Setup flag ternary true allow to do IT to convert model to ternary format then use llama.cpp to run GGUF conversation with llama.cpp Ternary support kernel.

• JiRack is a cloud-ready model that helps save money on cloud usage. It can be used as an expert model in RAG deployments on cloud, with the ONNX JiRack Java server as an alternative.
• Subscription: $1 per month per user (updated license for non-company use).
• Corp Subscription: $3 per month per user (updated license for company use).
• It works without subscription but send message about subscription.

Ternary version with weights

• https://huggingface.co/CMSManhattan/Llama3Ternary_2b
• https://huggingface.co/CMSManhattan/Llama3Ternary_10b

JiRack sevice options ONNX or bitnet.cpp engine or llama.cpp

• Current quantizations were done from the FP16 model, but the model allows for more compression thanks to its ternary architecture.
• If you need to do ternary compression, please write to me and I'll perform QAT from your dataset, tailored specifically to your task.
• Adapt train process to avoid catastrophic forgetting with NDA
• Adapt train process to avoid fast plato in training with NDA
• Convert model to bitnet.cpp kernel I2S_0 with support AVX2 and AVX-512 CPU instructions for high performance on CPU

JiRack Android Client DEMO:
https://www.youtube.com/watch?v=SaO6Jfb8R68

CMS Manhattan RAG & Email Reply + Document and Email Analytics:
https://www.youtube.com/watch?v=KRu2nLEh_6g&t=78s

"So I don’t read my emails — I just ask JiRack to tell me the news!"

Welcome to the CMS Manhattan AI Front Office Solution.

Training the Model

It is easy to train on a Blackwell GPU with 96 GB VRAM.
You do not need a data center for fine-tuning or QLoRA — it works well on consumer-grade GPU cards.

Let me know if you need the training code!

Quick Start

Watch JiRack 10B in action and run it with Docker.

Run with Docker

Default CPU INT8

docker run -d \
  --name jirack_10b \
  -p 7869:7869 \
  --restart unless-stopped \
  cmsmanhattan/jirack_10b_int8:latest

Default CPU INT4

docker run -d \
  --name jirack_10b \
  -p 7869:7869 \
  --restart unless-stopped \
  cmsmanhattan/jirack_10b_int4:latest

Multi CPU

docker run -d \
  --name jirack_10b \
  -p 7869:7869 \
  --restart unless-stopped \
  --memory=20g \
  --cpus=12 \
  cmsmanhattan/jirack_10b_int8:latest

GPU

docker run -d \
  --name jirack_10b \
  -p 7869:7869 \
  --gpus all \
  --restart unless-stopped \
  cmsmanhattan/jirack_10b_gpu_int8:latest

Docker Compose Example

services:
  jirack:
    image: cmsmanhattan/jirack_10b_int8:1.0.2
    container_name: jirack_onnx_service
    ports:
      - "7869:7869"
    volumes:
      - .:/app
      - ./web:/app/web
    environment:
      - MAX_TOKENS=1024
      - TEMPERATURE=0.7
      - TOP_P=0.9
      - DEFAULT_STREAM=False
      - INTRA_THREADS=4
      - USE_ENV_ALLOCATOR=1
    deploy:
      resources:
        limits:
          memory: 16g

Access the UI

Once the container is running, open your browser and go to:

http://localhost:7869

This opens the JiRack UI — a clean, modern web interface for chat.

Changing the Port

The listening port can be easily changed from the Settings panel inside the JiRack Chat UI.

Licensing

• The JiRack 10B model is provided under a commercial enterprise license.
• All JiRack UI clients are provided under a commercial license.
• UI clients can be used for free when running with the official JiRack Docker containers, as long as they are not redistributed separately.

Subscription Plans

• JiRack Enterprise: $36 per user per year
• JiRack Private: $12 per user per year

For commercial licensing, cluster deployment, performance tuning, or enterprise use, please contact us.

• JiRack Android Chat Client (voice + Ollama API):
  https://huggingface.co/kgrabko/JiRackTernary_1b/resolve/main/app-release.apk
  or Google Play

• JiRack Windows 11 Desktop Client (Ollama API):
  https://huggingface.co/kgrabko/JiRackTernary_1b/resolve/main/jirack-chat.zip

• Live email chat: support@cmsmanhattan.com

Hardware Recommendations for AMD Systems

Recommended Hardware for JiRack 10B INT8 (single Docker container)

Use Case	CPU	GPU (ROCm)	VRAM / RAM	Expected Speed	Recommendation
Recommended	Ryzen 7 7700 / 9700X	RX 7900 XTX / 7900 XT	24GB VRAM	50-75 tokens/s	Best choice
High Performance	Ryzen 9 7950X / 9950X	RX 7900 XTX	24GB+ VRAM	65-90 tokens/s	Excellent
Enterprise	EPYC 7003/9004 series	MI300X or 2x RX 7900 XTX	48GB+ VRAM	90-140 tokens/s	For 32B model
Budget Option	Ryzen 5 7600 / 9600X	RX 7800 XT (16GB)	16GB VRAM	35-50 tokens/s	Acceptable

Important Memory Notes

Even though the 10B INT8 model itself takes approximately 8–9 GB, we recommend at least 24GB VRAM for:

• KV-cache consumption during generation, especially with long context
• ONNX Runtime overhead and temporary buffers
• System stability and avoiding out-of-memory errors
• Support for larger context windows

Minimum recommended: 24GB VRAM (RX 7900 series)
Ideal: 24–32GB VRAM

For pure CPU inference, we recommend at least 64GB system RAM.

I added the default model in full FP32 precision (~62 GB). This serves as the base for quantization to find the best balance between size and performance.

---

JiRack Codding Agent IDE

• It is Agent Coding IDE for JiRack Models to run via Ollama on home PC
• It good choose for Agent Coding IDE such as Cursor , Windsurf IDE or Devin IDE etc but more safe that ask you to apply changes and review.
• Web site https://www.jirack.com
• Eclise Plugin site https://marketplace.eclipse.org/content/jirack-coding-agent
• Jetbrain Plugin site https://plugins.jetbrains.com/plugin/35073-jirack-agent
• VS Code Plugin site https://marketplace.visualstudio.com/items?itemName=JiRackautomationCodingAgentIDE.jirack-agent

JiRack Codding Agent IDE files

• Eclise Final release version https://huggingface.co/CMSManhattan/JiRackDeltaNet_27b/resolve/main/jirack_ide_final.zip
• Microsoft VS Code plugin https://huggingface.co/spaces/CMSManhattan/README/blob/main/jirack-agent-0.9.0.vsix
• Jetbrain plugin https://huggingface.co/spaces/CMSManhattan/README/resolve/main/jirack-jetbrains-0.11.0.zip
• JiRack DeskTop UI https://huggingface.co/spaces/CMSManhattan/README/resolve/main/windows_v11.zip

---

📧 Contact & Licensing

For joint ventures, hardware integration, or licensing inquiries:

• Email: grabko@cmsmanhattan.com
• Phone: +1 (516) 777-0945
• Location: New York, USA

NVIDIA Inception Program Member