Beforeat Food Nutrition Vision LoRA

beforeat-food-nutrition-vision-lora

beforeat-food-nutrition-vision-lora is a LoRA adapter for food image understanding. This checkpoint is an R&D prototype toward broader food and nutrition recognition. The current version is trained and evaluated for Food-101 dish-name recognition, not full nutrition estimation.

Model Details

• Adapter name: beforeat-food-nutrition-vision-lora
• Base model: Qwen/Qwen3-VL-4B-Instruct
• Adapter type: PEFT LoRA
• Task: food image to dish-name text response
• Current training focus: Food-101 dish classification via instruction-style visual question answering
• Developed for: Beforeat food recognition research and prototyping

Intended Use

This adapter is intended for research and prototype food recognition workflows where an image of a prepared dish is provided and the model returns a concise dish-name prediction.

Example prompt:

Identify the dish in this image. Reply with only the exact Food-101 class name.

For a production app, the recommended v1 deployment path is backend inference: the mobile app uploads a food image to a server, the server runs the base Qwen3-VL model with this LoRA adapter, and the app receives the predicted dish name.

Out-of-Scope Use

This checkpoint should not be treated as:

• A complete nutrition estimator
• A medical, dietary, or allergy safety tool
• A reliable portion-size estimator
• A production-grade model for all cuisines, restaurant conditions, or user-generated food photos

The adapter currently recognizes dish names better than it estimates ingredients, calories, macros, or micronutrients.

Training Data

This adapter was trained on Food-101 examples formatted as image-and-text instruction data. Food-101 is useful for dish-name research, but this checkpoint should be treated as an R&D prototype and downstream users should review the Food-101 dataset terms before any commercial or public deployment.

No Food-101 images are included in this adapter repository.

Training Procedure

The adapter was trained locally with PyTorch and PEFT LoRA on Apple Silicon using MPS acceleration.

LoRA configuration:

• Rank: 8
• Alpha: 16
• Dropout: 0.05
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Trainable parameters: about 16.5M

Training progression:

• 1000-step Food-101 adapter
• 1500-step broad Food-101 continuation
• 2000-step broad Food-101 continuation
• 2500-step mixed continuation with broad Food-101 coverage plus extra hard-class examples

Evaluation

Evaluation was run on a fixed 500-image Food-101 sample.

Adapter	Strict Accuracy	Alias/Near Accuracy
1000-step Food-101 LoRA	64.60%	78.40%
1500-step broad continuation	79.00%	84.80%
2000-step broad continuation	85.20%	87.80%
2500-step hard-only continuation	81.80%	82.80%
2500-step mixed continuation, this adapter	86.80%	87.60%

The 2500-step mixed adapter is the current best strict-label checkpoint. The 2000-step broad adapter remains a close fallback because it has slightly higher alias/near scoring.

Common remaining errors include visually similar food categories such as:

• tuna tartare vs beef tartare
• chocolate cake vs chocolate mousse
• donuts vs beignets
• steak, filet mignon, and prime rib
• bread pudding, panna cotta, and dessert-adjacent classes

Loading Example

import torch
from peft import PeftModel
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

base_model_id = "Qwen/Qwen3-VL-4B-Instruct"
adapter_id = "YOUR_USERNAME/beforeat-food-nutrition-vision-lora"

processor = AutoProcessor.from_pretrained(
    base_model_id,
    trust_remote_code=True,
)

model = Qwen3VLForConditionalGeneration.from_pretrained(
    base_model_id,
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True,
)

model = PeftModel.from_pretrained(model, adapter_id)
model.eval()

llama.cpp GGUF Adapter

This repository also includes a standalone llama.cpp-compatible GGUF LoRA adapter for offline runtime loading:

• GGUF adapter filename: beforeat-food-nutrition-vision-lora-f16.gguf
• File size: 33065184 bytes
• SHA-256: 6fa0950b1bc4df4c1018b60ff4e0d824789bbc52a3764d1d38078861a0c2d862
• Format: F16 GGUF LoRA adapter
• Recommended adapter scale: 1.0
• llama.cpp commit used: 00fa7cb284cbf133fc426733bd64238a3588a33e

Required runtime files:

• Base GGUF: Qwen3-VL-4B-Instruct-UD-Q4_K_XL.gguf
• Vision projector: mmproj-F16.gguf
• Base GGUF source: unsloth/Qwen3-VL-4B-Instruct-GGUF

Conversion command:

PYTHONPATH=tools/llama.cpp \
python tools/llama.cpp/convert_lora_to_gguf.py \
  adapters/qwen3vl4b_food101_peft_2500_mixed_hard20 \
  --base models/Qwen3-VL-4B-Instruct-bf16-remapped \
  --outfile adapters/qwen3vl4b_food101_peft_2500_mixed_hard20/beforeat-food-nutrition-vision-lora-f16.gguf \
  --outtype f16 \
  --verbose

Validation/runtime command:

tools/llama.cpp/build/bin/llama-mtmd-cli \
  -m gguf_base/unsloth-Qwen3-VL-4B-Instruct-GGUF/Qwen3-VL-4B-Instruct-UD-Q4_K_XL.gguf \
  --mmproj gguf_base/unsloth-Qwen3-VL-4B-Instruct-GGUF/mmproj-F16.gguf \
  --image food-101/images/prime_rib/2421701.jpg \
  --lora-scaled adapters/qwen3vl4b_food101_peft_2500_mixed_hard20/beforeat-food-nutrition-vision-lora-f16.gguf:1.0 \
  -p "Identify the dish in this image. Reply with only the exact Food-101 class name." \
  --jinja \
  --temp 0 \
  -n 16 \
  -c 4096 \
  --image-min-tokens 1024 \
  --no-warmup \
  --no-perf \
  --log-verbosity 1

Runtime validation confirmed that llama.cpp accepts the adapter with the Unsloth UD-Q4_K_XL base and mmproj-F16.gguf, loading all 504 LoRA tensors with no missing, incompatible, or unexpected tensor errors.

Small hard-class validation slice:

Expected	PyTorch PEFT	Base GGUF	GGUF + LoRA
bread pudding	panna cotta	dessert	bread pudding
filet mignon	beef fillet	Duck Confit	beef tenderloin
prime rib	prime rib	roast beef	prime rib
samosa	pork and cabbage roll	pie	samosa
fried calamari	fried calamari	fried_octopus	fried calamari
huevos rancheros	huevos rancheros	Egg	huevos rancheros
escargots	escargots	Fish	escargots
foie gras	beef chop	duck	beef chop
croque madame	croque madame	sandwich	croque madame
lobster bisque	lobster bisque	soup	cheese soup

On this intentionally hard 10-image slice:

• PyTorch PEFT: 6/10 strict, 6/10 alias/near
• Base GGUF without LoRA: 0/10 strict, 0/10 alias/near
• Base GGUF with GGUF LoRA: 7/10 strict, 8/10 alias/near

Limitations

• The model was evaluated on Food-101-style images, not yet on a broad real-world Beforeat app photo set.
• It may produce close but non-canonical dish names.
• It may confuse visually similar dishes.
• It should not be used as the sole source for dietary, medical, or nutrition decisions.
• Broader ingredient and nutrition recognition requires additional datasets and evaluation.
• The GGUF LoRA adapter was validated with Qwen3-VL-4B-Instruct-UD-Q4_K_XL.gguf and mmproj-F16.gguf; other base quantizations should be tested before app release.

Recommended Next Steps

• Test this adapter against real Beforeat-style food photos.
• Compare against the 2000-step broad adapter on real app images.
• Add ingredient and nutrition-specific training/evaluation data before claiming nutrition estimation capability.
• Consider a smaller distilled/mobile model for eventual on-device inference.

Framework Versions

• PEFT 0.19.1
• Transformers 4.57.0
• PyTorch nightly used locally for Apple MPS compatibility