Siglip2 Mbart Envi Vlmt

--- language: - en - vi pipeline_tag: image-to-text tags: - multimodal-machine-translation - vision-language - siglip2 - mbart - lora - english-vietnamese license: mit --- # SigLIP2-mBART English-Vietnamese Vision-Language Translation This model translates English visual phrases into Vietnamese using both the source phrase and image context. ## Model Overview The model is a lightweight Vision-Language Machine Translation system built with: - **Vision encoder:** `google/siglip2-base-patch16-256` - **Text generation model:** `facebook/mbart-large-50-many-to-many-mmt` - **Fusion module:** pooled visual token with a trainable visual projector - **Adaptation method:** LoRA on mBART - **Source language:** English - **Target language:** Vietnamese ## Architecture ```text Image -> SigLIP2 vision encoder -> pooled visual feature -> visual projector -> visual token English phrase -> mBART encoder [visual token + mBART encoder hidden states] -> mBART decoder with LoRA adapters -> Vietnamese translation ``` ## Intended Use This model is intended for English-to-Vietnamese multimodal phrase translation where image context can help disambiguate the source phrase. Example input: ```text Image: an image containing a bicycle English phrase: "a man riding a bicycle" Output: Vietnamese translation ``` ## Repository Files ```text model_state.pt model_config.json inference.py tokenizer/ image_processor/ requirements.txt README.md ``` ## Usage Install dependencies: ```bash pip install -r requirements.txt ``` Load and run the model: ```python from inference import load_model, translate_image_phrase repo_id = "AnhDau/siglip2-mbart-envi-vlmt" model, tokenizer, image_processor, config, device = load_model(repo_id) prediction = translate_image_phrase( model=model, tokenizer=tokenizer, image_processor=image_processor, image_path="example.jpg", english_phrase="a man riding a bicycle", config=config, device=device, ) print(prediction) ``` ## Notes This repository contains a custom Vision-Language model architecture. It should be loaded with `inference.py` rather than directly with `AutoModel.from_pretrained`.

Source context: 0 downloads · 0 likes · Pipeline image-to-text · Repo AnhDau/siglip2-mbart-envi-vlmt