Dinov2 SigLIP Phi3 LoRA
Vision Encoder - DinoV2 + SigLIP @384px resolution. Why 2 vision encoders? Connector - MLP (Dino and SigLIP features are concatenated and then projected to Phi3 representation space) Language Model - Phi3 + LoRA...
Model source
- Kind
- Other
- File size
- 210 MB–15.2 GB
- Version
- vRevision 69e46f5
- Updated
- Sep 25, 2026
- Downloads
- 0
- Source
- Hugging Face
Source description
DinoV2-SigLIP-Phi3(LoRA) VLM
- Vision Encoder - DinoV2 + SigLIP @384px resolution. Why 2 vision encoders?
- Connector - MLP (Dino and SigLIP features are concatenated and then projected to Phi3 representation space)
- Language Model - Phi3 + LoRA
- Pre-train (Align) Dataset - LLaVA-CC3M-Pretrain-595K
- Fine-tune (Instruction) Dataset - LLAVA-v1.5-Instruct + LRV-Instruct
Scripts to build and train the models are available at NMS05/DinoV2-SigLIP-Phi3-LoRA-VLM.
Sources
1 source
Sources
1 sourceVerified Aug 28
Model artifacts
2 artifacts
Model artifacts
2 artifactsSource excerpts
2 excerpts
Source excerpts
2 excerpts--- datasets: - liuhaotian/LLaVA-Instruct-150K - liuhaotian/LLaVA-CC3M-Pretrain-595K language: - en metrics: - accuracy pipeline_tag: visual-question-answering --- # DinoV2-SigLIP-Phi3(LoRA) VLM * **Vision Encoder** - DinoV2 + SigLIP @384px resolution. [Why 2 vision encoders?](https://arxiv.org/abs/2401.06209) * **Connector** - MLP (Dino and SigLIP features are concatenated and then projected to Phi3 representation space) * **Language Model** - Phi3 + LoRA * **Pre-train (Align) Dataset** - LLaVA-CC3M-Pretrain-595K * **Fine-tune (Instruction) Dataset** - LLAVA-v1.5-Instruct + LRV-Instruct Scripts to build and train the models are available at [NMS05/DinoV2-SigLIP-Phi3-LoRA-VLM](https://github.com/NMS05/DinoV2-SigLIP-Phi3-LoRA-VLM).
Source context: 0 downloads · 0 likes · Pipeline visual-question-answering · Repo nms05/Dinov2-SigLIP-Phi3-LoRA
