Skip to content
ModelListedExternal
Browse files

PMC Kvasir VQA X1 LoRA 250918 1352

This model combines PMC-CLIP and PMC-LLaMA for medical visual question answering, specifically fine-tuned on the Kvasir-VQA-x1 dataset.

Sourcehuggingface.co/JustATalentedGuy/PMC-Kvasir-VQA-x1-lora_250918-1352/tree/ef82b953dade87d7ed7551e105eab255d93b8496
KindOtherBase modelchaoyi-wu/PMC_LLAMA_7BVersionImage projectorCgrade

Model source

Kind
Other
File size
32.1–153 MB
Base model
chaoyi-wu/PMC_LLAMA_7B
Version
vImage projector
Updated
Oct 23, 2026
Source
Hugging Face

Source description

PMC-VLM: Medical Visual Question Answering

This model combines PMC-CLIP and PMC-LLaMA for medical visual question answering, specifically fine-tuned on the Kvasir-VQA-x1 dataset.

Model Architecture

  • Vision Encoder: PMC-CLIP (frozen)
  • Language Model: PMC-LLaMA-7B with LoRA adapters
  • Image Projector: Linear projection with 4 soft prompt tokens
  • Training: QLoRA (4-bit quantization) fine-tuning

Training Details

  • Dataset: Kvasir-VQA-x1 (Medical VQA)
  • Learning Rate: 0.0002
  • Batch Size: 2 (with 4x accumulation)
  • LoRA Rank: 16
  • LoRA Alpha: 32
  • Epochs: 1

Usage

Load the model and run inference for medical VQA tasks.

Sources

1 source
Hugging FacePrimaryVersion ef82b953dade87d7ed7551e105eab255d93b8496
huggingface.co/JustATalentedGuy/PMC-Kvasir-VQA-x1-lora_250918-1352/tree/ef82b953dade87d7ed7551e105eab255d93b8496

Verified Oct 23

Download files

Model artifacts

2 artifacts

image_projector.pt

pt · 32.1 MB · SHA-256 76bd5622904a…c854 · Hugging Face

Download file

peft_adapters/adapter_model.safetensors

safetensors · 153 MB · SHA-256 6e60e955c659…ea1c · Hugging Face

Download file

Source excerpts

2 excerpts
ReadmeAug 6huggingface.co/JustATalentedGuy/PMC-Kvasir-VQA-x1-lora_250918-1352

--- library_name: transformers base_model: chaoyi-wu/PMC_LLAMA_7B license: apache-2.0 tags: - medical - vqa - multimodal - pmc-clip - pmc-llama - kvasir datasets: - SimulaMet/Kvasir-VQA-x1 metrics: - exact_match - f1_score pipeline_tag: visual-question-answering --- # PMC-VLM: Medical Visual Question Answering This model combines **PMC-CLIP** and **PMC-LLaMA** for medical visual question answering, specifically fine-tuned on the Kvasir-VQA-x1 dataset. ## Model Architecture - **Vision Encoder**: PMC-CLIP (frozen) - **Language Model**: PMC-LLaMA-7B with LoRA adapters - **Image Projector**: Linear projection with 4 soft prompt tokens - **Training**: QLoRA (4-bit quantization) fine-tuning ## Training Details - **Dataset**: Kvasir-VQA-x1 (Medical VQA) - **Learning Rate**: 0.0002 - **Batch Size**: 2 (with 4x accumulation) - **LoRA Rank**: 16 - **LoRA Alpha**: 32 - **Epochs**: 1 ## Usage Load the model and run inference for medical VQA tasks.

NoteAug 6huggingface.co/JustATalentedGuy/PMC-Kvasir-VQA-x1-lora_250918-1352

Source context: 0 downloads · 0 likes · Pipeline visual-question-answering · Library transformers · Repo JustATalentedGuy/PMC-Kvasir-VQA-x1-lora_250918-1352