Image Captioning Vit Transformer

--- license: mit language: - en tags: - image-captioning - vision-transformer - computer-vision - deep-learning - pytorch - transformers datasets: - coco pipeline_tag: image-to-text --- # Image Captioning with Vision Transformers This repository contains a transformer-based image captioning model trained on the MS COCO dataset. ## 🚀 Demo Try the live demo here: https://huggingface.co/spaces/mostafahagali/vit-image-captioning ## 🧠 Architecture - Vision Transformer (ViT-B/16) encoder - Transformer encoder - Transformer decoder ## 📊 Training - Dataset: MS COCO 2017 - Loss: Cross-Entropy - Optimizer: AdamW - Framework: PyTorch ## 📦 Files - `best_model.pth`: Trained model weights - `vocab.pkl`: Vocabulary mapping ## 🛠️ Usage ```python import torch import pickle from image_captioning_model import ImageCaptioningModel with open("vocab.pkl", "rb") as f: vocab = pickle.load(f) model = ImageCaptioningModel( vocab_size=len(vocab), pad_id=vocab.word2idx["<pad>"], use_vit=True ) state_dict = torch.load("best_model.pth", map_location="cpu") model.load_state_dict(state_dict) model.eval() --- license: mit --- --- Author: Mostafa Hagali ---

Source context: 0 downloads · 0 likes · Pipeline image-to-text · Library transformers · Repo mostafahagali/image-captioning-vit-transformer