Whisper Medium Hindi Hinglish Asr Fine Tuned

Whisper Medium Hindi-Hinglish ASR (Fine-Tuned)

A fine-tuned version of OpenAI Whisper Medium for Hindi and Hinglish Automatic Speech Recognition (ASR).

This model was developed to improve Whisper's performance on conversational Hindi and code-mixed Hindi-English speech using a carefully curated synthetic speech corpus.

---

Model Details

Property	Value
Base Model	openai/whisper-medium
Model Type	Whisper Encoder-Decoder Transformer
Task	Automatic Speech Recognition
Languages	Hindi, Hinglish
Framework	Hugging Face Transformers

---

Motivation

OpenAI Whisper provides excellent multilingual speech recognition, but conversational Hindi and Hinglish often remain challenging due to code-mixing, pronunciation variations, and domain-specific vocabulary.

This project fine-tunes Whisper Medium using a curated synthetic dataset to significantly improve recognition quality while preserving Whisper's multilingual capabilities.

---

Dataset

The training dataset was generated using a multi-stage synthetic data generation pipeline consisting of:

• LLM-generated conversational Hindi/Hinglish text
• High-quality speech synthesis
• Transcript refinement and normalization
• Quality filtering
• Curated speech-text pairs for Whisper fine-tuning

The focus was on maximizing transcript quality while maintaining linguistic diversity.

---

Evaluation

Evaluation was performed on an unseen held-out test set.

Overall Performance

Metric	Score
Test Samples	373
Word Error Rate (WER)	0.0456
Character Error Rate (CER)	0.0262
Relative WER Improvement	86.99%

---

Performance by Difficulty

Difficulty	Samples	WER
Easy	317	0.0303
Medium	49	0.1301
Hard	7	0.2182

---

Performance by Language

Language	Samples	WER
Hindi	280	0.0514
Hinglish	93	0.0300

---

Performance by Error Category

Category	Samples	WER
Perfect	288	0.0384
Minor Phonetic	50	0.0628
Digit Format	29	0.0758
Script Switch	3	0.0682
Genuine Error	2	0.0588
Loanword Mishear	1	0.0000

---

Usage

Load the model using the Hugging Face Transformers library.

import torch
from transformers import WhisperProcessor, WhisperForConditionalGeneration

processor = WhisperProcessor.from_pretrained(
    "shujaAK/whisper-medium-hindi-hinglish-asr-fine-tuned"
)

model = WhisperForConditionalGeneration.from_pretrained(
    "shujaAK/whisper-medium-hindi-hinglish-asr-fine-tuned"
)

device = "cuda" if torch.cuda.is_available() else "cpu"

model = model.to(device)
model.eval()

---

Intended Uses

This model is intended for:

• Hindi Automatic Speech Recognition
• Hinglish Automatic Speech Recognition
• Academic Research
• Speech Recognition Experiments
• Fine-tuning Research

---

Limitations

• The model is primarily trained on synthetic speech.
• Performance may decrease on noisy recordings.
• Performance may vary across unseen accents and speaking styles.
• Not evaluated for streaming ASR.

---

Training Framework

• Hugging Face Transformers
• PyTorch

---

Results Summary

Metric	Value
WER	0.0456
CER	0.0262
Test Samples	373
WER Improvement	86.99%

---

Acknowledgements

This project builds upon:

• OpenAI Whisper
• Hugging Face Transformers

---

Citation

If you use this model in your research or applications, please cite this Hugging Face repository.

@misc{whisper_medium_hindi_hinglish_asr,
  title={Whisper Medium Hindi-Hinglish ASR (Fine-Tuned)},
  author={Suja Akhter},
  year={2026},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/shujaAK/whisper-medium-hindi-hinglish-asr-fine-tuned}}
}