YORENG100 SSAST Emotion Recognition

YORENG100-SSAST-Emotion-Recognition

This model is a specialized Speech Emotion Recognition (SER) system fine-tuned from the SSAST (Self-Supervised Audio Spectrogram Transformer). It is specifically designed to handle the linguistic complexities of Yoruba-English code-switching, common in Nigerian urban centers.

🌟 Model Highlights

• Architecture: SSAST-Tiny (16x16 Patches)
• Dataset: YORENG100 (100 hours of curated Yoruba-English bilingual speech)
• Top Accuracy: 96.45% on validation set.
• Innovation: Captures emotional prosody across language boundaries where traditional monolingual models often fail.

🚀 How to Use (Manual Inference)

Since SSAST requires specific tensor dimension handling for its patch-based architecture, we recommend the following manual inference style over the standard pipeline.

import torch
import librosa
from transformers import AutoFeatureExtractor, ASTForAudioClassification

# 1. Load Model & Processor
model_id = "Professor/YORENG100-SSAST-Emotion-Recognition"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

feature_extractor = AutoFeatureExtractor.from_pretrained(model_id)
model = ASTForAudioClassification.from_pretrained(model_id).to(device)
model.eval()

# 2. Prepare Audio (Resample to 16kHz)
audio_path = "path_to_your_audio.wav"
speech, _ = librosa.load(audio_path, sr=16000)

# 3. Preprocess & Predict
inputs = feature_extractor(speech, sampling_rate=16000, return_tensors="pt")
# SSAST Fix: Ensure [Batch, Time, Freq] shape
input_values = inputs.input_values.squeeze().unsqueeze(0).to(device)

with torch.no_grad():
    logits = model(input_values).logits
    prediction = torch.argmax(logits, dim=-1).item()

print(f"Predicted Emotion: {model.config.id2label[prediction]}")

📊 Training Results

The model was trained for 3 epochs on a TPU-v3-8. Note the high stability in accuracy even as loss continued to decrease.

Epoch	Step	Training Loss	Validation Loss	Accuracy
1.0	2501	0.2156	0.2080	0.9645
2.0	5002	0.1943	0.2044	0.9645
3.0	7503	0.1842	0.2023	0.9645

🌍 Background: Yoruba-English Emotion Recognition

Emotion recognition in tonal languages like Yoruba is uniquely challenging because pitch (F0) is used to distinguish word meanings (lexical tone) as well as emotional state. When speakers code-switch into English, the "emotional acoustic profile" shifts mid-sentence.

This model uses the patch-based attention mechanism of SSAST to focus on local time-frequency features that are invariant to these language shifts.

🛠 Limitations

• Background Noise: Not yet robust to high-noise environments (e.g., market or traffic noise).
• Sampling Rate: Only supports 16kHz audio input.
• Labels: Performance is optimized for [Angry, Happy, Sad, Neutral, etc.].

📜 Citation

If you use this model or the YORENG100 dataset, please cite us