Wav2vec2 Emotion Classifier V2

Wav2Vec2 Multi-Label Emotion Classifier

This model is a fine-tuned version of facebook/wav2vec2-base for multi-label emotion classification from audio.

Model Description

The model uses a Hydra architecture with multiple binary classification heads, one for each emotion. It supports ignore labels (-100) for handling uncertain or unlabeled data points.

Supported Emotions

• angry, disgust, fear, happy, neutral, depress, surprise, frustrated, excited

Training Results

Overall Performance

• Best Validation F1 Score: 0.0000
• Best Validation Accuracy: 0.0000
• Best Validation Loss: inf
• Best Epoch: 0

Per-Emotion Performance (at best epoch)

Emotion	Accuracy	F1 Score	Valid Samples	Ignored Samples
angry	0.8762	0.0000	4749	0
disgust	0.9872	0.0000	4749	0
fear	0.9821	0.0000	4749	0
happy	0.8793	0.0000	4749	0
neutral	0.6477	0.0000	4749	0
depress	0.9111	0.0000	4749	0
surprise	0.9299	0.0000	4749	0
frustrated	0.6949	0.0000	1996	2753
excited	0.7996	0.0000	1996	2753

Sample Statistics

• Total Training Samples: 18994
• Total Validation Samples: 4749
• Total Valid Labels: 37235
• Total Ignored Labels: 5506
• Ignore Rate: 12.9%

Training Configuration

{
  "model_id": "facebook/wav2vec2-base",
  "emotion_labels": [
    "angry",
    "disgust",
    "fear",
    "happy",
    "neutral",
    "depress",
    "surprise",
    "frustrated",
    "excited"
  ],
  "num_classes": 9,
  "ignore_label": -100,
  "batch_size": 4,
  "num_epochs": 6,
  "learning_rate": 0.0001,
  "warmup_steps": 100,
  "gradient_accumulation_steps": 4,
  "classification_threshold": 0.5,
  "weight_decay": 0.01,
  "max_grad_norm": 1.0,
  "seed": 42
}

Usage

Handling Ignore Labels

This model was trained with support for ignore labels (-100). During inference:

• The model outputs raw logits for each emotion
• Apply sigmoid to get probabilities
• Use a threshold (default: 0.5) for binary classification
• The model can handle inputs where some emotions were marked as "ignore" during training

Example Code

import torch
from transformers import Wav2Vec2Processor

# Load model and processor
model = # Load your model here
processor = Wav2Vec2Processor.from_pretrained('mangoesai/wav2vec2-emotion-classifier-V2')

# Process audio
inputs = processor(audio_array, sampling_rate=16000, return_tensors="pt", padding=True)

# Get predictions
with torch.no_grad():
    outputs = model(inputs.input_values, inputs.attention_mask)
    probabilities = torch.sigmoid(outputs)
    predictions = (probabilities > 0.5).int()

Training Infrastructure

• Hardware: NVIDIA A100-SXM4-40GB
• Training Duration: 1.0 hours
• PyTorch Version: 2.6.0+cu124
• Transformers Version: 4.36.0

Limitations and Biases

• The model's performance may vary for emotions with limited training data
• Emotions with high ignore rates during training may have reduced accuracy
• The model is trained on 16kHz audio and may need resampling for other sample rates

Citation

If you use this model, please cite:

@misc{wav2vec2-emotion-classifier,
  author = {Your Name},
  title = {Wav2Vec2 Multi-Label Emotion Classifier},
  year = {2024},
  publisher = {HuggingFace},
  url = {https://huggingface.co/mangoesai/wav2vec2-emotion-classifier-V2}
}