Deepfake Detector Dfd Sota

SPDX-License-Identifier: Apache-2.0

""" SOTA Deepfake Detector - DFD Model card + inference utilities for Hugging Face repository: Arko007/deepfake-detector-dfd-sota

This file contains:

• A model card in Hugging Face model README format (YAML frontmatter + sections).
• Inference helper functions to:
  • load frames or an MP4 video,
  • extract 12 evenly-spaced frames,
  • preprocess using the CLIP image processor,
  • run a forward pass through the provided checkpoint and return logits,
  • return softmax probabilities and predicted class.

Notes:

• Architecture described: CLIP-ViT-Large (frozen backbone) + Spatiotemporal Adapters
• Input: 12 frames, 3x384x384, dtype=torch.float32 / BF16 where supported
• Labels: 0 = real, 1 = fake
• License: Apache-2.0
• This file is intended as a convenience reference for model consumers and deployers. """

SOTA Deepfake Detector - DFD

Model: SOTA Deepfake Detector - DFD
Repository: https://huggingface.co/Arko007/deepfake-detector-dfd-sota

Model Description

The "SOTA Deepfake Detector - DFD" is a spatiotemporal adaptation built on a frozen CLIP-ViT-Large backbone with lightweight spatiotemporal adapters inserted to learn temporal relationships across frames. The backbone parameters are frozen; only the adapters and final classification head are trainable.

• Architecture: CLIP-ViT-Large (frozen backbone) + Spatiotemporal Adapters
• Input resolution: 384×384 pixels
• Temporal frames: 12 frames per video
• Trainable parameters: 5,255,938 (≈1.7% of total 308M)
• Precision used for training: BF16 (mixed precision)
• Framework: PyTorch + Transformers (Hugging Face)
• HF Hub repo: Arko007/deepfake-detector-dfd-sota
• Model files:
  • pytorch_model.bin (1.28GB)
  • config.json

Intended Use

This model is intended to classify short video clips (or frame sequences) as either:

• Label 0: Real videos
• Label 1: Deepfake/manipulated videos

Primary use-cases:

• Research on deepfake detection
• Benchmarking against other detectors
• Integration into content-moderation pipelines with caution

Not intended:

• Medical, legal, or other high-stakes decisions without human review.
• Use on domains/styles substantially different from the DFD dataset without further fine-tuning.

Training Data

Dataset: DFD (Deep Fake Detection)

• Total videos: 3,431
  • Real videos: 363 (10.6%)
  • Fake videos: 3,068 (89.4%)
• Preprocessing:
  • 12 evenly-spaced frames extracted per video at 384×384 resolution (total frames: 41,172)
• Train/validation split: 90/10 stratified
• Class balancing during training: WeightedRandomSampler (inverse frequency weights)

Training Configuration

• Batch size: 16 (effective 32 with gradient accumulation)
• Optimizer: AdamW (weight_decay=0.05)
• Learning rate: 5e-6 with cosine decay, 10% warmup
• Epochs: 12
• Loss: Cross-entropy with label smoothing = 0.1
• Gradient clipping: max_grad_norm = 1.0
• Sampling: WeightedRandomSampler to mitigate class imbalance
• Augmentation (training only): random horizontal flip, brightness jitter
• Hardware: NVIDIA L4 GPU (24GB VRAM)
• Random seed: 42 (all RNGs fixed for reproducibility)

Training speed: ~1.4 seconds per batch (batch_size=16) on the reported hardware.

Evaluation / Metrics

• Best validation accuracy: 84.88%
• Validation detection (approx. ranges due to small real class):
  • Real class detection: ~47–55% (low number of real samples)
  • Fake class detection: ~64–93%
• Training loss convergence (cross-entropy w/ label smoothing): 0.7097 → 0.6921 (12 epochs)

Known Limitations

• Validation set is highly imbalanced (89% fake), which affects stability of metrics.
• Small number of real videos (363) limits generalization to unseen real samples.
• Model optimized for the DFD dataset; transfer to other deepfake types may require fine-tuning.
• Temporal context limited to 12 frames (approx. 0.4–1s depending on FPS), so long-term artifacts may be missed.

Usage

Quick inference instructions:

1. Load checkpoint: checkpoint = torch.load("best_model.pt", map_location="cpu")
2. Extract model state: model_state = checkpoint["model_state_dict"]
3. Initialize model: model = DeepfakeDetector(config)
4. Load state dict: model.load_state_dict(model_state)
5. Set to eval: model.eval()
6. Inference: Pass a tensor of shape (batch_size, 12, 3, 384, 384) with dtype float32 (or bf16 where supported).
7. Output:
   • logits: (batch_size, 2)
   • probabilities: softmax(logits)
   • prediction: argmax(logits) -> 0=real, 1=fake

How to Cite

If you use this model in your work, please cite the repository and include details about the DFD dataset and this specific model configuration.

"""

----------------------------

Inference code

----------------------------

The following is a compact, self-contained inference utility. It assumes:

- checkpoint is saved as 'best_model.pt'

- a config object or config.json for the model is available

- CLIP image processor is available via transformers

Important: This implementation is a minimal example to run inference. For

production, wrap in a robust server, add batching, async IO, error handling,

and pre-warming for BF16 on supported accelerators.

import os import math from typing import List, Tuple, Union, Dict

import numpy as np from PIL import Image import torch import torch.nn.functional as F from torchvision import transforms from torchvision.io import read_video # optional, requires torchvision installation from transformers import CLIPImageProcessor, CLIPModel

----------------------------------------------------------------------

Model definition stub

----------------------------------------------------------------------

The real model used for training is a CLIP-ViT-Large backbone (frozen) with

spatiotemporal adapters and a small classification head. For inference the

exact architecture must match the checkpoint. Below is a minimal class

to demonstrate expected load / inference semantics. Replace this with the

model class used during training (and the one saved to the checkpoint).

class DeepfakeDetector(torch.nn.Module): """ Minimal wrapper around CLIP backbone + temporal adapters + classification head.

NOTE: This is a lightweight placeholder for demonstration. Replace with the
exact model definition used during training to successfully load the
provided checkpoint (pytorch_model.bin / best_model.pt).
"""

def __init__(self, clip_model_name: str = "openai/clip-vit-large-patch14", num_frames: int = 12):
    super().__init__()
    self.num_frames = num_frames
    # Load CLIP and freeze it (backbone frozen)
    self.clip = CLIPModel.from_pretrained(clip_model_name, torch_dtype=torch.float32)
    for p in self.clip.parameters():
        p.requires_grad = False

    # Spatiotemporal adapters & head (trainable)
    # NOTE: The real training used custom adapters; here we provide a representative head.
    embed_dim = self.clip.visual_projection.shape[1] if hasattr(self.clip, "visual_projection") else self.clip.config.projection_dim
    self.adapter_pool = torch.nn.AdaptiveAvgPool1d(1)  # placeholder
    # small trainable head consistent with ~5.26M params total
    self.classifier = torch.nn.Sequential(
        torch.nn.Linear(embed_dim, 1024),
        torch.nn.ReLU(),
        torch.nn.Dropout(p=0.1),
        torch.nn.Linear(1024, 2)
    )

def forward(self, x: torch.Tensor) -> torch.Tensor:
    """
    Forward pass.

    x: Tensor of shape (batch, num_frames, 3, H, W)
    Returns logits: (batch, 2)
    """
    b, t, c, h, w = x.shape
    # reshape to (b*t, c, h, w) to run through CLIP's visual encoder
    xt = x.view(b * t, c, h, w)

    # CLIP's image encoder expects pixel values and image processing externally.
    # Use clip.vision_model to obtain image embeddings
    outputs = self.clip.vision_model(pixel_values=xt)
    pooled = outputs.pooler_output  # (b*t, embed_dim) or adjust per CLIP impl

    # reshape back to (b, t, embed_dim) and aggregate temporally
    embed_dim = pooled.shape[-1]
    pooled = pooled.view(b, t, embed_dim)  # (b, t, embed_dim)

    # Simple temporal aggregation (real model uses adapters). For inference placeholder:
    # mean across temporal dimension
    video_repr = pooled.mean(dim=1)  # (b, embed_dim)

    logits = self.classifier(video_repr)  # (b, 2)
    return logits

----------------------------------------------------------------------

Helper utilities for frame extraction, preprocessing and inference

----------------------------------------------------------------------

def extract_evenly_spaced_frames_from_video( video_path: str, num_frames: int = 12, target_size: Tuple[int, int] = (384, 384), ) -> List[Image.Image]: """ Extract num_frames evenly spaced frames from a video file.

Returns a list of PIL Image objects resized to target_size.
Requires torchvision.read_video; as a fallback, uses ffmpeg via PIL + imageio
if read_video is unavailable.
"""
if not os.path.exists(video_path):
    raise FileNotFoundError(f"Video file not found: {video_path}")

try:
    # torchvision's read_video returns (frames, audio, info)
    frames, _, info = read_video(video_path, pts_unit="sec")
    # frames: (num_total_frames, H, W, C) uint8 tensor
    total = frames.shape[0]
    if total == 0:
        raise RuntimeError("No frames extracted from video.")
    indices = np.linspace(0, total - 1, num_frames, dtype=int)
    pil_frames = []
    for i in indices:
        frame = frames[i].numpy()
        img = Image.fromarray(frame)
        img = img.convert("RGB").resize(target_size, resample=Image.BILINEAR)
        pil_frames.append(img)
    return pil_frames
except Exception:
    # Fallback: use imageio-ffmpeg or other mechanism (not implemented here)
    raise RuntimeError("Video reading failed. Ensure torchvision is installed and supports read_video.")

def load_frames_from_folder( folder: str, num_frames: int = 12, target_size: Tuple[int, int] = (384, 384), ) -> List[Image.Image]: """ Load frames (PNG/JPG) from a folder. Picks num_frames evenly across available images. """ files = sorted( [ os.path.join(folder, f) for f in os.listdir(folder) if f.lower().endswith((".png", ".jpg", ".jpeg")) ] ) if not files: raise FileNotFoundError(f"No image files found in folder: {folder}") total = len(files) indices = np.linspace(0, total - 1, num_frames, dtype=int) pil_frames = [] for i in indices: img = Image.open(files[i]).convert("RGB").resize(target_size, resample=Image.BILINEAR) pil_frames.append(img) return pil_frames

def preprocess_frames( pil_frames: List[Image.Image], processor: CLIPImageProcessor, device: Union[str, torch.device] = "cpu", ) -> torch.Tensor: """ Given a list of PIL images (len == 12), apply the CLIP image processor and return a tensor shaped (1, 12, 3, 384, 384) suitable for the model. """ # processor can batch process a list of images; it returns dict with pixel_values # pixel_values shape: (num_images, 3, H, W) proc = processor(images=pil_frames, return_tensors="pt") pixel_values = proc["pixel_values"] # (num_images, 3, H, W) # Ensure exactly num_frames pixel_values = pixel_values.to(device) # reshape to (1, num_frames, 3, H, W) pixel_values = pixel_values.unsqueeze(0) if pixel_values.ndim == 4...