Face Multitask v2

face_multitask_v2

A single multi-task convolutional model for facial behavior analysis, used by py-feat's Detectorv2. From one face crop it jointly predicts action units, categorical emotion, valence/arousal, eye gaze, a 478-point face mesh, 6-DoF head pose, and 52 MediaPipe/ARKit blendshapes (the v2.7 model; replaces v2.6).

• Backbone: ConvNeXt-V2 Tiny (FCMAE + IN-22k/IN-1k pretrained)
• Heads: ME-GraphAU AU graph (AFG/FGG/SC) + unified-feature emotion/V-A heads
  • landmark, pose, and blendshape regression heads + the v2.6 eye-aware gaze head: RoI-pooled eye features (localized by the predicted mesh), conditioned on predicted head pose (6D), with an L2CS-style binned prediction over the full ±180° range (v2.7: 2° bins)
• Params: ~42M · Input: 224×224 RGB (from a 256×256 face crop)
• File: face_multitask_v27.safetensors (safetensors; ModelV2Config JSON in the file metadata)
• Weights: soup (equal average) of five consecutive fine-tuning checkpoints

Outputs

Task	Output	Notes
Action Units	20 probabilities [0,1]	AU01,02,04,05,06,07,09,10,11,12,14,15,17,20,23,24,25,26,28,43
Emotion	7-class softmax	Neutral, Happy, Sad, Surprise, Fear, Disgust, Anger
Valence / Arousal	2 × [−1,1]	tanh
Gaze	(yaw, pitch) radians	RAW convention is y-down: yaw+ = subject's right (image-left), pitch+ = looking DOWN. Detectorv2 negates pitch so Fex columns are canonical +up (since py-feat 2.1.1)
Face mesh	478 × (x,y,z)	MediaPipe topology, chip-pixel coords (z = relative depth)
Head pose	(pitch, yaw, roll, tx, ty, tz)	radians / pixels; RAW pitch+ = down (img2pose teacher frame); Detectorv2 outputs canonical +up (since py-feat 2.1.1)
68 landmarks	derived	dlib-68 subset sampled from the 478 mesh
Blendshapes	52 coefficients [0,1]	MediaPipe/ARKit standard names (browInnerUp, jawOpen, mouthSmileLeft, ...)

Benchmarks (held-out, file-verified — v2.7 deployed checkpoint)

All gaze splits are identity-disjoint from training (held-out subjects), and EYEDIAP is never trained on by any py-feat model.

Task	Dataset	Metric	v2.7	v2.6	v2.5
AU	DISFA+ (12-AU, Cheong protocol)	macro-F1	0.686	0.696	0.693
AU	DISFA+ (8-AU subset)	macro-F1	0.740	0.738	0.740
Emotion	AffectNet val (7-cls, drop Contempt)	acc / macro-F1	0.612 / 0.607	0.615 / 0.610	0.616 / 0.612
Emotion	RAF-DB test	acc / macro-F1	0.876 / 0.817	0.873 / 0.818	0.910 / 0.885
Valence/Arousal	AffectNet val	CCC (V / A)	0.773 / 0.647	0.775 / 0.653	0.780 / 0.646
Valence/Arousal	AFEW-VA	CCC (V / A)	0.711 / 0.480	0.718 / 0.411	0.833 / 0.863
Valence/Arousal	Aff-Wild2 val	CCC (V / A)	0.331 / 0.418	0.397 / 0.458	0.852 / 0.799
Gaze	ETH-XGaze (held-out subjects)	mean angular err	5.1°	5.0°	43.2°
Gaze	EYEDIAP (never-train, 15.2K frames)	mean angular err	12.6°	13.4°	15.3°
Gaze	Gaze360 (held-out split)	mean angular err	13.0°	13.0°	12.9°
Gaze	MPIIGaze (leave-subject-out)	mean angular err	8.0°	7.4°	7.0°
Gaze	Columbia (held-out subjects)	mean angular err	4.1°	5.4°	— (trained)
Blendshapes	FacePlace (teacher agreement)	mean active-ch. r	0.761	0.748	0.756

Notes: v2.7 = the v2.6 architecture retrained with a rescaled gaze loss, a rebalanced within-gaze data mix, per-source augmentation, 2° gaze bins, and a head-pose label fix. Vs v2.6 it improves out-of-distribution gaze (EYEDIAP −0.7°, Columbia −1.3°), RAF-DB macro-F1 end-to-end (+2.7), AFEW-VA arousal, blendshape fidelity (best of any release), and occlusion robustness, at the cost of ~0.6° on frontal MPIIGaze and 0.01 on the 12-AU set.

Known limitation (v2.6 and v2.7): continuous valence/arousal on video-frame corpora (Aff-Wild2, AFEW-VA) is substantially below v2.5 (e.g. Aff-Wild2 CCC-V 0.85 → 0.33). AffectNet (still-image) V/A is unaffected. If frame-wise continuous V/A on video is your primary measure, prefer the v2.5 weights (face_multitask_v2.safetensors, still published in this repo). A fix is under investigation. Numbers are from the deployed checkpoint (v2.7 stage-3 soup ep05-09), weight-verified against the published .safetensors.

Usage

from feat import Detectorv2
detector = Detectorv2(device="cuda")
fex = detector.detect("image.jpg")   # returns a py-feat Fex

The model expects a face crop produced by RetinaFace + py-feat's extract_face_from_bbox_torch(frame, bbox, face_size=256, expand_bbox=1.2), then resized to 224 and ImageNet-normalized. Detectorv2 handles this.

License

Research / non-commercial use only. Trained on datasets (AffectNet, DISFA+, RAF-DB, Aff-Wild2, BP4D, etc.) whose licenses restrict use to academic research. The ConvNeXt-V2 backbone is MIT-licensed. Confirm each constituent dataset's terms before any non-research use.