Fundus 9model Benchmark

Fundus Lesion Image Classification — 9-Model Comparative Benchmark

Companion artifact for the Master's thesis "Classification of Fundus Lesion Images Using Deep Learning Models" (Xidian University, 2026), by Daryl Panashe Katiyo.

Reproducible weights, predictions, and full statistical analysis for nine deep-learning backbones evaluated on a 10-class colour-fundus dataset with a group-aware (perceptual-hash) 5-fold cross-validation protocol.

---

Important Note on Tables

Two result tables appear in this README. They measure different things and must not be mixed:

Table	Protocol	# runs	Use for
§ 5.1 — 5-fold CV (authoritative)	5 independent train/val splits, fixed holdout test	5	Paper, model comparison, all citations
§ 5.2 — Single-run baseline	One training run, fold-0 split	1	Legacy reference, documents CLIP variance

Always cite § 5.1 in the paper. The single-run baseline (§ 5.2) is retained for completeness — CLIP underperforms there (86.25%) due to a single unlucky initialisation versus 90.15% over 5 folds.

---

1. Abstract

Automatic interpretation of colour fundus photographs is a foundational task for screening prevalent blinding diseases such as diabetic retinopathy, glaucoma and age-related macular degeneration. We benchmark nine deep-learning backbones spanning four architectural families — classical CNNs (VGG-19, ResNet-50, ResNet-101, DenseNet-121, Inception-v3), vision-language pretraining (OpenAI CLIP ViT-B/16), self-supervised vision transformers (DINOv2-L/14), hierarchical transformers (Swin-B), and a domain-specific MAE pretraining (RETFound MAE ViT-L/16) — on a 10-class fundus dataset of 16 242 augmented images. To suppress augmentation-induced label leakage we construct a group-aware (perceptual-hash) stratified 5-fold split and report mean accuracy, F1, ROC-AUC, ECE, Cohen's κ, Brier score, bootstrap 95% confidence intervals, Bonferroni-corrected McNemar tests, and 90% Mondrian conformal sets.

Headline result (5-fold CV). Classical CNNs and CLIP ViT-B/16 are statistically indistinguishable at the top, with Inception-v3 leading at 90.18% accuracy (F1 = 92.54%, κ = 0.884, ROC-AUC = 0.9930). Contrary to expectations, foundation models pretrained on general vision (DINOv2-L: 89.61%, Swin-B: 87.00%) or fundus images (RETFound: 83.35%) do not outperform classical CNNs on this moderate-sized dataset. A soft-vote ensemble of all nine models reaches ROC-AUC = 0.9941 and accuracy = 89.68%.

---

2. Motivation & Model Selection

Modern fundus screening pipelines are increasingly built on pre-trained image backbones, but the question "which backbone family is best for fundus disease classification on a moderately-sized, imbalanced dataset?" has no consensus answer. We deliberately chose backbones that exercise four distinct inductive biases / pretraining regimes:

Family	Backbone(s)	Why included
Classical CNNs	VGG-19, ResNet-50, ResNet-101, DenseNet-121, Inception-v3	Established baselines used in virtually all prior fundus benchmarks ([Gulshan 2016][1], [Ting 2017][2]). Locally-connected convolutions suit texture-dominant retinal pathology.
Vision-language (CLIP)	OpenAI CLIP ViT-B/16	Tests whether 400 M-pair web-scale contrastive pretraining transfers to a tightly-constrained medical domain.
Self-supervised ViT	DINOv2-L/14	State-of-the-art general-purpose features without language supervision ([Oquab 2024][3]).
Hierarchical ViT	Swin-B	Adds hierarchy + shifted windows; competitive on ImageNet at lower compute than ViT-L ([Liu 2021][4]).
Domain MAE	RETFound MAE ViT-L/16	Pretrained on 1.6 M colour fundus images ([Zhou 2023, Nature][5]); the strongest published prior on this modality.

This grid isolates three confounders: (i) scale (ResNet-50 vs ResNet-101; ViT-B vs ViT-L); (ii) pretraining modality (ImageNet supervised vs CLIP language-supervised vs DINOv2 self-supervised vs RETFound domain-MAE); and (iii) architecture class (CNN vs ViT vs hierarchical).

---

3. Dataset

• Source. [Mendeley Data][6] (10 classes; 5 335 original images).
• Augmentation. Class-balancing augmentation expanded the pool to 16 242 images (rotation, horizontal flip, brightness/contrast jitter, Gaussian blur). Each augmented image carries its source's diagnostic label.
• Companion dataset: DoB24/fundus-10class-augmented.

3.1 Class distribution

Index	Class	Original	Augmented
0	Central Serous Chorioretinopathy	101	606
1	Diabetic Retinopathy	1 509	3 444
2	Disc Edema	127	762
3	Glaucoma	1 349	2 880
4	Healthy	1 024	2 676
5	Macular Scar	444	1 937
6	Myopia	500	2 251
7	Pterygium	17	102
8	Retinal Detachment	125	750
9	Retinitis Pigmentosa	139	834
—	Total	5 335	16 242

3.2 Group-aware splitting (data-leakage prevention)

Because the augmented set contains visually near-duplicate copies of each original image, a naïve train_test_split over the augmented pool would let models memorise patient-level identities. We prevent this by:

1. Computing a 64-bit perceptual hash (pHash) on every image.
2. Linking each augmented image to its nearest original at Hamming distance ≤ 8 → defines a group_id.
3. Running scikit-learn StratifiedGroupKFold(n_splits=5) so that all augmented children of a given original sit in exactly one fold.

The held-out test set is fixed across all 5 folds: 3 208 images (≈ 19.8% of the augmented pool). Exact manifest: splits/holdout_split_augmented.json (3.2 MB).

---

4. Training Protocol

4.1 CNN / CLIP backbones

Hyper-parameter	Value
Optimizer	AdamW (β1=0.9, β2=0.999, weight-decay=1×10−4)
Initial LR	2×10−4
LR schedule	3-epoch linear warm-up + cosine decay to 0
Epochs	Up to 60 (early stop patience=12 on val F1)
Batch size	32
Image size	224×224 (Inception-v3: 299×299)
Preprocessing	CLAHE (LAB L-channel) → RandAugment (n=2, m=9) → ImageNet normalisation
Imbalance	WeightedRandomSampler (weights ∝ 1/class_count)
Regularisation	MixUp (α=0.2) + CutMix (α=1.0, p=0.7)
Mixed precision	torch.amp.autocast + GradScaler
Test-time aug	6 views (centre + 4 corners + h-flip), soft-vote mean
Hardware	PyTorch 2.11 + CUDA 12.8, NVIDIA Tesla T4 (16 GB)

4.2 Foundation model backbones (DINOv2-L, Swin-B, RETFound)

Two-stage schedule per fold:

Stage	Layers trained	Epochs	Head LR	Backbone LR
Linear probe	Head only	20	1×10−3	frozen
Full fine-tune	All layers	15	1×10−4	1×10−5

Batch size: 24. Early stopping patience: 8 epochs on val F1.

4.3 Ensemble

F1-weighted soft-vote across all 9 models, using each model's validation F1 as the weight.

---

5. Results

All metrics are on the fixed 3 208-image holdout test set.

---

5.1 Five-fold cross-validation — authoritative results (use in paper)

Acc, F1, 95% CI, ROC-AUC, and ECE are means over 5 independent training runs. κ and Brier for CNN/CLIP are from 5-fold pooled predictions. For foundation models (†), κ and Brier are from single-run inference on the same holdout (fold-level predictions not stored); acc/F1/ROC/ECE are still 5-fold averages.

Rank	Model	Acc (%)	95% CI	F1 (%)	F1 95% CI	κ	Brier	ROC-AUC	ECE
1	inception_v3	90.18	[89.24, 91.24]	92.54	[91.68, 93.40]	0.884	0.150	0.9930	0.0194
2	clip_openai	90.15	[89.18, 91.24]	92.83	[92.00, 93.61]	0.884	0.140	0.9944	0.0217
3	vgg19	90.12	[89.09, 91.12]	92.59	[91.77, 93.41]	0.884	0.150	0.9933	0.0228
4	resnet101	90.09	[89.15, 91.12]	92.63	[91.77, 93.47]	0.883	0.140	0.9941	0.0243
5	densenet121	89.65	[88.62, 90.71]	92.29	[91.37, 93.07]	0.878	0.150	0.9937	0.0272
6	dinov2_l	89.61	[88.57, 90.64]	92.27	[91.38, 93.08]	0.876†	0.160†	0.9934	0.0299
7	resnet50	89.50	[88.40, 90.59]	92.20	[91.34, 93.03]	0.876	0.140	0.9945	0.0339
8	swin_b	87.00	[85.92, 88.15]	90.26	[89.31, 91.19]	0.845†	0.190†	0.9896	0.0294
9	retfound	83.35	[82.16, 84.65]	87.27	[86.22, 88.35]	0.810†	0.240†	0.9834	0.0242
—	9-Model Ensemble	89.68	[88.65, 90.74]	92.25	—	0.878	0.144	0.9941	0.0198

† κ and Brier for DINOv2-L, Swin-B, RETFound from single-run holdout inference. All other foundation model metrics are 5-fold CV averages.

Key findings:

• Top-4 models (Inception-v3, CLIP, VGG-19, ResNet-101) are statistically indistinguishable: all 6 pairwise McNemar tests p > 0.05 after Bonferroni correction (see kfold/cnn_clip/mcnemar.json).
• DINOv2-L (89.61%) matches DenseNet-121 (89.65%) within noise despite having 7× more parameters.
• RETFound, pretrained on 1.6 M fundus images, ranks last — the LP+FT protocol with patience=8 may be insufficient on this dataset size.
• The ensemble gains ROC-AUC parity with the best individual model.

Machine-readable full table: kfold/kfold_v2_summary.csv

---

5.2 Single-run baseline (§ 5.2 — for reference only, do not cite in paper)

Retained to document the original preliminary experiment (one run, fold-0 split). CLIP's 86.25% here vs 90.15% in § 5.1 is single-run variance, not an architectural effect.

Rank	Model	Acc (%)	95% CI	F1 (%)	κ	Brier	ROC-AUC
1	densenet121	89.78	[88.71, 90.80]	92.26	0.879	0.148	0.9931
2	dinov2_l	89.50	[88.47, 90.55]	92.15	0.876	0.155	0.9938
3	vgg19	89.31	[88.22, 90.40]	92.12	0.874	0.154	0.9930
4	resnet101	89.25	[88.19, 90.34]	92.05	0.873	0.149	0.9941
5	inception_v3	89.21	[88.15, 90.28]	91.97	0.873	0.157	0.9934
6	resnet50	89.09	[88.00, 90.12]	91.87	0.871	0.147	0.9944
7	swin_b	86.85	[85.69, 88.03]	90.44	0.845	0.185	0.9904
8	clip_openai	86.25	[85.10, 87.41]	89.99	0.838	0.195	0.9896
9	retfound	83.88	[82.64, 85.10]	87.68	0.810	0.238	0.9838
—	9-Model Ensemble	89.68	[88.65, 90.74]	92.25	0.878	0.144	0.9941

The ensemble row is identical in both tables — it is computed once on the fixed holdout and does not depend on the training run.

---

5.3 Statistical significance

All 15 pairwise Bonferroni-corrected McNemar tests among the 6 CNN/CLIP models: p > 0.05 — no statistically significant pairwise differences. Full χ2 and p-value matrix: kfold/cnn_clip/mcnemar.json.

---

5.4 Calibration (ECE)

Model	ECE
inception_v3	0.0194	Best calibrated
clip_openai	0.0217
vgg19	0.0228
retfound	0.0242
resnet101	0.0243
densenet121	0.0272
swin_b	0.0294
dinov2_l	0.0299
resnet50	0.0339
9-Model Ensemble	0.0198	Best overall