CLEAR

CLEAR — Vision-Language Backbone

This repository hosts the pretrained vision-language backbone for CLEAR (Concept-Level Embeddings for Auditable Radiology). The checkpoint contains a contrastive image–text encoder that maps chest X-rays and radiological text into a shared 768-dimensional embedding space. Given a chest X-ray, the image encoder produces a feature vector whose cosine similarity with each of 368,294 text-encoded radiological observations yields a concept score vector. This concept score vector is then projected through LLM-derived semantic embeddings to produce the final CLEAR image embedding used for zero-shot classification, supervised linear probing, and concept bottleneck models. The concept bank, LLM embeddings, and downstream inference code are available in the GitHub repository.

Files in This Repository

File	Description
best_model.pt	CLEAR vision-language backbone checkpoint (DINOv2 ViT-B/14 + text encoder)
mimic_concepts.csv	Full concept vocabulary (368,294 radiological observations extracted from reports)
concept_embeddings_368294.pt	Precomputed SFR-Embedding-Mistral embeddings for all 368,294 concepts (4096-dim)

Quick Start

import torch
from PIL import Image
from torchvision.transforms import Compose, Resize, CenterCrop, ToTensor, Normalize, InterpolationMode
from huggingface_hub import hf_hub_download

# Download the checkpoint
ckpt_path = hf_hub_download(repo_id="peterhan91/CLEAR", filename="best_model.pt")

# Clone the CLEAR repo for model code: git clone https://github.com/peterhan91/CLEAR
from clear.model import CLIP
from clear import tokenize
from examples.train import load_clip

# Load the CLEAR checkpoint (DINOv2 ViT-B/14 + text encoder)
model = load_clip(
    model_path=ckpt_path,
    use_dinov2=True,
    dinov2_model_name="dinov2_vitb14",
)
model.eval()

# Preprocess a chest X-ray
preprocess = Compose([
    Resize(448, interpolation=InterpolationMode.BICUBIC),
    CenterCrop(448),
    ToTensor(),
    Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
device = next(model.parameters()).device
image = preprocess(Image.open("path/to/cxr.jpg").convert("RGB")).unsqueeze(0).to(device)

# Encode image and text
with torch.no_grad():
    image_features = model.encode_image(image)
    text_tokens = tokenize(["pleural effusion", "no pleural effusion"]).to(device)
    text_features = model.encode_text(text_tokens)

    # Cosine similarity → softmax probability
    image_features = image_features / image_features.norm(dim=-1, keepdim=True)
    text_features = text_features / text_features.norm(dim=-1, keepdim=True)
    logits = (image_features @ text_features.T).softmax(dim=-1)

print(logits)  # [[prob_positive, prob_negative]]

The full CLEAR pipeline projects these concept similarity scores through LLM-derived semantic embeddings (SFR-Embedding-Mistral) for auditable zero-shot classification. See the GitHub repository for benchmarking, concept bottleneck models, and model auditing scripts.

Checkpoint Details

Attribute	Value
Image Encoder	DINOv2 ViT-B/14 with registers
Text Encoder	12-layer Transformer (GPT-2 style, 512-dim, 8 heads)
Embedding Dimension	768
Input Resolution	448 x 448 pixels
File	best_model.pt

Training Data

The model was trained on 873,342 CXR–report pairs (239,091 patients) using symmetric contrastive learning (InfoNCE):

• MIMIC-CXR (377,110 pairs)
• CheXpert-Plus (223,228 pairs)
• ReXGradient (273,004 pairs)

Training details: 3 x NVIDIA A6000, effective batch size 768, AdamW (lr=1e-4, weight decay 0.2), cosine annealing with 500 warmup steps, 40 epochs.

Citation

@article{han2026clear,
  title={CLEAR: An Auditable Foundation Model for Radiology Grounded in Clinical Concepts},
  author={Han, Tianyu and Wu, Riga and Tian, Yu and Khader, Firas and Adams, Lisa and Bressem, Keno and Davatzikos, Christos and Kather, Jakob Nikolas and Shen, Li and Mankoff, David and others},
  journal={medRxiv},
  pages={2026--01},
  year={2026},
  publisher={Cold Spring Harbor Laboratory Press}
}

License

Apache-2.0