MarkushGrapher 2

--- license: apache-2.0 language: - en tags: - chemistry - markush - cxsmiles - molecular-structure - ocr - document-understanding - image-to-text - vision-language-model - patent-analysis library_name: transformers pipeline_tag: image-to-text datasets: - docling-project/MarkushGrapher-2-Datasets --- <p align="center"> <img src="markushgrapher_2_repo_banner.png" alt="MarkushGrapher-2 Banner" width="100%"> </p> **MarkushGrapher-2** is an end-to-end multimodal model for recognizing chemical structures from patent document images. It jointly encodes vision, text, and layout information to convert Markush structure images into machine-readable CXSMILES representations. ## Overview MarkushGrapher-2 is a transformer-based model that integrates two complementary encoders: - A **Vision Encoder** (Swin-B ViT), pretrained for Optical Chemical Structure Recognition (OCSR) (taken from Model: MolScribe*) - A **Vision-Text-Layout (VTL) Encoder** (T5-base), trained for Markush feature extraction The model also includes **ChemicalOCR**, a dedicated OCR module fine-tuned for chemical images, enabling fully end-to-end processing without external OCR dependencies. *MolScribe: https://github.com/thomas0809/MolScribe ### Architecture The input image is processed through two parallel pipelines: 1. **Vision pipeline**: The image is encoded by the OCSR vision encoder and projected via an MLP. 2. **VTL pipeline**: The image is passed through ChemicalOCR to extract text and bounding boxes, which are then fused with image patches in the VTL encoder. The outputs of both pipelines are concatenated and fed to a text decoder that autoregressively generates a CXSMILES sequence describing the Markush backbone and a substituent table. ### Two-Stage Training Strategy - **Phase 1 (Adaptation)**: The Vision Encoder is frozen while the projector and text decoder are trained on 243K real-world image-SMILES pairs for standard molecular structure recognition. - **Phase 2 (Fusion)**: The VTL encoder is introduced and trained jointly with the text decoder on 235K synthetic and 145K real-world Markush structure samples for CXSMILES and substituent table prediction. ## Performance ### Markush Structure Recognition (CXSMILES Accuracy) | Benchmark | MarkushGrapher-2 | MolParser-Base | MolScribe | MarkushGrapher-1 | DeepSeek-OCR | GPT-5 | |---|---|---|---|---|---|---| | **M2S*...

Source context: 175 downloads · 4 likes · Pipeline image-to-text · Library transformers · Repo docling-project/MarkushGrapher-2