SAM‑Med2D Dataset Overview

🌤️ Highlights

• Collected and organized the largest medical image segmentation dataset to date (4.6 million images and 19.7 million masks).
• Conducted the most comprehensive fine‑tuning based on the Segment Anything Model (SAM).
• Performed a thorough evaluation of SAM‑Med2D on a large‑scale dataset.

🔥 Updates

• (2023.12.05) Dataset released for download on the Hugging Face platform.
• (2023.11.23) Released the SA‑Med2D‑20M dataset.
• (2023.11.21) Published a paper introducing the SA‑Med2D‑20M dataset.
• (2023.10.24) Released SAM‑Med3D, focusing on 3D medical image segmentation.
• (2023.09.14) Released training code.
• (2023.09.02) Released testing code.
• (2023.08.31) Released pretrained model.
• (2023.08.31) Released paper.
• (2023.08.26) Released online demo.

👉 Dataset

SAM‑Med2D was trained and evaluated on a dataset containing 4.6 million images and 19.7 million masks. The dataset covers 10 medical data modalities, 4 anatomical structures + lesion categories, and 31 major human organs. To the best of our knowledge, this is the largest medical image segmentation dataset in terms of volume and category coverage.

👉 Framework

SAM‑Med2D workflow. We freeze the image encoder and insert learnable adapter layers into each Transformer block to acquire domain‑specific knowledge. Prompt encoder is fine‑tuned with point, Bbox, and mask information, and the mask decoder is updated via interactive training.

👉 Results

Quantitative Comparison

Generalization Validation

👉 Training

Prepare your own dataset and replace the examples in SAM‑Med2D/data_demo. Before running train.py, generate the image2label_train.json file.

cd ./SAM‑Med2D
python train.py

Parameters

• work_dir: Directory for training outputs (default workdir).
• image_size: Default 256.
• mask_num: Number of masks per image (default 5).
• data_path: Dataset directory, e.g., data_demo.
• resume: Path to pretrained weights; if provided, ignore sam_checkpoint.
• sam_checkpoint: Load SAM checkpoint.
• iter_point: Iteration count for mask decoder.
• multimask: Whether to output multiple masks (default True).
• encoder_adapter: Whether to fine‑tune adapter layers; set False when only decoder is fine‑tuned.
• use_amp: Use mixed‑precision training.

👉 Testing

Prepare your own dataset and replace the examples in SAM‑Med2D/data_demo. Before running test.py, generate the label2image_test.json file.

cd ./SAM‑Med2D
python test.py

Parameters

• work_dir: Directory for testing outputs (default workdir).
• batch_size: Default 1.
• image_size: Default 256.
• boxes_prompt: Use Bbox prompts for segmentation.
• point_num: Number of points (default 1).
• iter_point: Iterations for point prompts.
• sam_checkpoint: Load SAM or SAM‑Med checkpoint.
• encoder_adapter: Set to True when using SAM‑Med2D pretrained weights.
• save_pred: Save prediction results.
• prompt_path: Path to a fixed prompt file; if None, prompts are generated automatically during inference.

👉 Deployment

Export to ONNX

• Export encoder model

python3 scripts/export_onnx_encoder_model.py --sam_checkpoint /path/to/sam‑med2d_b.pth --output /path/to/sam‑med2d_b.encoder.onnx --model-type vit_b --image_size 256 --encoder_adapter True

• Export decoder model

python3 scripts/export_onnx_model.py --checkpoint /path/to/sam‑med2d_b.pth --output /path/to/sam‑med2d_b.decoder.onnx --model-type vit_b --return-single-mask

• Run inference with onnxruntime

# cd examples/SAM‑Med2D‑onnxruntime
python3 main.py --encoder_model /path/to/sam‑med2d_b.encoder.onnx --decoder_model /path/to/sam‑med2d_b.decoder.onnx

🚀 Try SAM‑Med2D

• 🏆 Gradio online demo: Available on OpenXLab.
• 🏆 Notebook demo: Use predictor_example.ipynb locally to view predictions under different prompts.
• 🏆 Local Gradio deployment: Deploy app.ipynb locally and upload test cases.

🗓️ Ongoing

• Dataset release
• Dataset paper release
• Training code release
• Testing code release
• Pretrained model release
• Paper release
• Online demo release

🎫 License

The project is released under the Apache 2.0 License.

💬 Discussion Group

For any questions about SAM‑Med2D, add the WeChat ID to join the discussion group.

🤝 Acknowledgements

• Thanks to all medical professionals and dataset owners for providing public datasets.
• Thanks to the open‑source projects: Segment Anything.

👋 Recruitment & Global Collaboration

• Recruitment: Our General Vision team at the Shanghai AI Lab is hiring researchers, engineers, and interns. If you are interested in medical foundation models and general medical AI, contact us.
• Global Collaboration: We aim to redefine medical research and advance the medical community. Partners are welcome to increase competitiveness, reduce risk, and expand markets.
• Contact: Junjun He (hejunjun@pjlab.org.cn), Jin Ye (yejin@pjlab.org.cn), and Tianbin Li (litianbin@pjlab.org.cn).

References

@misc{cheng2023sammed2d, title={SAM‑Med2D}, author={Junlong Cheng and Jin Ye and Zhongying Deng and Jianpin Chen and Tianbin Li and Haoyu Wang and Yanzhou Su and Ziyan Huang and Jilong Chen and Lei Jiang and Hui Sun and Junjun He and Shaoting Zhang and Min Zhu and Yu Qiao}, year={2023}, eprint={2308.16184}, archivePrefix={arXiv}, primaryClass={cs.CV} }

@misc{ye2023samed2d20m, title={...} }