CoMix: Comics Dataset Framework for Comics Understanding

Introduction

This project aims to reproduce (on the validation set) the following benchmarks:

• (Detection) Comics Datasets Framework: Mix of Comics datasets for detection benchmarking
• (Multi‑task) CoMix: A Comprehensive Benchmark for Multi‑Task Comic Understanding
• (Caption Generation) ComiCap: A VLMs pipeline for dense captioning of Comic Panels

The main limitation is the inability to share images. To address this, we created this framework, allowing the use of our (validation) annotations and downloading images from the original sources without violating licenses.

comix uses the following datasets:

• DCM
• comics
• eBDtheque
• PopManga
• Manga109

Installation

The project is written in Python 3.8. Create a conda environment:

conda create --name myenv python=3.8
conda activate myenv

Install dependencies:

pip install -e .

Workflow

The project is divided into the following steps:

• Manually obtain and place images and annotations into the correct folder (e.g., data/)
• Process images into a unified naming and folder structure – comix/process
• Model performance (using pretrained or custom models) – benchmarks
• Evaluate model performance against the provided Ground Truth – comix/evaluators

Model Performance and Evaluation

In the benchmarks folder, multiple scripts are provided for benchmarking the dataset on various tasks. Detection scripts generate COCO‑format JSON files, which can be evaluated with the comix/evaluators/detection.py script. Captioning scripts generate multiple .txt files, which can be post‑processed into captions.csv and objects.csv and evaluated with the comix/evaluators/captioning.py script.

Documentation

Documentation is located in the /docs folder.

Main documents:

• docs/README.md
  • docs/datasets/README.md
  • docs/tasks/detection/README.md
    • docs/tasks/detection/generation.md
    • docs/tasks/detection/evaluation.md
  • docs/tasks/captioning/README.md
    • docs/tasks/captioning/generation.md
    • docs/tasks/captioning/postprocessing.md
    • docs/tasks/captioning/evaluation.md