WorldCuisines: A Massive‑Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines

Overview

WorldCuisines is a massive‑scale VQA benchmark for multilingual and multicultural understanding via global cuisines. The dataset spans 30 languages and dialects across nine language families, containing over one million data points, the largest multicultural VQA benchmark as of October 17 2024.

Dataset Composition

• WC‑VQA: VQA dataset built on the WC‑KB.
• WC‑KB: Knowledge base of global cuisines, comprising 2,414 dishes, 6,045 images and metadata, covering coarse‑ and fine‑grained classification, geography, and regional cuisine.

Task Design

1. Task 1: Dish Name Prediction

   • (a) Context‑free Questions: Predict dish name without additional context.
   • (b) Contextualized Questions: Provide context and predict dish name.
   • (c) Adversarial Contextualized Questions: Provide misleading context and predict dish name.

2. Task 2: Geographic Location Prediction

   • Based on dish image, question, and context, predict the typical consumption or origin location of the dish.

Dataset Scale

• WC‑VQA: 1 M samples across 30 languages and dialects.
• Two evaluation subsets (12 k and 60 k instances) and a training set (1.08 M instances) are provided.

Dataset Construction

Data Sources

• Data collected from Wikipedia and Wikimedia Commons.

Construction Steps

1. Dish Selection: Filter culturally significant dishes from Wikipedia.
2. Metadata Annotation: Manually compile dish metadata, including visual representation, classification, description, cuisine, and geographic distribution.
3. Quality Assurance: Multi‑round checks to ensure data quality.
4. Data Compilation: Merge metadata into a single file.

VQA Generation

Generation Process

1. Dish Name Similarity Search: Use multilingual models to compute text embeddings and identify similar dishes.
2. Question and Context Construction: Build various question types based on context.
3. Multilingual Translation: Translate questions and contexts into 30 languages and dialects.
4. VQA Triplet Generation: Ensure no overlap between train and test sets; randomly sample dishes and question pairs to generate VQA data.

Ethical Considerations

• Crowdsourced annotation with transparency and fairness.
• Dataset released under CC‑BY‑SA 4.0 license.

Contact Information

• Email: Genta Indra Winata and Frederikus Hudi

Citation

bibtex @misc{winata2024worldcuisinesmassivescalebenchmarkmultilingual, title={WorldCuisines: A Massive‑Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines}, author={Genta Indra Winata and Frederikus Hudi and Patrick Amadeus Irawan and David Anugraha and Rifki Afina Putri and Yutong Wang and Adam Nohejl and Ubaidillah Ariq Prathama and Nedjma Ousidhoum and Afifa Amriani and Anar Rzayev and Anirban Das and Ashmari Pramodya and Aulia Adila and Bryan Wilie and Candy Olivia Mawalim and Ching Lam Cheng and Daud Abolade and Emmanuele Chersoni and Enrico Santus and Fariz Ikhwantri and Garry Kuwanto and Hanyang Zhao and Haryo Akbarianto Wibowo and Holy Lovenia and Jan Christian Blaise Cruz and Jan Wira Gotama Putra and Junho Myung and Lucky Susanto and Maria Angelica Riera Machin and Marina Zhukova and Michael Anugraha and Muhammad Farid Adilazuarda and Natasha Santosa and Peerat Limkonchotiwat... (truncated for brevity), year={2024}, eprint={2410.12705}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2410.12705}, }