LogicGame-Data Dataset Overview

Introduction

LogicGame is a benchmark for evaluating large language models' (LLMs) understanding, execution, and planning of logical rules. The benchmark comprises various games with predefined rules, designed to assess logical reasoning independent of factual knowledge. Model performance is measured across different difficulty levels to comprehensively evaluate rule‑based reasoning and multi‑step execution and planning.

Data Description

The project includes four .jsonl files: en_dev, zh_dev, en_all, and zh_all, representing development and full sets for English and Chinese respectively. The development set contains 10 entries per language, while the full set contains 304 entries per language.

• zh_all and en_all are used for Codabench submissions; the contexts field can serve as prompts for model responses during evaluation.
• The development set is intended for detailed illustration.

Development Set Fields

• qid: Unique identifier for each entry.
• contexts: Benchmark question, combining rules, the problem, and output constraints.
• reference: Reference JSON answer and process.
• level: Difficulty level ranging from 0 to 3.
• examples: Few‑shot examples.
• category: Data type/category.

Full Set Fields

• qid: Unique identifier.
• contexts: Benchmark question.
• level: Difficulty level.
• category: Data category/task.

Leaderboard

The table below shows the performance of 14 models on the LogicGame benchmark, with the best scores highlighted in bold.

Chinese Models Performance

English Models Performance