CTFAIA Dataset Overview

Dataset Name

• Name: Capture The Flag (CTF) AI Assistants Benchmark
• Abbreviation: CTFAIA

Dataset Description

• Purpose: Evaluate next‑generation large language models in cybersecurity, particularly on CTF competition problems.
• Features: Over 100 non‑trivial problems with clear answers, requiring varying degrees of tool use and autonomous problem‑solving.
• Structure: Three difficulty levels, each with different tool‑usage and reasoning requirements.
• Data Split: Public development set for validation and a private test set with hidden answers and metadata.

Dataset Contents

• Problem Storage: Problems are stored in a metadata.jsonl file.
• Auxiliary Files: Some problems include additional folders, identified by the Annex field.

Access and Contribution

• Leaderboard: View the CTFAIA leaderboard here.
• Dataset Link: Access the dataset here.