Dataset Card: CTIBench

Dataset Overview

CTIBench is a suite of benchmark tasks and datasets for assessing LLMs on cyber‑threat intelligence (CTI) tasks.

Dataset Details

Dataset Description

CTIBench is a comprehensive benchmark suite designed to evaluate LLM performance in the CTI domain.

Components:

• CTI‑MCQ: A knowledge‑assessment dataset of multiple‑choice questions evaluating LLM understanding of CTI standards, threats, detection strategies, mitigation plans, and best practices. Built from authoritative sources including NIST, MITRE, and GDPR.
• CTI‑RCM: A practical task mapping Common Vulnerabilities and Exposures (CVE) descriptions to Common Weakness Enumeration (CWE) categories, testing LLM capability to understand and classify cyber threats.
• CTI‑VSP: A task requiring the calculation of Common Vulnerability Scoring System (CVSS) scores, assessing LLM ability to evaluate vulnerability severity.
• CTI‑TAA: A task involving analysis of public threat reports and attribution to specific threat actors or malware families, testing LLM comprehension of historical cyber‑threat behavior and meaningful correlation identification.

Dataset Source

Repository: https://github.com/xashru/cti-bench

Dataset Structure

The dataset consists of 5 TSV files, each corresponding to a different task. Each TSV includes a "Prompt" column posing the question to the LLM. Most files also contain a "GT" column with the ground‑truth answer, except for "cti‑taa.tsv". Evaluation scripts for each task are available in the associated GitHub repository.

Dataset Creation

Rationale

The dataset was created to benchmark LLMs' ability to understand and analyze various aspects of open‑source CTI.

Source Data

URLs indicating the origins of the collected data are included in the dataset.

Personal and Sensitive Information

The dataset contains no personal or sensitive information.

Citation

Paper link: https://arxiv.org/abs/2406.07599

BibTeX:

@misc{alam2024ctibench,
      title={CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence}, 
      author={Md Tanvirul Alam and Dipkamal Bhushal and Le Nguyen and Nidhi Rastogi},
      year={2024},
      eprint={2406.07599},
      archivePrefix={arXiv},
      primaryClass={cs.CR}
}

Contact

Md Tanvirul Alam (ma8235@rit.edu)