Difficult Long‑context Retrieval Tasks Dataset

Dataset Overview

The dataset supports the paper "Hyper‑multi‑step: The Truth Behind Difficult Long‑context Tasks" and contains a series of long‑context retrieval tasks.

Task Types

Simple Tasks

• simple_k2v: Direct key‑to‑value retrieval. Given a key, the model must retrieve the corresponding value.
• simple_v2k: Direct value‑to‑key retrieval. Given a value, the model must retrieve the corresponding key.
• multi_step(kv): Multi‑step (formal) KV retrieval. The model retrieves multiple values through several queries, concatenates them to form a new key, and finally retrieves the associated value.

Difficult Tasks

• logic(kv): Logic‑based KV retrieval. All values range from 0‑9. Given a value range, the model must retrieve the matching keys.
• logic(resume): Logic‑based resume retrieval. Given a GPA range, the model must retrieve students whose GPA falls within that range.
• multi_match(kv): Multi‑match KV retrieval. Given a value, the model must retrieve multiple corresponding keys.
• multi_match(resume): Multi‑match resume retrieval. Given a university name, the model must retrieve multiple students from that university.
• multi_match_last(kv): Multi‑match KV retrieval where the last key is omitted; all other gold keys are provided in the prompt.

File Naming Conventions

• logic_kv_10: Logic‑based KV retrieval task with a context containing 10 KV pairs.
• 3_match_resume_100: Multi‑match resume retrieval with a context of 100 students, requiring the model to retrieve 3 students.
• concat_3_kv_100_cot: Multi‑step KV retrieval with a context of 100 KV pairs; the model must retrieve 3 values via 3 queries and concatenate them. The prompt follows a Chain‑of‑Thought (CoT) style.

Dataset Columns

• prompt: The complete task prompt.
• gold_keys: Gold keys for KV retrieval tasks. A single string if only one gold key; otherwise a list of strings. For resume tasks, this corresponds to student names (or a list).
• gold_values: Gold values for KV retrieval tasks. A single string if only one gold value; otherwise a list. For resume tasks, this corresponds to GPAs or university names (or a list).

Note: In logic‑based and multi‑match tasks, gold_keys actually represent the answer contained in the prompt.