gonglinyuan/safim
SAFIM (Syntax-Aware Fill-in-the-Middle) is a benchmark for evaluating large language models (LLMs) on code fill-in-the-middle (FIM) tasks. SAFIM comprises three sub-tasks: algorithmic block completion, control-flow expression completion, and API function call completion. The dataset is sourced from code submitted between April 2022 and January 2023 to minimize data contamination affecting evaluation results.
Dataset description and usage context
SAFIM Benchmark
Overview
SAFIM (Syntax-Aware Fill-in-the-Middle) is a benchmark for evaluating large language models (LLMs) on code fill-in-the-middle (FIM) tasks. SAFIM comprises three sub-tasks: Algorithmic Block Completion, Control-Flow Expression Completion, and API Function Call Completion. The data source is code submitted from April 2022 to January 2023, aiming to reduce data contamination affecting evaluation outcomes.
Dataset Information
- License: cc-by-4.0
- Task Category: text2text-generation
- Language: en
- Tags: code-generation, code-infilling, fill-in-the-middle
- Dataset Name: SAFIM
- Data Volume: 10K < n < 100K
Configuration Information
-
config_name: block
- Data Files:
- split: test
- path: block_completion.jsonl.gz
- Data Files:
-
config_name: control
- Data Files:
- split: test
- path: control_completion.jsonl.gz
- Data Files:
-
config_name: api
- Data Files:
- split: test
- path: api_completion.jsonl.gz
- Data Files:
-
config_name: block_v2
- Data Files:
- split: test
- path: block_completion_v2.jsonl.gz
- Data Files:
Pair the dataset with AI analysis and content workflows.
Once the source passes your review, move straight into summarization, transformation, report drafting, or presentation generation with the JuheAI toolchain.