SAFIM Benchmark

Overview

SAFIM (Syntax-Aware Fill-in-the-Middle) is a benchmark for evaluating large language models (LLMs) on code fill-in-the-middle (FIM) tasks. SAFIM comprises three sub-tasks: Algorithmic Block Completion, Control-Flow Expression Completion, and API Function Call Completion. The data source is code submitted from April 2022 to January 2023, aiming to reduce data contamination affecting evaluation outcomes.

Dataset Information

• License: cc-by-4.0
• Task Category: text2text-generation
• Language: en
• Tags: code-generation, code-infilling, fill-in-the-middle
• Dataset Name: SAFIM
• Data Volume: 10K < n < 100K

Configuration Information

• config_name: block

  • Data Files:
    • split: test
    • path: block_completion.jsonl.gz

• config_name: control

  • Data Files:
    • split: test
    • path: control_completion.jsonl.gz

• config_name: api

  • Data Files:
    • split: test
    • path: api_completion.jsonl.gz

• config_name: block_v2

  • Data Files:
    • split: test
    • path: block_completion_v2.jsonl.gz