ProcessTBench
ProcessTBench is a synthetic dataset for evaluating the planning capabilities of large language models (LLMs) within a process mining framework. Built upon TaskBench, it contains 532 base queries, each paraphrased 5–6 times, with an average of 4.08 solution plans per query. The dataset involves action sequences using 40 distinct tools and provides corresponding ground‑truth plans in Petri‑net format. Creation involved selecting the most challenging subset from TaskBench, generating plans with LLMs, and processing them using an event‑log parser and a plan‑conformance checker. ProcessTBench aims to support research on LLM plan generation in complex and dynamic environments, especially regarding multilingual and paraphrased queries.
Description
ProcessTBench Dataset Overview
Dataset Content
Generated Plans and Variants
- Description: Contains various plans to address the problem objectives in each process ID.
- Generation Method: Produced using
generate_plans_and_variants.py.
Paraphrased Queries
- Description: Queries paraphrased from the TaskBench dataset.
- Generation Method: Produced using
paraphrase_queries.py. - Source: TaskBench dataset (https://github.com/microsoft/JARVIS/tree/main/taskbench).
Process Models
- Description: Process models of the generated plans.
- Generation Method: Created with the Inductive Miner at thresholds 0, 0.1, and 0.2.
- Additional Information: Convert the reference DAG of a query to a Petri net; example generation code is in
dag_to_petri_net_results.py.
Conformance Quality
- Description: Results of conformance checks evaluating the quality of paraphrased queries.
- Generation Method: Produced using
generate_plans_conformance_quality_rephrased.pyandgenerate_plans_conformance_quality_original.py.
TaskBench Data
- Files:
taskbench_multimedia.jsontaskbench_multimedia_dag.jsontaskbench_multimedia_dag_partitioned.json(multi‑process partitioning)tool_desc_multimedia.json
Additional Files
- Tools and Models:
utils.pymy_model.py(embedding and LLM model configuration)readme.mdrequirements.txt
AI studio
Generate PPTs instantly with Nano Banana Pro.
Generate PPT NowAccess Dataset
Please login to view download links and access full dataset details.
Topics
Source
Organization: arXiv
Created: 9/14/2024
Power Your Data Analysis with Premium AI Models
Supporting GPT-5, Claude-4, DeepSeek v3, Gemini and more.
Enjoy a free trial and save 20%+ compared to official pricing.