JUHE API Marketplace
Back to datasets
Dataset assetOpen Source CommunityMachine LearningScientific Workflow

massw

MASSW is a comprehensive text dataset summarizing multiple aspects of scientific workflows. It contains over 152,000 peer‑reviewed publications from 17 leading computer‑science conferences spanning the past 50 years. The dataset defines five core aspects of a scientific workflow: context, key idea, method, outcome, and projected impact, and systematically extracts and structures these aspects from each publication using large language models (LLMs). MASSW is large‑scale and high‑accuracy, verified through extensive checks and comparisons with human annotations and alternative methods. It supports various novel and benchmarkable machine‑learning tasks such as idea generation and outcome prediction, providing a benchmark for evaluating LLM agents in scientific research.

Source
huggingface
Created
Oct 22, 2024
Updated
Oct 22, 2024
Signals
208 views
Availability
Linked source ready
Overview

Dataset description and usage context

MASSW Dataset Overview

Dataset Information

Configuration massw_data

  • Features:
    • id: string
    • context: string
    • key_idea: string
    • method: string
    • outcome: string
    • projected_impact: string
  • Splits:
    • train:
      • num_bytes: 153,085,133
      • num_examples: 154,275
  • Download Size: 86,202,576
  • Dataset Size: 153,085,133

Configuration massw_metadata

  • Features:
    • id: string
    • venue: string
    • title: string
    • year: integer
    • partition: string
    • abstract: string
  • Splits:
    • train:
      • num_bytes: 178,427,074
      • num_examples: 191,055
  • Download Size: 97,735,018
  • Dataset Size: 178,427,074

Dataset Characteristics

  • Structured Scientific Workflow: Includes five core aspects – context (background), key idea (main intellectual contribution), method (research approach), outcome (objective results), projected impact (anticipated influence).
  • Large‑Scale: Over 152,000 peer‑reviewed papers from 17 top computer‑science conferences spanning more than 50 years.
  • High Accuracy: Validated through comprehensive checks and comparison with human annotations and alternative extraction methods.
  • Rich Benchmark Tasks: Supports multiple machine‑learning tasks such as idea generation and outcome prediction.

Core Aspect Definitions

AspectDefinitionExample
ContextThe surrounding literature or real‑world situation, usually a problem, research question, or unresolved gap.Making language models bigger does not inherently make them better at following a user's intent, as large models can generate outputs that are untruthful, toxic, or not helpful.
Key IdeaThe primary intellectual contribution of the paper, often a novel concept or solution compared with the context.The authors propose InstructGPT, a method to align language models with user intent by fine‑tuning GPT‑3 using a combination of supervised learning with labeler demonstrations and reinforcement learning from human feedback.
MethodThe concrete research approach used to validate the key idea, which may include experimental setup, theoretical framework, or other validation techniques.The authors evaluate the performance of InstructGPT by humans on a given prompt distribution and compare it with a much larger model GPT‑3.
OutcomeAn objective statement of the research output, such as experimental results or other measurable findings.InstructGPT, even with 100× fewer parameters, is preferred over GPT‑3 in human evaluations. It shows improvements in truthfulness and reductions in toxic outputs with minimal performance regressions on public NLP datasets.
Projected ImpactThe authors' anticipated influence of the research on the field, and any identified avenues for further improvement or extension.Fine‑tuning with human feedback is a promising direction for aligning language models with human intent.

Coverage

  • Includes 17 leading computer‑science conferences, such as:
    • Artificial Intelligence: AAAI, IJCAI
    • Computer Vision: CVPR, ECCV, ICCV
    • Machine Learning: ICLR, ICML, NeurIPS, KDD
    • Natural Language Processing: ACL, EMNLP, NAACL
    • Information Retrieval: SIGIR, WWW
    • Databases: SIGMOD, VLDB
    • Interdisciplinary: CHI

Citation

@article{zhang2024massw,
  title={MASSW: A New Dataset and Benchmark Tasks for AI‑Assisted Scientific Workflows},
  author={Zhang, Xingjian and Xie, Yutong and Huang, Jin and Ma, Jinge and Pan, Zhaoying and Liu, Qijia and Xiong, Ziyang and Ergen, Tolga and Shim, Dongsub and Lee, Honglak and others},
  journal={arXiv preprint arXiv:2406.06357},
  year={2024}
}
Need downstream help?

Pair the dataset with AI analysis and content workflows.

Once the source passes your review, move straight into summarization, transformation, report drafting, or presentation generation with the JuheAI toolchain.

Explore AI studio