VBench Dataset Overview

Dataset Introduction

VBench is a comprehensive benchmark suite for video generative models. It provides a thorough and hierarchical set of evaluation dimensions, decomposing “video generation quality” into multiple clearly defined metrics to enable fine‑grained and objective assessment. Each dimension and each content category has a carefully designed prompt suite as a test case, and videos generated by a set of video generation models are sampled for evaluation.

Dataset Content

• Evaluation Dimensions: includes subject_consistency, background_consistency, temporal_flickering, motion_smoothness, dynamic_degree, aesthetic_quality, imaging_quality, object_class, multiple_objects, human_action, color, spatial_relationship, scene, temporal_style, appearance_style, overall_consistency and other 16 dimensions.
• Prompt Suites: test cases designed for each dimension and content category.
• Generated Videos: videos sampled from a set of video generation models.
• Evaluation Method Suites: specific evaluation methods or pipelines designed for each dimension to enable automatic objective assessment.

Dataset Download

• Video Data: all videos used for VBench evaluation can be downloaded from Google Drive.

Dataset Updates

• VBench++: released in November 2024, supporting a broader range of video generation tasks, including text‑to‑video and image‑to‑video, and evaluating model trustworthiness.
• VBench-Long Leaderboard: released in September 2024, containing 10 long‑video generation models.
• VBench Leaderboard: updated in August 2024, containing 28 T2V models and 12 I2V models.

Dataset Usage

• Installation: install vbench via pip and, if needed, install detectron2.
• Evaluation: supports custom video evaluation as well as evaluation with the standard prompt suites.

Citation

If you use this dataset, please cite the following papers:

bibtex @InProceedings{huang2023vbench, title={{VBench}: Comprehensive Benchmark Suite for Video Generative Models}, author={Huang, Ziqi and He, Yinan and Yu, Jiashuo and Zhang, Fan and Si, Chenyang and Jiang, Yuming and Zhang, Yuanhan and Wu, Tianxing and Jin, Qingyang and Chanpaisit, Nattapol and Wang, Yaohui and Chen, Xinyuan and Wang, Limin and Lin, Dahua and Qiao, Yu and Liu, Ziwei}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, year={2024} }

@article{huang2024vbench++, title={VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models}, author={Huang, Ziqi and He, Yinan and Yu, Jiashuo and Zhang, Fan and Si, Chenyang and Jiang, Yuming and Zhang, Yuanhan and Wu, Tianxing and Jin, Qingyang and Chanpaisit, Nattapol and Wang, Yaohui and Chen, Xinyuan and Wang, Limin and Lin, Dahua and Qiao, Yu and Liu, Ziwei}, journal={arXiv preprint arXiv:2411.13503}, year={2024} }