BEHAVE Dataset Overview

Dataset Description

BEHAVE is a dataset for full‑body human‑object interaction in natural environments. The dataset provides multi‑view RGB‑D frames along with corresponding 3D SMPL and object fittings, and annotated contacts between them.

Dataset Content

Dataset Structure

After extraction the dataset contains three main sub‑folders: calibs, objects, sequences, and a split.json file.

1. calibs: Stores Kinect camera intrinsic and extrinsic parameters for calibration at different positions.
2. objects: Contains 3D scans of 20 objects.
3. sequences: Provides multi‑view RGB‑D images together with SMPL and object registration information.
4. split.json: Supplies the official training and test split based on sequence names. The training set includes 231 sequences, the test set 90 sequences.

Detailed Structure

calibs

• Content: Intrinsic and extrinsic parameters of the Kinect camera.

• Structure:

  DATASET_PATH |--calibs |----Date[xx] |------background |------config |---intrinsics

objects

• Content: 3D scan data of objects.

• Structure:

  DATASET_PATH |--objects |----object_name |------object_name.jpg |------object_name.obj |------object_name.obj.mtl |------object_name_tex.jpg |------object_name_fxxx.ply

sequences

• Content: Multi‑view RGB‑D images with SMPL and object registration information.

• Structure:

  DATASET_PATH |--sequences |----sequence_name |------info.json |------t*.000 |--------k[0-3].color.jpg |--------k[0-3].depth.png |--------k[0-3].person_mask.jpg |--------k[0-3].obj_rend_mask.jpg |--------k[0-3].color.json |--------k[0-3].mocap.[json|ply] |--------person |----------person.ply |----------fit02 |--------object_name |----------object_name.ply |----------fit01

Usage Examples

Generating Contact Labels

Use the compute_contacts.py script to generate contact labels.

Visualizing Ground‑Truth Data

Use the behave_demo.py script to visualize SMPL and object registration.

Parsing Object Pose Parameters

Use the tools/parse_obj_pose.py script to parse object registration parameters.

Parsing SMPL Pose Parameters

Use the tools/smpl_params2mesh.py script to convert SMPL parameters to meshes.

Converting Raw Video to Images

Use the tools/video2images.py script to extract frames from videos.

Converting RGB‑D Images to Point Clouds

Use the tools/rgbd2pclouds.py script to generate point clouds from RGB‑D images.

License

Please read the LICENSE file carefully.

Citation

If you use this dataset, please cite:

@inproceedings{bhatnagar22behave,
  title={Behave: Dataset and method for tracking human object interactions},
  author={Bhatnagar, Bharat Lal and Xie, Xianghui and Petrov, Ilya A and Sminchisescu, Cristian and Theobalt, Christian and Pons-Moll, Gerard},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={15935--15946},
  year={2022}
}