Learning a Human-centric Motion Representation for Action Analysis
Zhanbo Huang, Xiaoming Liu, Yu Kong.
PyTorch implementation for H-MoRe. For details, see the paper: H-MoRe: Learning a Human-centric Motion Representation for Action Analysis.
The training and evaluation code requires PyTorch 2.0 as well as a number of other 3rd party packages. Note that the code has only been tested with the specified versions and also expects a Linux environment. To setup all the required dependencies for training and evaluation, please follow the instructions below:
Conda - clone this repository and create a new conda virtual environment:
conda env create -n hmore python=3.12PyTorch - install PyTorch following the official instructions, for example:
pip install torch torchvision torchaudio3rd Party Packages - install the required packages using pip:
pip install -r requirements.txtPlease download the CASIA-B dataset and reformat the data as follows:
<ROOT>/001(subject)/bg-01(condition)/072(view)/072-aligned-rgbs.pkl<ROOT>/001(subject)/bg-01(condition)/090(view)/090-aligned-rgbs.pkl- ...
Separate the training and testing according to OpenGait.
Please download the Diving48 dataset and reformat the data as follows:
<ROOT>/sFO6XlfgxNQ_00036(uuid)/img_00001.jpg<ROOT>/sFO6XlfgxNQ_00036(uuid)/img_00002.jpg- ...
Separate the training and testing according to official splits version 2.
Please download the UTD-MHAD dataset and reformat the data as follows:
<ROOT>/a1_s1_t1_color/frame.pkl<ROOT>/a1_s1_t3_color/frame.pkl- ...
Following the official split, the subject numbers in odd are used for training, while those in even are used for testing.
To quickly train the model, you can use the following command:
python script/train.py -c config/casia/full.yamlWhen using 16 NVIDIA 6000 Ada GPUs, the training will take about 1 day.
You can also try to adjust some hyperparameters in config files, details about each parameter can be found here.
You can predict world-local flow maps by using the following command:
python script/predict.py -c config/casia/full.yamlThis script will inference and save the predicted flow maps according to predict section in the config file.
Different codebases are used for evaluating different tasks. Please refer to the following links for the evaluation code:
⚠️ Currently, you have to manually adjust their codes to load our predicted flow maps (just the same as loading other optical flow estimation methods).
💡 We are trying to build a unified evaluation api for all tasks, which will be released soon.
H-MoRe code and model weights are released under the MIT License.