OpenHands-SMILE2

This repository extends the OpenHands framework by AI4Bharat with support for the SMILE2 dataset (Swiss German Sign Language) and was developed as part of a bachelor's thesis at the University of Zurich (UZH). The original OpenHands documentation continues to apply.

SMILE2 preparation

In the directory SMILE2_preparation, the following files and scripts can be found:

Annotation file

• 20241120_annotation_L1.json

This file contains the original SMILE2 annotations for the recordings of the (M1)L1 signers.

Each recording session has an ID like this: SMILE2_H024_L1_M6_recording, with H0XX representing the signers. Only two tasks are annotated:

• the Sentence Transformation Task (STT), which starts with e5_ in the annotations
• the Sentence Repetition Task (SRT), which starts with e6_ in the annotations

Some recordings IDs have _pt1 or _pt2 appended (e.g. SMILE2_H016_L1_M6_recording_pt1), this is the case when one recording session was split into two, i.e. due to the signer requesting a break

Some recording IDs occur several times with different suffixes each:

• SMILE2_H024_L1_M6_recording
• SMILE2_H024_L1_M6_recording_per
• SMILE2_H024_L1_M6_recording_tik
• SMILE2_H025_L1_M6_recording
• SMILE2_H025_L1_M6_recording_smm
• SMILE2_H025_L1_M6_recording_tik

These signify different annotation runs by different annotators.

Creating Minimal, Clean and Large Subsets

• create_minimal_splits.py
  • creates the Minimal subset: signs that are present for all ten signers
• create_clean_and_large_splits.py
  • creates the
    • Clean subset: signs that are present for at least five signers and that are present in train, val, test at least once
    • Large subset: signs that are present for at least one signer and that are present in train, val, test at least once

Normalization

• all_glosses_and_normalizations_in_all_datasets.csv
  • contains the normalizations for all datasets used for the multilingual SL-GCN
• normalized_glosses_smile2.csv
  • contains the normalizations for SMILE2, designed to create a common label space with all_glosses_and_normalizations_in_all_datasets.csv
• create_normalized_split_files.py
  • creates the normalized pose to class mapping files

Extract Poses from Videos

• cut_videos.def, cut_videos.sh and cut_videos_into_single_sign_videos.py
  • cuts the full recordings of continuous singing from SMILE2 into 1-sign-videos
• mediapipe_extract_patched.py and mediapipe_extract_patched.sh
  • the OpenHands pose extraction pipeline adapted to the cluster environment

Statistics

• get_annotation_statistics.py -> 20241120_annotation_L1_annotation_statistics.json
• get_annotation_examples.py -> annotation_examples.json
• compare_datasets.py
• get_dataset_statistics.py

Various

• split_into_train_val_test.py -> creates training, validation and test split
• create_filename_gloss_mapping.py -> creates the mapping between the pose files and the corresponding glosses
• check_smile2.py -> used to check the implementation of the SMILE2 dataset in OpenHands

OpenHands experiments

The directory openhands_experiments contains:

• the original OpenHands framework
  • https://openhands.ai4bharat.org/en/latest/
• the multilingual SL-GCN
  • https://github.com/ZurichNLP/openhands-itm

The experiment_scripts folder contains the scripts required to train the models, the training log as well as the test results.

The configs used for training can be found in openhands-smile2/openhands_experiments/examples/configs/smile2

The training, validation and test sets can be found in openhands-smile2/openhands_experiments/openhands/datasets/assets/smile2_metadata

PLEASE NOTE: You will need to adapt the paths in the config files to your local environment.

Cluster environment setup

These files were used to set up and test the environment on the cluster.

• openhands.def
• cluster_env.yml
• check_env.py

SMILE2 Future Work

The folder SMILE2_future_work contains a version of the SMILE2 dataset with a new annotation file, designed to fix this issues with the annotations where some signs were split like this example:

                  {
                    "feature": "BH-Glosse",
                    "value": "FLASCHE_1A'lok",
                    "time_start": "00:14:24:040000",
                    "time_end": "00:14:24:330000"
                },
                {
                    "feature": "BH-Glosse",
                    "value": "FLASCHE_1A'lok",
                    "time_start": "00:14:24:330000",
                    "time_end": "00:14:24:460000"
                },

In total, 108 merges of such "split signs" were performed in the new annotation file. The new annotation file is called annotation_L1_fixed.json.