Repository navigation
Loading pretrained model error #5
Description
Hey, I'm coming again! When I do Step 3. Train SLTUnet Model, I moved required files in two folders in train.sh file and run train.sh. When the code run to loading pretrained model, I got a warning below:
INFO:tensorflow:Trying restore pretrained parameters
WARNING:tensorflow:No Existing Model detected
INFO:tensorflow:Trying restore existing parameters
WARNING:tensorflow:No Existing Model detected
How can I load pretrained model? Is pretrained model trained in Step 2? Thanks!
This is my train.sh
data=preprocessed-corpus/
feature=smkd-sign-features/
python3 run.py --mode train --parameters=\
hidden_size=256,embed_size=256,filter_size=4096,\
sep_layer=0,num_encoder_layer=6,num_decoder_layer=6,\
ctc_enable=True,ctc_alpha=0.3,ctc_repeated=True,\
src_bpe_dropout=0.2,tgt_bpe_dropout=0.2,bpe_dropout_stochastic_rate=0.6,\
initializer="uniform_unit_scaling",initializer_gain=0.5,\
dropout=0.3,label_smooth=0.1,attention_dropout=0.3,relu_dropout=0.5,residual_dropout=0.4,\
max_len=256,max_img_len=512,batch_size=80,eval_batch_size=32,\
token_size=1600,batch_or_token='token',beam_size=8,remove_bpe=True,decode_alpha=1.0,\
scope_name="transformer",buffer_size=50000,data_leak_ratio=0.1,\
img_feature_size=1024,img_aug_size=11,\
clip_grad_norm=0.0,\
num_heads=4,\
process_num=2,\
lrate=1.0,\
estop_patience=100,\
warmup_steps=4000,\
epoches=5000,\
update_cycle=16,\
gpus=[0],\
disp_freq=1,\
eval_freq=500,\
sample_freq=100,\
checkpoints=5,\
best_checkpoints=10,\
max_training_steps=30000,\
nthreads=8,\
beta1=0.9,\
beta2=0.998,\
random_seed=1234,\
src_codes="$data/ende.bpe",tgt_codes="$data/ende.bpe",\
src_vocab_file="$data/vocab.zero.drop",\
tgt_vocab_file="$data/vocab.zero.drop",\
img_train_file="$feature/train.h5",\
src_train_file="$data/train.bpe.en.shuf",\
tgt_train_file="$data/train.bpe.de.shuf",\
img_dev_file="$feature/dev.h5",\
src_dev_file="$data/dev.bpe.en",\
tgt_dev_file="$data/dev.bpe.de",\
img_test_file="$feature/test.h5",\
src_test_file="$data/test.bpe.en",\
tgt_test_file="$data/test.bpe.de",\
output_dir="train",\
test_output="",\
shared_source_target_embedding=True,\
Activity
bzhangGo commentedon Jan 27, 2024
Hey, the logging information is a little bit confusing here.
The
pretrained modelhere doesn't mean the pretrained sign embeddings, but pretrained SLT model. so it's normal and not a problem. More details are below:It tries to restore a separately pretrained SLT model, e.g. pretrained encoders or decoders, which we never used.
It tries to restore from existing working directory. If your job got corrupted, it should recover the training from the working directory, i.e. output_dir.
EyjafjalIa commentedon Jan 28, 2024
Oh! I'm sorry, loading pretrained model may not the important problem. The original error seems like h5 file.
When I run Step 2. command 4. I got dev.h5 test.h5 train.h5 and train_(0-9).h5 in path smkd/features and then I combine different training features and move dev/test/train.h5 to path smkd-sign-features/, which wrote in train.sh.
Finally, I run the train.sh by command below in root path sltunet/ and I got KeyError above. I have run Step 2. command 4 and combine train.h5 twice. Both of them have KeyError. What should I do to check the problem? I'm really confused. Thanks!
bzhangGo commentedon Jan 28, 2024
This could be checked by inspecting the source_train_file and the resulted train.h5.
Could you please show a few lines in your train file? and also read train.h5 with h5py and check its keys? there might be some mismatch.
EyjafjalIa commentedon Jan 29, 2024
After I run command below, I got dev.h5, test.h5, train.h5 and train_(0-9).h5 in sltunet/smkd/features
This is my sign_feature_cmb.py file. Should I combine train.h5 and train_(0-9).h5 in a new h5 file or only combine train_(0-9).h5? I guess this line
writer = h5py.File('train.h5', 'w')may overwrite train.h5 because I run this python script on the same path of those h5 files and finally I lose some data.bzhangGo commentedon Jan 29, 2024
could you please list some keys from your
train.h5? e.g. 5142_8 is missing based on the error, then could you please take a look what keys for 5142 are contained in your training data?EyjafjalIa commentedon Jan 30, 2024
I have solved this error. It happens when I run sign_feature_cmb.py on the same path of train.h5 and train_(0-9).h5. I show my path below.
When the script runs to
writer = h5py.File('train.h5', 'w'), it open a file train.h5 with mode write. It may clean train.h5 file if exist on the path of script and write new content.I change the line to
writer = h5py.File('train123.h5', 'w'). After the script finished, I move to right path and rename it to train.h5.My script path is:
smkd/features ├── dev.h5 ├── test.h5 └── train ├── sign_feature_cmb.py ├── train_0.h5 ├── train_1.h5 ├── train_2.h5 ├── train_3.h5 ├── train_4.h5 ├── train_5.h5 ├── train_6.h5 ├── train_7.h5 ├── train_8.h5 ├── train_9.h5 └── train.h5 1 directory, 12 filesEyjafjalIa commentedon Jan 30, 2024
When I follow the instruction below in sltunet/example, I can't get right combined train.h5 file because that after I run Step 2. extract sign features, I got directory below.
python sign_feature_cmb.py train\*h5Directory after extract: