Question about the "wrap" padding mode

[Ang-Li-95]

Dear Huilin,

Sorry to bother you and thanks a lot for this handy package!

I have some question about the "wrap" padding mode in the data loader. If I understand correctly, assuming a jagged array as below:
array = [ [1,2,3], [4,5] ]

Doing a "wrap" padding to make the array have the shape of (2,6), in which case each row has 6 elements, will give us:
array_pad = [ [1,2,3,1,2,3], [4,5,4,5,4,5] ]

In the code, I think the "wrap" padding is performed through this function.
And padding the array mentioned above using the _repeat_pad function gives us:
array_pad = [ [1,2,3,4,5,1], [4,5,4,5,1,2] ]

I am wondering whether I misunderstood the concept of "wrap" padding or this is a bug in the implementation?

Thank you in advance and I look forward to your reply!

Best regards,
Ang

[mykhailynab]

I faced the same issue, and I don't think this behavior is intended. The pf_mask parameter masked the padded inputs anyway, so this wasn't impacting execution of the models in the particle_transformer repo. However, if you still need this padding mode for any reason, I've created #23 which should fix the issue.

[hqucms]

wrap is probably not a very good name but this is actually well intended. The idea is to pad not with zeros, but with values that are similar to the original data distribution, so the normalization layers behave more stable. We found it improving the performance slightly when developing the ParT model so it was kept as a default.

[mykhailynab]

@hqucms thanks for your explanation. Turns out we didn't need to fix it 😉

I saw that the padded inputs are masked out by

x = self.embed(x).masked_fill(~mask.permute(2, 0, 1), 0)

and made the conclusion.

However, I now see how it is only done after the embedding, and the same is not applied for the v values, which would make the BatchNorms inside of Embed/PairEmbed more stable.

Thanks for your clarification! 😄