Fix RMSNorm scale zero-initialization - #481
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
self.scale was initialized with jnp.zeros instead of jnp.ones, causing RMSNorm to output all zeros on a fresh forward pass (before checkpoint loading overwrites it). LayerNorm in the same file correctly uses jnp.ones for its scale param — this looks like a copy-paste bug.
1fd9281 to
b6f1be2
Compare
CLA is now signed and verified — ready for review whenever a maintainer gets a chance. |
RMSNorm.__init__insrc/timesfm/flax/normalization.pyinitializesself.scalewithjnp.zerosinstead ofjnp.ones:Since
__call__doesnormed_inputs *= self.scale, this makesRMSNorm output all zeros on a fresh forward pass, before any
checkpoint load overwrites the parameter.
LayerNormin the same file correctly initializes its scale withjnp.ones— this looks like a copy-paste bug between the two classes.Currently masked in inference when checkpoints are always loaded
before a forward pass, but affects:
Fix:
jnp.zeros→jnp.ones, matchingLayerNorm's pattern.