Skip to content

Fix RMSNorm scale zero-initialization - #481

Open
meemoprasad wants to merge 1 commit into
google-research:masterfrom
meemoprasad:fix/rmsnorm-zero-init
Open

Fix RMSNorm scale zero-initialization#481
meemoprasad wants to merge 1 commit into
google-research:masterfrom
meemoprasad:fix/rmsnorm-zero-init

Conversation

@meemoprasad

Copy link
Copy Markdown

RMSNorm.__init__ in src/timesfm/flax/normalization.py initializes
self.scale with jnp.zeros instead of jnp.ones:

self.scale = nnx.Param(jnp.zeros(shape=(num_features,)))

Since __call__ does normed_inputs *= self.scale, this makes
RMSNorm output all zeros on a fresh forward pass, before any
checkpoint load overwrites the parameter.

LayerNorm in the same file correctly initializes its scale with
jnp.ones — this looks like a copy-paste bug between the two classes.

Currently masked in inference when checkpoints are always loaded
before a forward pass, but affects:

  • standalone instantiation of RMSNorm (e.g. in tests)
  • any future from-scratch training / fine-tuning of the flax backend

Fix: jnp.zerosjnp.ones, matching LayerNorm's pattern.

@google-cla

google-cla Bot commented Sep 2, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

  self.scale was initialized with jnp.zeros instead of jnp.ones,
  causing RMSNorm to output all zeros on a fresh forward pass
  (before checkpoint loading overwrites it). LayerNorm in the
  same file correctly uses jnp.ones for its scale param —
  this looks like a copy-paste bug.
@meemoprasad
meemoprasad force-pushed the fix/rmsnorm-zero-init branch from 1fd9281 to b6f1be2 Compare September 2, 2026 17:35
@meemoprasad

Copy link
Copy Markdown
Author

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

CLA is now signed and verified — ready for review whenever a maintainer gets a chance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant