Diagnostics for SNPE

[sesirota]

Hi! I am using a SNPE framework currently, and am finding some difficulty in running diagnostic checks. The diagnostic checks seem to take an extreme amount of time, and I've never had one run long enough to finish (they have run for over a week before). I am not sure if this indicates something wrong with my posterior / the way I am training my neural net, or if I am using the wrong type of diagnostic. The diagnostic checks always seem to get stuck on the sampling steps. Any insight would be very much appreciated!!

[janfb]

Just to be sure, are you running this with SNPE (multi round) or NPE (amortized). Only the latter is feasible with the current framework.

Additionally, you have to make sure that the theta and x you pass to the SBC function are sampled / simulated from the prior and the same prior you trained on.

The long sampling / NPE getting stuck likely indicates that the data x is misspecified or out of distribution.

So the overall workflow would be:

• sample theta from prior, simulate x
• train NPE with those.
• build the posterior
• generate 100 or so test samples by sampling from the prior and simulating
• pass those to run_SBc alongside the NPE posterior.

Does this help?

[sesirota]

I am running this with multi round. Do you mean that the current diagnostics are not suited for use with sequential estimators? If so, what would you recommend for ensuring the posterior estimates are correct when using sequential methods?

[janfb]

Yes, the current implementation of SBC is designed for amortized NPE (single round). The reason is that for SBC, we basically repeat inference with different simulated test observations N times (N usually on the order of 100s). For single round NPE this is cheap, just one forward pass through the network. But for multi-round NPE this would imply repeating the training rounds for each x . It's not impossible, e.g., by parallelizing it, but you would need to do it manually.

Alternatively, I would mainly rely on posterior predictive checks: sampling 1000 times from the posterior given your observed x_o, simulating with those parameters and comparing the simulations to your observed x_o.
Additionally, you could think of running single-round NPE with as many simulations from the prior as you have lying around or can possibly generate and then running SBC.