Is this optimization done to this fork? (load the model weights directly in half precision)

[mhnoni]

Hi, there is an optimization to save more GPU memory, since we usually only use half-precision, then no need to load the full precision:

Half precision weights
To save more GPU memory, you can load the model weights directly in half precision. This involves loading the float16 version of the weights, which was saved to a branch named fp16, and telling PyTorch to use the float16 type when loading them:

pipe = StableDiffusionPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4",
    revision="fp16",
    torch_dtype=torch.float16,
    use_auth_token=True
)

https://github.com/huggingface/diffusers/blob/v0.3.0/docs/source/optimization/fp16.mdx

[basujindal]

Yes, this fork uses fp16 calculations. It also has an option of using full precision in case the GPU doesn't support half precision.

[FerMG]

The answer is actually no. The weights are not loaded from a half-precision model file. The full precision is loaded into RAM and then halved after being loaded. It is sent to the VRAM already halved though.

Unless it is possible to use this optimized fp16 file from the diffusers with this repo. I didn't test this option.

Using this optimized file makes room for lowering the RAM peak usage and faster loading of the model from disk.

[mhnoni]

The answer is actually no. The weights are not loaded from a half-precision model file. The full precision is loaded into RAM and then halved after being loaded. It is sent to the VRAM already halved though.

Unless it is possible to use this optimized fp16 file from the diffusers with this repo. I didn't test this option.

Using this optimized file makes room for lowering the RAM peak usage and faster loading of the model from disk.

I see, thanks for the explanation, the issue is with VRam most of the time, so I think it's not a high priority to shrink the file to have only f16.