Add extra model paths for WAN models discovery

[Shellishack]

Is your feature request related to a problem? Please describe.
I am using WAN2.2 s2v's official workflows, and I ran in to "Value not in list" issues like

[{'type': 'value_not_in_list', 'message': 'Value not in list', 'details': "clip_name: 'umt5_xxl_fp8_e4m3fn_scaled.safetensors' not in []", 'extra_info': {'input_name': 'clip_name', 'input_config': [[], {}], 'received_value': 'umt5_xxl_fp8_e4m3fn_scaled.safetensors'}}]

According to WAN2.2 (s2v) usage manual, the following models need to be present in this structure:

• diffusion_models
  • wan2.2_s2v_14B_fp8_scaled.safetensors
  • wan2.2_s2v_14B_bf16.safetensors
• audio_encoders
  • wav2vec2_large_english_fp16.safetensors
• vae
  • wan_2.1_vae.safetensors
• text_encoders
  • umt5_xxl_fp8_e4m3fn_scaled.safetensors

Upon checking #48, I notice these paths are not present in src/extra_model_paths.yaml:

  diffusion_models: models/diffusion_models/
  audio_encoders: models/audio_encoders/
  text_encoders: models/text_encoders/

Describe the solution you'd like

Add the following to src/extra_model_paths.yaml:

  diffusion_models: models/diffusion_models/
  audio_encoders: models/audio_encoders/
  text_encoders: models/text_encoders/

Describe alternatives you've considered

Additional context

[Shellishack]

Added a PR to add the above paths. While this only fixes this issue for WAN2.2 S2V, it could potentially also help with usage of other WAN models.

[BenDes21]

Added a PR to add the above paths. While this only fixes this issue for WAN2.2 S2V, it could potentially also help with usage of other WAN models.

is Wan 2.2 compatible with this worker template ? Lets said a basic i2v workflow

[Shellishack]

Added a PR to add the above paths. While this only fixes this issue for WAN2.2 S2V, it could potentially also help with usage of other WAN models.

is Wan 2.2 compatible with this worker template ? Lets said a basic i2v workflow

Hi there,

Yes Wan 2.2 i2v is also compatible.

However, I am not sure if RunPod can handle base64 value or S3 bucket URL for WAN i2v's video output. Similarly thing goes with s2v's input, which requires an audio file as input. Without knowing how non-image assets like videos are uploaded to RunPod's storage, I am not sure if there is a way to integrate video models similar to how worker-comfyui already does for image models.

As a work around, I had to make my own custom nodes in ComfyUI (https://github.com/Shellishack/comfyui_remote_media_io). These nodes would download assets from URL as inputs, and upload outputs to Azure blob. And I had to swap out ComfyUI's default loaders and savers with my custom nodes. The down side is that RunPod's response will no long return an output value. But I could get around this by simply checking the response status is okay and looking for result artifacts in my CDN. Here is a WAN i2v ComfyUI workflow I used:

For example, I made these two dockerfiles here that support WAN2.2 i2v and WAN2.2 s2v (and many other models), as long as you copy models into docker image or put models in a network volume.

• For network volume: https://github.com/Shellishack/pulse-app-comfyui-workflow/blob/main/runpod/network-volume/dockerfile
• For custom image: https://github.com/Shellishack/pulse-app-comfyui-workflow/blob/main/runpod/custom-image/dockerfile

I tried to make a public endpoint last week. But I got stuck on handling non-image inputs/outputs again. I am happy to work on that in the future. Probably need some help from RunPod team for that.

Hope this helps!

[BenDes21]

Added a PR to add the above paths. While this only fixes this issue for WAN2.2 S2V, it could potentially also help with usage of other WAN models.

is Wan 2.2 compatible with this worker template ? Lets said a basic i2v workflow

Hi there,

Yes Wan 2.2 i2v is also compatible.

However, I am not sure if RunPod can handle base64 value or S3 bucket URL for WAN i2v's video output. Similarly thing goes with s2v's input, which requires an audio file as input. Without knowing how non-image assets like videos are uploaded to RunPod's storage, I am not sure if there is a way to integrate video models similar to how worker-comfyui already does for image models.

As a work around, I had to make my own custom nodes in ComfyUI (https://github.com/Shellishack/comfyui_remote_media_io). These nodes would download assets from URL as inputs, and upload outputs to Azure blob. And I had to swap out ComfyUI's default loaders and savers with my custom nodes. The down side is that RunPod's response will no long return an output value. But I could get around this by simply checking the response status is okay and looking for result artifacts in my CDN. Here is a WAN i2v ComfyUI workflow I used:

For example, I made these two dockerfiles here that support WAN2.2 i2v and WAN2.2 s2v (and many other models), as long as you copy models into docker image or put models in a network volume.

• For network volume: https://github.com/Shellishack/pulse-app-comfyui-workflow/blob/main/runpod/network-volume/dockerfile
• For custom image: https://github.com/Shellishack/pulse-app-comfyui-workflow/blob/main/runpod/custom-image/dockerfile

I tried to make a public endpoint last week. But I got stuck on handling non-image inputs/outputs again. I am happy to work on that in the future. Probably need some help from RunPod team for that.

Hope this helps!

Thanks a lot, will take a look, not gonna lie its look a bit complex for me haha, btw do you know how temporary fix the extra_model_paths.yaml with the path missing ? Thanks

[Shellishack]

Added a PR to add the above paths. While this only fixes this issue for WAN2.2 S2V, it could potentially also help with usage of other WAN models.

is Wan 2.2 compatible with this worker template ? Lets said a basic i2v workflow

Hi there,
Yes Wan 2.2 i2v is also compatible.
However, I am not sure if RunPod can handle base64 value or S3 bucket URL for WAN i2v's video output. Similarly thing goes with s2v's input, which requires an audio file as input. Without knowing how non-image assets like videos are uploaded to RunPod's storage, I am not sure if there is a way to integrate video models similar to how worker-comfyui already does for image models.
As a work around, I had to make my own custom nodes in ComfyUI (https://github.com/Shellishack/comfyui_remote_media_io). These nodes would download assets from URL as inputs, and upload outputs to Azure blob. And I had to swap out ComfyUI's default loaders and savers with my custom nodes. The down side is that RunPod's response will no long return an output value. But I could get around this by simply checking the response status is okay and looking for result artifacts in my CDN. Here is a WAN i2v ComfyUI workflow I used:
For example, I made these two dockerfiles here that support WAN2.2 i2v and WAN2.2 s2v (and many other models), as long as you copy models into docker image or put models in a network volume.

• For network volume: https://github.com/Shellishack/pulse-app-comfyui-workflow/blob/main/runpod/network-volume/dockerfile
• For custom image: https://github.com/Shellishack/pulse-app-comfyui-workflow/blob/main/runpod/custom-image/dockerfile

I tried to make a public endpoint last week. But I got stuck on handling non-image inputs/outputs again. I am happy to work on that in the future. Probably need some help from RunPod team for that.
Hope this helps!

Thanks a lot, will take a look, not gonna lie its look a bit complex for me haha, btw do you know how temporary fix the extra_model_paths.yaml with the path missing ? Thanks

I really wish this is a way to simplify this too!

ComfyUI uses extra_model_paths.yaml to discover model files outside of installation directory (https://github.com/comfyanonymous/ComfyUI/blob/master/extra_model_paths.yaml.example). RunPod's network volume mounts a separate volume at /workspace, but I remember they mentioned something like models/ would resolve to /workspace/models/. Hence the extra paths like models/diffusion_models/ would make ComfyUI discover models in network volume.

In hindsight, I am actually not sure if custom image requires this modification? It seems like custom docker image puts models directly into /comfyui/models

[BenDes21]

Added a PR to add the above paths. While this only fixes this issue for WAN2.2 S2V, it could potentially also help with usage of other WAN models.

is Wan 2.2 compatible with this worker template ? Lets said a basic i2v workflow

Hi there,
Yes Wan 2.2 i2v is also compatible.
However, I am not sure if RunPod can handle base64 value or S3 bucket URL for WAN i2v's video output. Similarly thing goes with s2v's input, which requires an audio file as input. Without knowing how non-image assets like videos are uploaded to RunPod's storage, I am not sure if there is a way to integrate video models similar to how worker-comfyui already does for image models.
As a work around, I had to make my own custom nodes in ComfyUI (https://github.com/Shellishack/comfyui_remote_media_io). These nodes would download assets from URL as inputs, and upload outputs to Azure blob. And I had to swap out ComfyUI's default loaders and savers with my custom nodes. The down side is that RunPod's response will no long return an output value. But I could get around this by simply checking the response status is okay and looking for result artifacts in my CDN. Here is a WAN i2v ComfyUI workflow I used:
For example, I made these two dockerfiles here that support WAN2.2 i2v and WAN2.2 s2v (and many other models), as long as you copy models into docker image or put models in a network volume.

• For network volume: https://github.com/Shellishack/pulse-app-comfyui-workflow/blob/main/runpod/network-volume/dockerfile
• For custom image: https://github.com/Shellishack/pulse-app-comfyui-workflow/blob/main/runpod/custom-image/dockerfile

I tried to make a public endpoint last week. But I got stuck on handling non-image inputs/outputs again. I am happy to work on that in the future. Probably need some help from RunPod team for that.
Hope this helps!

Thanks a lot, will take a look, not gonna lie its look a bit complex for me haha, btw do you know how temporary fix the extra_model_paths.yaml with the path missing ? Thanks

I really wish this is a way to simplify this too!

ComfyUI uses extra_model_paths.yaml to discover model files outside of installation directory (https://github.com/comfyanonymous/ComfyUI/blob/master/extra_model_paths.yaml.example). RunPod's network volume mounts a separate volume at /workspace, but I remember they mentioned something like models/ would resolve to /workspace/models/. Hence the extra paths like models/diffusion_models/ would make ComfyUI discover models in network volume.

In hindsight, I am actually not sure if custom image requires this modification? It seems like custom docker image puts models directly into /comfyui/models

extra_model_paths.yaml.example must be fixed for add

diffusion_models: models/diffusion_models/
audio_encoders: models/audio_encoders/
text_encoders: models/text_encoders/

as wan looking for theses models/folders. Even after set all files into /workspace/models/diffusion_models/ and /workspace/models/text_encoders/ , wan 2.2 throw and error who said that we cannot find the files

I did this for a quick temporary fix into my dockerfile :

RUN cat > /comfyui/extra_model_paths.yaml <<EOF
runpod_worker_comfy:
base_path: /runpod-volume
checkpoints: models/checkpoints/
clip: models/clip/
clip_vision: models/clip_vision/
configs: models/configs/
controlnet: models/controlnet/
embeddings: models/embeddings/
loras: models/loras/
upscale_models: models/upscale_models/
vae: models/vae/
unet: models/unet/
diffusion_models: models/diffusion_models/
text_encoders: models/text_encoders/
audio_encoders: models/audio_encoders/
EOF

seems to work ( at least it's find my models, now I need to find a way to get the output video " easily " lol )

[BenDes21]

Added a PR to add the above paths. While this only fixes this issue for WAN2.2 S2V, it could potentially also help with usage of other WAN models.

is Wan 2.2 compatible with this worker template ? Lets said a basic i2v workflow

Hi there,

Yes Wan 2.2 i2v is also compatible.

However, I am not sure if RunPod can handle base64 value or S3 bucket URL for WAN i2v's video output. Similarly thing goes with s2v's input, which requires an audio file as input. Without knowing how non-image assets like videos are uploaded to RunPod's storage, I am not sure if there is a way to integrate video models similar to how worker-comfyui already does for image models.

As a work around, I had to make my own custom nodes in ComfyUI (https://github.com/Shellishack/comfyui_remote_media_io). These nodes would download assets from URL as inputs, and upload outputs to Azure blob. And I had to swap out ComfyUI's default loaders and savers with my custom nodes. The down side is that RunPod's response will no long return an output value. But I could get around this by simply checking the response status is okay and looking for result artifacts in my CDN. Here is a WAN i2v ComfyUI workflow I used:

For example, I made these two dockerfiles here that support WAN2.2 i2v and WAN2.2 s2v (and many other models), as long as you copy models into docker image or put models in a network volume.

• For network volume: https://github.com/Shellishack/pulse-app-comfyui-workflow/blob/main/runpod/network-volume/dockerfile
• For custom image: https://github.com/Shellishack/pulse-app-comfyui-workflow/blob/main/runpod/custom-image/dockerfile

I tried to make a public endpoint last week. But I got stuck on handling non-image inputs/outputs again. I am happy to work on that in the future. Probably need some help from RunPod team for that.

Hope this helps!

Ah I forget to ask but is your custom nodes compatible with others cdn like bunny cdn or only Azur ? Also I need to set theses 3 env variable for my runpod serverless endpoint if I understand :

S3_ENDPOINT_URL
S3_ACCESS_KEY_ID
S3_SECRET_ACCESS_KEY

[Shellishack]

Added a PR to add the above paths. While this only fixes this issue for WAN2.2 S2V, it could potentially also help with usage of other WAN models.

is Wan 2.2 compatible with this worker template ? Lets said a basic i2v workflow

Hi there,
Yes Wan 2.2 i2v is also compatible.
However, I am not sure if RunPod can handle base64 value or S3 bucket URL for WAN i2v's video output. Similarly thing goes with s2v's input, which requires an audio file as input. Without knowing how non-image assets like videos are uploaded to RunPod's storage, I am not sure if there is a way to integrate video models similar to how worker-comfyui already does for image models.
As a work around, I had to make my own custom nodes in ComfyUI (https://github.com/Shellishack/comfyui_remote_media_io). These nodes would download assets from URL as inputs, and upload outputs to Azure blob. And I had to swap out ComfyUI's default loaders and savers with my custom nodes. The down side is that RunPod's response will no long return an output value. But I could get around this by simply checking the response status is okay and looking for result artifacts in my CDN. Here is a WAN i2v ComfyUI workflow I used:
For example, I made these two dockerfiles here that support WAN2.2 i2v and WAN2.2 s2v (and many other models), as long as you copy models into docker image or put models in a network volume.

• For network volume: https://github.com/Shellishack/pulse-app-comfyui-workflow/blob/main/runpod/network-volume/dockerfile
• For custom image: https://github.com/Shellishack/pulse-app-comfyui-workflow/blob/main/runpod/custom-image/dockerfile

I tried to make a public endpoint last week. But I got stuck on handling non-image inputs/outputs again. I am happy to work on that in the future. Probably need some help from RunPod team for that.
Hope this helps!

Ah I forget to ask but is your custom nodes compatible with others cdn like bunny cdn or only Azur ? Also I need to set theses 3 env variable for my runpod serverless endpoint if I understand :

S3_ENDPOINT_URL S3_ACCESS_KEY_ID S3_SECRET_ACCESS_KEY

Custom nodes:
They can download from any URL. But uploading only supports Azure right now.

Env variables:
Well, if the custom nodes support AWS somehow right now, then these env variables will be needed.

I will see if I have time to implement AWS in the custom nodes some time later this week. Or feel free to add AWS yourself in the custom nodes repo. Shellishack/comfyui_remote_media_io#1

[BenDes21]

Added a PR to add the above paths. While this only fixes this issue for WAN2.2 S2V, it could potentially also help with usage of other WAN models.

is Wan 2.2 compatible with this worker template ? Lets said a basic i2v workflow

Hi there,
Yes Wan 2.2 i2v is also compatible.
However, I am not sure if RunPod can handle base64 value or S3 bucket URL for WAN i2v's video output. Similarly thing goes with s2v's input, which requires an audio file as input. Without knowing how non-image assets like videos are uploaded to RunPod's storage, I am not sure if there is a way to integrate video models similar to how worker-comfyui already does for image models.
As a work around, I had to make my own custom nodes in ComfyUI (https://github.com/Shellishack/comfyui_remote_media_io). These nodes would download assets from URL as inputs, and upload outputs to Azure blob. And I had to swap out ComfyUI's default loaders and savers with my custom nodes. The down side is that RunPod's response will no long return an output value. But I could get around this by simply checking the response status is okay and looking for result artifacts in my CDN. Here is a WAN i2v ComfyUI workflow I used:
For example, I made these two dockerfiles here that support WAN2.2 i2v and WAN2.2 s2v (and many other models), as long as you copy models into docker image or put models in a network volume.

• For network volume: https://github.com/Shellishack/pulse-app-comfyui-workflow/blob/main/runpod/network-volume/dockerfile
• For custom image: https://github.com/Shellishack/pulse-app-comfyui-workflow/blob/main/runpod/custom-image/dockerfile

I tried to make a public endpoint last week. But I got stuck on handling non-image inputs/outputs again. I am happy to work on that in the future. Probably need some help from RunPod team for that.
Hope this helps!

Ah I forget to ask but is your custom nodes compatible with others cdn like bunny cdn or only Azur ? Also I need to set theses 3 env variable for my runpod serverless endpoint if I understand :
S3_ENDPOINT_URL S3_ACCESS_KEY_ID S3_SECRET_ACCESS_KEY

Custom nodes: They can download from any URL. But uploading only supports Azure right now.

Env variables: Well, if the custom nodes support AWS somehow right now, then these env variables will be needed.

I will see if I have time to implement AWS in the custom nodes some time later this week. Or feel free to add AWS yourself in the custom nodes repo. Shellishack/comfyui_remote_media_io#1

Hi there, ok thanks a lot ! Unfortunately Im not skilled enough for modify the custom nodes :( Will see with chat gpt for Cloudflare / bunny cdn but Im scared to spend hours for just make something unusable :(

[brunoj1]

Added a PR to add the above paths. While this only fixes this issue for WAN2.2 S2V, it could potentially also help with usage of other WAN models.

is Wan 2.2 compatible with this worker template ? Lets said a basic i2v workflow

Hi there,

Yes Wan 2.2 i2v is also compatible.

However, I am not sure if RunPod can handle base64 value or S3 bucket URL for WAN i2v's video output. Similarly thing goes with s2v's input, which requires an audio file as input. Without knowing how non-image assets like videos are uploaded to RunPod's storage, I am not sure if there is a way to integrate video models similar to how worker-comfyui already does for image models.

As a work around, I had to make my own custom nodes in ComfyUI (https://github.com/Shellishack/comfyui_remote_media_io). These nodes would download assets from URL as inputs, and upload outputs to Azure blob. And I had to swap out ComfyUI's default loaders and savers with my custom nodes. The down side is that RunPod's response will no long return an output value. But I could get around this by simply checking the response status is okay and looking for result artifacts in my CDN. Here is a WAN i2v ComfyUI workflow I used:

For example, I made these two dockerfiles here that support WAN2.2 i2v and WAN2.2 s2v (and many other models), as long as you copy models into docker image or put models in a network volume.

• For network volume: https://github.com/Shellishack/pulse-app-comfyui-workflow/blob/main/runpod/network-volume/dockerfile
• For custom image: https://github.com/Shellishack/pulse-app-comfyui-workflow/blob/main/runpod/custom-image/dockerfile

I tried to make a public endpoint last week. But I got stuck on handling non-image inputs/outputs again. I am happy to work on that in the future. Probably need some help from RunPod team for that.

Hope this helps!

good workaround and I also use azure, I am starting to implement my own runpod worker for WAN video generation first to last frame, I would love to have some input.
Do you have the worker template ready?

[Shellishack]

Added a PR to add the above paths. While this only fixes this issue for WAN2.2 S2V, it could potentially also help with usage of other WAN models.

is Wan 2.2 compatible with this worker template ? Lets said a basic i2v workflow

Hi there,
Yes Wan 2.2 i2v is also compatible.
However, I am not sure if RunPod can handle base64 value or S3 bucket URL for WAN i2v's video output. Similarly thing goes with s2v's input, which requires an audio file as input. Without knowing how non-image assets like videos are uploaded to RunPod's storage, I am not sure if there is a way to integrate video models similar to how worker-comfyui already does for image models.
As a work around, I had to make my own custom nodes in ComfyUI (https://github.com/Shellishack/comfyui_remote_media_io). These nodes would download assets from URL as inputs, and upload outputs to Azure blob. And I had to swap out ComfyUI's default loaders and savers with my custom nodes. The down side is that RunPod's response will no long return an output value. But I could get around this by simply checking the response status is okay and looking for result artifacts in my CDN. Here is a WAN i2v ComfyUI workflow I used:
For example, I made these two dockerfiles here that support WAN2.2 i2v and WAN2.2 s2v (and many other models), as long as you copy models into docker image or put models in a network volume.

• For network volume: https://github.com/Shellishack/pulse-app-comfyui-workflow/blob/main/runpod/network-volume/dockerfile
• For custom image: https://github.com/Shellishack/pulse-app-comfyui-workflow/blob/main/runpod/custom-image/dockerfile

I tried to make a public endpoint last week. But I got stuck on handling non-image inputs/outputs again. I am happy to work on that in the future. Probably need some help from RunPod team for that.
Hope this helps!

good workaround and I also use azure, I am starting to implement my own runpod worker for WAN video generation first to last frame, I would love to have some input. Do you have the worker template ready?

Hey there,

Yes I have both i2v and s2v workers up and running with no issues. My worker templates used the custom image method as it reduces a lot of cold start time with flashboot. Happy to help

[brunoj1]

Added a PR to add the above paths. While this only fixes this issue for WAN2.2 S2V, it could potentially also help with usage of other WAN models.

is Wan 2.2 compatible with this worker template ? Lets said a basic i2v workflow

Hi there,
Yes Wan 2.2 i2v is also compatible.
However, I am not sure if RunPod can handle base64 value or S3 bucket URL for WAN i2v's video output. Similarly thing goes with s2v's input, which requires an audio file as input. Without knowing how non-image assets like videos are uploaded to RunPod's storage, I am not sure if there is a way to integrate video models similar to how worker-comfyui already does for image models.
As a work around, I had to make my own custom nodes in ComfyUI (https://github.com/Shellishack/comfyui_remote_media_io). These nodes would download assets from URL as inputs, and upload outputs to Azure blob. And I had to swap out ComfyUI's default loaders and savers with my custom nodes. The down side is that RunPod's response will no long return an output value. But I could get around this by simply checking the response status is okay and looking for result artifacts in my CDN. Here is a WAN i2v ComfyUI workflow I used:
For example, I made these two dockerfiles here that support WAN2.2 i2v and WAN2.2 s2v (and many other models), as long as you copy models into docker image or put models in a network volume.

• For network volume: https://github.com/Shellishack/pulse-app-comfyui-workflow/blob/main/runpod/network-volume/dockerfile
• For custom image: https://github.com/Shellishack/pulse-app-comfyui-workflow/blob/main/runpod/custom-image/dockerfile

I tried to make a public endpoint last week. But I got stuck on handling non-image inputs/outputs again. I am happy to work on that in the future. Probably need some help from RunPod team for that.
Hope this helps!

good workaround and I also use azure, I am starting to implement my own runpod worker for WAN video generation first to last frame, I would love to have some input. Do you have the worker template ready?

Hey there,

Yes I have both i2v and s2v workers up and running with no issues. My worker templates used the custom image method as it reduces a lot of cold start time with flashboot. Happy to help

https://github.com/Shellishack/pulse-app-comfyui-workflow
this project right?
From what I understood, you are using shared network to store models, custom image with the custom node for saving the result video on azure blob and base worker-template is this repository.
I will try to replicate for my workflow which happens to be mostly first to last frame wan 2.2 first to last frame with 2 image inputs

[Shellishack]

Added a PR to add the above paths. While this only fixes this issue for WAN2.2 S2V, it could potentially also help with usage of other WAN models.

is Wan 2.2 compatible with this worker template ? Lets said a basic i2v workflow

Hi there,
Yes Wan 2.2 i2v is also compatible.
However, I am not sure if RunPod can handle base64 value or S3 bucket URL for WAN i2v's video output. Similarly thing goes with s2v's input, which requires an audio file as input. Without knowing how non-image assets like videos are uploaded to RunPod's storage, I am not sure if there is a way to integrate video models similar to how worker-comfyui already does for image models.
As a work around, I had to make my own custom nodes in ComfyUI (https://github.com/Shellishack/comfyui_remote_media_io). These nodes would download assets from URL as inputs, and upload outputs to Azure blob. And I had to swap out ComfyUI's default loaders and savers with my custom nodes. The down side is that RunPod's response will no long return an output value. But I could get around this by simply checking the response status is okay and looking for result artifacts in my CDN. Here is a WAN i2v ComfyUI workflow I used:
For example, I made these two dockerfiles here that support WAN2.2 i2v and WAN2.2 s2v (and many other models), as long as you copy models into docker image or put models in a network volume.

• For network volume: https://github.com/Shellishack/pulse-app-comfyui-workflow/blob/main/runpod/network-volume/dockerfile
• For custom image: https://github.com/Shellishack/pulse-app-comfyui-workflow/blob/main/runpod/custom-image/dockerfile

I tried to make a public endpoint last week. But I got stuck on handling non-image inputs/outputs again. I am happy to work on that in the future. Probably need some help from RunPod team for that.
Hope this helps!

good workaround and I also use azure, I am starting to implement my own runpod worker for WAN video generation first to last frame, I would love to have some input. Do you have the worker template ready?

Hey there,
Yes I have both i2v and s2v workers up and running with no issues. My worker templates used the custom image method as it reduces a lot of cold start time with flashboot. Happy to help

https://github.com/Shellishack/pulse-app-comfyui-workflow this project right? From what I understood, you are using shared network to store models, custom image with the custom node for saving the result video on azure blob and base worker-template is this repository. I will try to replicate for my workflow which happens to be mostly first to last frame wan 2.2 first to last frame with 2 image inputs

You got it right.

For your reference, I was using the following workflow (API Json) for s2v for testing (as an example). First frame, last frame should also be doable via a similar approach.

{
  "147": {
    "inputs": {
      "clip_name": "umt5_xxl_fp8_e4m3fn_scaled.safetensors",
      "type": "wan",
      "device": "default"
    },
    "class_type": "CLIPLoader",
    "_meta": {
      "title": "Load CLIP"
    }
  },
  "148": {
    "inputs": {
      "samples": [
        "156",
        0
      ],
      "vae": [
        "150",
        0
      ]
    },
    "class_type": "VAEDecode",
    "_meta": {
      "title": "VAE Decode"
    }
  },
  "149": {
    "inputs": {
      "text": "色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走",
      "clip": [
        "147",
        0
      ]
    },
    "class_type": "CLIPTextEncode",
    "_meta": {
      "title": "CLIP Text Encode (Negative Prompt)"
    }
  },
  "150": {
    "inputs": {
      "vae_name": "wan_2.1_vae.safetensors"
    },
    "class_type": "VAELoader",
    "_meta": {
      "title": "Load VAE"
    }
  },
  "151": {
    "inputs": {
      "audio_encoder_name": "wav2vec2_large_english_fp16.safetensors"
    },
    "class_type": "AudioEncoderLoader",
    "_meta": {
      "title": "AudioEncoderLoader"
    }
  },
  "152": {
    "inputs": {
      "seed": 20,
      "steps": [
        "177",
        0
      ],
      "cfg": [
        "178",
        0
      ],
      "sampler_name": "uni_pc",
      "scheduler": "simple",
      "denoise": 1,
      "model": [
        "154",
        0
      ],
      "positive": [
        "153",
        0
      ],
      "negative": [
        "153",
        1
      ],
      "latent_image": [
        "153",
        2
      ]
    },
    "class_type": "KSampler",
    "_meta": {
      "title": "KSampler"
    }
  },
  "153": {
    "inputs": {
      "width": 1024,
      "height": 704,
      "length": [
        "166",
        0
      ],
      "batch_size": 1,
      "positive": [
        "159",
        0
      ],
      "negative": [
        "149",
        0
      ],
      "vae": [
        "150",
        0
      ],
      "audio_encoder_output": [
        "173",
        0
      ],
      "ref_image": [
        "186",
        0
      ]
    },
    "class_type": "WanSoundImageToVideo",
    "_meta": {
      "title": "WanSoundImageToVideo"
    }
  },
  "154": {
    "inputs": {
      "shift": 8,
      "model": [
        "161",
        0
      ]
    },
    "class_type": "ModelSamplingSD3",
    "_meta": {
      "title": "ModelSamplingSD3"
    }
  },
  "155": {
    "inputs": {
      "batch_index": [
        "169",
        0
      ],
      "length": 4096,
      "image": [
        "148",
        0
      ]
    },
    "class_type": "ImageFromBatch",
    "_meta": {
      "title": "ImageFromBatch"
    }
  },
  "156": {
    "inputs": {
      "dim": "t",
      "samples1": [
        "157",
        0
      ],
      "samples2": [
        "198:78",
        0
      ]
    },
    "class_type": "LatentConcat",
    "_meta": {
      "title": "LatentConcat"
    }
  },
  "157": {
    "inputs": {
      "dim": "t",
      "index": 0,
      "amount": 1,
      "samples": [
        "198:78",
        0
      ]
    },
    "class_type": "LatentCut",
    "_meta": {
      "title": "LatentCut"
    }
  },
  "159": {
    "inputs": {
      "text": "Dr. Doom played by Robert Downey Jr., defeating the MCU heroes. Captivating cinematic style.",
      "clip": [
        "147",
        0
      ]
    },
    "class_type": "CLIPTextEncode",
    "_meta": {
      "title": "CLIP Text Encode (Positive Prompt)"
    }
  },
  "161": {
    "inputs": {
      "unet_name": "wan2.2_s2v_14B_fp8_scaled.safetensors",
      "weight_dtype": "default"
    },
    "class_type": "UNETLoader",
    "_meta": {
      "title": "Load Diffusion Model"
    }
  },
  "166": {
    "inputs": {
      "value": 77
    },
    "class_type": "PrimitiveInt",
    "_meta": {
      "title": "Chunk Length"
    }
  },
  "167": {
    "inputs": {
      "fps": 16,
      "images": [
        "155",
        0
      ],
      "audio": [
        "188",
        0
      ]
    },
    "class_type": "CreateVideo",
    "_meta": {
      "title": "Create Video"
    }
  },
  "169": {
    "inputs": {
      "value": 2
    },
    "class_type": "PrimitiveInt",
    "_meta": {
      "title": "Batch sizes"
    }
  },
  "173": {
    "inputs": {
      "audio_encoder": [
        "151",
        0
      ],
      "audio": [
        "188",
        0
      ]
    },
    "class_type": "AudioEncoderEncode",
    "_meta": {
      "title": "AudioEncoderEncode"
    }
  },
  "177": {
    "inputs": {
      "value": 20
    },
    "class_type": "PrimitiveInt",
    "_meta": {
      "title": "Steps"
    }
  },
  "178": {
    "inputs": {
      "value": 6
    },
    "class_type": "PrimitiveFloat",
    "_meta": {
      "title": "CFG"
    }
  },
  "186": {
    "inputs": {
      "image_url": "https://cdn.pulse-editor.com/artifact/doom.png"
    },
    "class_type": "LoadRemoteImage",
    "_meta": {
      "title": "Load Remote Image"
    }
  },
  "188": {
    "inputs": {
      "audio_url": "https://cdn.pulse-editor.com/artifact/doom.mp3"
    },
    "class_type": "LoadRemoteAudio",
    "_meta": {
      "title": "Load Remote Audio"
    }
  },
  "189": {
    "inputs": {
      "filename": "doom.mp4",
      "video": [
        "167",
        0
      ]
    },
    "class_type": "SaveVideoToRemote",
    "_meta": {
      "title": "Save Video To Remote"
    }
  },
  "172:76": {
    "inputs": {
      "length": [
        "166",
        0
      ],
      "positive": [
        "159",
        0
      ],
      "negative": [
        "149",
        0
      ],
      "vae": [
        "150",
        0
      ],
      "video_latent": [
        "152",
        0
      ],
      "audio_encoder_output": [
        "173",
        0
      ],
      "ref_image": [
        "186",
        0
      ]
    },
    "class_type": "WanSoundImageToVideoExtend",
    "_meta": {
      "title": "WanSoundImageToVideoExtend"
    }
  },
  "172:78": {
    "inputs": {
      "dim": "t",
      "samples1": [
        "152",
        0
      ],
      "samples2": [
        "172:77",
        0
      ]
    },
    "class_type": "LatentConcat",
    "_meta": {
      "title": "LatentConcat"
    }
  },
  "172:77": {
    "inputs": {
      "seed": 1,
      "steps": [
        "177",
        0
      ],
      "cfg": [
        "178",
        0
      ],
      "sampler_name": "uni_pc",
      "scheduler": "simple",
      "denoise": 1,
      "model": [
        "154",
        0
      ],
      "positive": [
        "172:76",
        0
      ],
      "negative": [
        "172:76",
        1
      ],
      "latent_image": [
        "172:76",
        2
      ]
    },
    "class_type": "KSampler",
    "_meta": {
      "title": "KSampler"
    }
  },
  "198:76": {
    "inputs": {
      "length": [
        "166",
        0
      ],
      "positive": [
        "159",
        0
      ],
      "negative": [
        "149",
        0
      ],
      "vae": [
        "150",
        0
      ],
      "video_latent": [
        "172:78",
        0
      ],
      "audio_encoder_output": [
        "173",
        0
      ],
      "ref_image": [
        "186",
        0
      ]
    },
    "class_type": "WanSoundImageToVideoExtend",
    "_meta": {
      "title": "WanSoundImageToVideoExtend"
    }
  },
  "198:78": {
    "inputs": {
      "dim": "t",
      "samples1": [
        "172:78",
        0
      ],
      "samples2": [
        "198:77",
        0
      ]
    },
    "class_type": "LatentConcat",
    "_meta": {
      "title": "LatentConcat"
    }
  },
  "198:77": {
    "inputs": {
      "seed": 1,
      "steps": [
        "177",
        0
      ],
      "cfg": [
        "178",
        0
      ],
      "sampler_name": "uni_pc",
      "scheduler": "simple",
      "denoise": 1,
      "model": [
        "154",
        0
      ],
      "positive": [
        "198:76",
        0
      ],
      "negative": [
        "198:76",
        1
      ],
      "latent_image": [
        "198:76",
        2
      ]
    },
    "class_type": "KSampler",
    "_meta": {
      "title": "KSampler"
    }
  }
}

[BenDes21]

Added a PR to add the above paths. While this only fixes this issue for WAN2.2 S2V, it could potentially also help with usage of other WAN models.

is Wan 2.2 compatible with this worker template ? Lets said a basic i2v workflow

Hi there,
Yes Wan 2.2 i2v is also compatible.
However, I am not sure if RunPod can handle base64 value or S3 bucket URL for WAN i2v's video output. Similarly thing goes with s2v's input, which requires an audio file as input. Without knowing how non-image assets like videos are uploaded to RunPod's storage, I am not sure if there is a way to integrate video models similar to how worker-comfyui already does for image models.
As a work around, I had to make my own custom nodes in ComfyUI (https://github.com/Shellishack/comfyui_remote_media_io). These nodes would download assets from URL as inputs, and upload outputs to Azure blob. And I had to swap out ComfyUI's default loaders and savers with my custom nodes. The down side is that RunPod's response will no long return an output value. But I could get around this by simply checking the response status is okay and looking for result artifacts in my CDN. Here is a WAN i2v ComfyUI workflow I used:
For example, I made these two dockerfiles here that support WAN2.2 i2v and WAN2.2 s2v (and many other models), as long as you copy models into docker image or put models in a network volume.

• For network volume: https://github.com/Shellishack/pulse-app-comfyui-workflow/blob/main/runpod/network-volume/dockerfile
• For custom image: https://github.com/Shellishack/pulse-app-comfyui-workflow/blob/main/runpod/custom-image/dockerfile

I tried to make a public endpoint last week. But I got stuck on handling non-image inputs/outputs again. I am happy to work on that in the future. Probably need some help from RunPod team for that.
Hope this helps!

good workaround and I also use azure, I am starting to implement my own runpod worker for WAN video generation first to last frame, I would love to have some input. Do you have the worker template ready?

Hey there,
Yes I have both i2v and s2v workers up and running with no issues. My worker templates used the custom image method as it reduces a lot of cold start time with flashboot. Happy to help

https://github.com/Shellishack/pulse-app-comfyui-workflow this project right? From what I understood, you are using shared network to store models, custom image with the custom node for saving the result video on azure blob and base worker-template is this repository. I will try to replicate for my workflow which happens to be mostly first to last frame wan 2.2 first to last frame with 2 image inputs

You got it right.

For your reference, I was using the following workflow (API Json) for s2v for testing (as an example). First frame, last frame should also be doable via a similar approach.

{
  "147": {
    "inputs": {
      "clip_name": "umt5_xxl_fp8_e4m3fn_scaled.safetensors",
      "type": "wan",
      "device": "default"
    },
    "class_type": "CLIPLoader",
    "_meta": {
      "title": "Load CLIP"
    }
  },
  "148": {
    "inputs": {
      "samples": [
        "156",
        0
      ],
      "vae": [
        "150",
        0
      ]
    },
    "class_type": "VAEDecode",
    "_meta": {
      "title": "VAE Decode"
    }
  },
  "149": {
    "inputs": {
      "text": "色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走",
      "clip": [
        "147",
        0
      ]
    },
    "class_type": "CLIPTextEncode",
    "_meta": {
      "title": "CLIP Text Encode (Negative Prompt)"
    }
  },
  "150": {
    "inputs": {
      "vae_name": "wan_2.1_vae.safetensors"
    },
    "class_type": "VAELoader",
    "_meta": {
      "title": "Load VAE"
    }
  },
  "151": {
    "inputs": {
      "audio_encoder_name": "wav2vec2_large_english_fp16.safetensors"
    },
    "class_type": "AudioEncoderLoader",
    "_meta": {
      "title": "AudioEncoderLoader"
    }
  },
  "152": {
    "inputs": {
      "seed": 20,
      "steps": [
        "177",
        0
      ],
      "cfg": [
        "178",
        0
      ],
      "sampler_name": "uni_pc",
      "scheduler": "simple",
      "denoise": 1,
      "model": [
        "154",
        0
      ],
      "positive": [
        "153",
        0
      ],
      "negative": [
        "153",
        1
      ],
      "latent_image": [
        "153",
        2
      ]
    },
    "class_type": "KSampler",
    "_meta": {
      "title": "KSampler"
    }
  },
  "153": {
    "inputs": {
      "width": 1024,
      "height": 704,
      "length": [
        "166",
        0
      ],
      "batch_size": 1,
      "positive": [
        "159",
        0
      ],
      "negative": [
        "149",
        0
      ],
      "vae": [
        "150",
        0
      ],
      "audio_encoder_output": [
        "173",
        0
      ],
      "ref_image": [
        "186",
        0
      ]
    },
    "class_type": "WanSoundImageToVideo",
    "_meta": {
      "title": "WanSoundImageToVideo"
    }
  },
  "154": {
    "inputs": {
      "shift": 8,
      "model": [
        "161",
        0
      ]
    },
    "class_type": "ModelSamplingSD3",
    "_meta": {
      "title": "ModelSamplingSD3"
    }
  },
  "155": {
    "inputs": {
      "batch_index": [
        "169",
        0
      ],
      "length": 4096,
      "image": [
        "148",
        0
      ]
    },
    "class_type": "ImageFromBatch",
    "_meta": {
      "title": "ImageFromBatch"
    }
  },
  "156": {
    "inputs": {
      "dim": "t",
      "samples1": [
        "157",
        0
      ],
      "samples2": [
        "198:78",
        0
      ]
    },
    "class_type": "LatentConcat",
    "_meta": {
      "title": "LatentConcat"
    }
  },
  "157": {
    "inputs": {
      "dim": "t",
      "index": 0,
      "amount": 1,
      "samples": [
        "198:78",
        0
      ]
    },
    "class_type": "LatentCut",
    "_meta": {
      "title": "LatentCut"
    }
  },
  "159": {
    "inputs": {
      "text": "Dr. Doom played by Robert Downey Jr., defeating the MCU heroes. Captivating cinematic style.",
      "clip": [
        "147",
        0
      ]
    },
    "class_type": "CLIPTextEncode",
    "_meta": {
      "title": "CLIP Text Encode (Positive Prompt)"
    }
  },
  "161": {
    "inputs": {
      "unet_name": "wan2.2_s2v_14B_fp8_scaled.safetensors",
      "weight_dtype": "default"
    },
    "class_type": "UNETLoader",
    "_meta": {
      "title": "Load Diffusion Model"
    }
  },
  "166": {
    "inputs": {
      "value": 77
    },
    "class_type": "PrimitiveInt",
    "_meta": {
      "title": "Chunk Length"
    }
  },
  "167": {
    "inputs": {
      "fps": 16,
      "images": [
        "155",
        0
      ],
      "audio": [
        "188",
        0
      ]
    },
    "class_type": "CreateVideo",
    "_meta": {
      "title": "Create Video"
    }
  },
  "169": {
    "inputs": {
      "value": 2
    },
    "class_type": "PrimitiveInt",
    "_meta": {
      "title": "Batch sizes"
    }
  },
  "173": {
    "inputs": {
      "audio_encoder": [
        "151",
        0
      ],
      "audio": [
        "188",
        0
      ]
    },
    "class_type": "AudioEncoderEncode",
    "_meta": {
      "title": "AudioEncoderEncode"
    }
  },
  "177": {
    "inputs": {
      "value": 20
    },
    "class_type": "PrimitiveInt",
    "_meta": {
      "title": "Steps"
    }
  },
  "178": {
    "inputs": {
      "value": 6
    },
    "class_type": "PrimitiveFloat",
    "_meta": {
      "title": "CFG"
    }
  },
  "186": {
    "inputs": {
      "image_url": "https://cdn.pulse-editor.com/artifact/doom.png"
    },
    "class_type": "LoadRemoteImage",
    "_meta": {
      "title": "Load Remote Image"
    }
  },
  "188": {
    "inputs": {
      "audio_url": "https://cdn.pulse-editor.com/artifact/doom.mp3"
    },
    "class_type": "LoadRemoteAudio",
    "_meta": {
      "title": "Load Remote Audio"
    }
  },
  "189": {
    "inputs": {
      "filename": "doom.mp4",
      "video": [
        "167",
        0
      ]
    },
    "class_type": "SaveVideoToRemote",
    "_meta": {
      "title": "Save Video To Remote"
    }
  },
  "172:76": {
    "inputs": {
      "length": [
        "166",
        0
      ],
      "positive": [
        "159",
        0
      ],
      "negative": [
        "149",
        0
      ],
      "vae": [
        "150",
        0
      ],
      "video_latent": [
        "152",
        0
      ],
      "audio_encoder_output": [
        "173",
        0
      ],
      "ref_image": [
        "186",
        0
      ]
    },
    "class_type": "WanSoundImageToVideoExtend",
    "_meta": {
      "title": "WanSoundImageToVideoExtend"
    }
  },
  "172:78": {
    "inputs": {
      "dim": "t",
      "samples1": [
        "152",
        0
      ],
      "samples2": [
        "172:77",
        0
      ]
    },
    "class_type": "LatentConcat",
    "_meta": {
      "title": "LatentConcat"
    }
  },
  "172:77": {
    "inputs": {
      "seed": 1,
      "steps": [
        "177",
        0
      ],
      "cfg": [
        "178",
        0
      ],
      "sampler_name": "uni_pc",
      "scheduler": "simple",
      "denoise": 1,
      "model": [
        "154",
        0
      ],
      "positive": [
        "172:76",
        0
      ],
      "negative": [
        "172:76",
        1
      ],
      "latent_image": [
        "172:76",
        2
      ]
    },
    "class_type": "KSampler",
    "_meta": {
      "title": "KSampler"
    }
  },
  "198:76": {
    "inputs": {
      "length": [
        "166",
        0
      ],
      "positive": [
        "159",
        0
      ],
      "negative": [
        "149",
        0
      ],
      "vae": [
        "150",
        0
      ],
      "video_latent": [
        "172:78",
        0
      ],
      "audio_encoder_output": [
        "173",
        0
      ],
      "ref_image": [
        "186",
        0
      ]
    },
    "class_type": "WanSoundImageToVideoExtend",
    "_meta": {
      "title": "WanSoundImageToVideoExtend"
    }
  },
  "198:78": {
    "inputs": {
      "dim": "t",
      "samples1": [
        "172:78",
        0
      ],
      "samples2": [
        "198:77",
        0
      ]
    },
    "class_type": "LatentConcat",
    "_meta": {
      "title": "LatentConcat"
    }
  },
  "198:77": {
    "inputs": {
      "seed": 1,
      "steps": [
        "177",
        0
      ],
      "cfg": [
        "178",
        0
      ],
      "sampler_name": "uni_pc",
      "scheduler": "simple",
      "denoise": 1,
      "model": [
        "154",
        0
      ],
      "positive": [
        "198:76",
        0
      ],
      "negative": [
        "198:76",
        1
      ],
      "latent_image": [
        "198:76",
        2
      ]
    },
    "class_type": "KSampler",
    "_meta": {
      "title": "KSampler"
    }
  }
}

Hi there !

Quick question, Im using same workflow / same gpu on a pod and generate a video take around 3 minutes

Seems the TE spend 5 minutes for be loaded which is super long :

mbeyhgy5ls93gt[info]\rT5Encoder: 0%| | 0/24 [00:00<?, ?it/s]\rT5Encoder: 4%|▍ | 1/24 [00:14<05:28, 14.27s/it]\rT5Encoder: 8%|▊ | 2/24 [00:27<05:04, 13.86s/it]\rT5Encoder: 12%|█▎ | 3/24 [00:40<04:40, 13.34s/it]\rT5Encoder: 17%|█▋ | 4/24 [00:53<04:27, 13.35s/it]\rT5Encoder: 21%|██ | 5/24 [01:07<04:13, 13.35s/it]\rT5Encoder: 25%|██▌ | 6/24 [01:20<03:59, 13.30s/it]\rT5Encoder: 29%|██▉ | 7/24 [01:33<03:42, 13.09s/it]\rT5Encoder: 33%|███▎ | 8/24 [01:44<03:21, 12.62s/it]\rT5Encoder: 38%|███▊ | 9/24 [01:55<03:02, 12.18s/it]\rT5Encoder: 42%|████▏ | 10/24 [02:07<02:48, 12.05s/it]\rT5Encoder: 46%|████▌ | 11/24 [02:20<02:40, 12.34s/it]\rT5Encoder: 50%|█████ | 12/24 [02:33<02:31, 12.62s/it]\rT5Encoder: 54%|█████▍ | 13/24 [02:47<02:20, 12.81s/it]\rT5Encoder: 58%|█████▊ | 14/24 [03:01<02:11, 13.10s/it]\rT5Encoder: 62%|██████▎ | 15/24 [03:13<01:56, 12.94s/it]\rT5Encoder: 67%|██████▋ | 16/24 [03:26<01:42, 12.83s/it]\rT5Encoder: 71%|███████ | 17/24 [03:38<01:28, 12.70s/it]\rT5Encoder: 75%|███████▌ | 18/24 [03:51<01:15, 12.66s/it]\rT5Encoder: 79%|███████▉ | 19/24 [04:04<01:03, 12.75s/it]\rT5Encoder: 83%|████████▎ | 20/24 [04:16<00:51, 12.77s/it]\rT5Encoder: 88%|████████▊ | 21/24 [04:30<00:38, 12.88s/it]\rT5Encoder: 92%|█████████▏| 22/24 [04:42<00:25, 12.71s/it]\rT5Encoder: 96%|█████████▌| 23/24 [04:54<00:12, 12.68s/it]\rT5Encoder: 100%|██████████| 24/24 [05:07<00:00, 12.67s/it]\rT5Encoder: 100%|██████████| 24/24 [05:07<00:00, 12.82s/it]\n
mbeyhgy5ls93gt[info]worker-comfyui - Websocket receive timed out. Still waiting...\n