Repository navigation
video support #12
Description
(rewriting sloppy request)
I was wondering if video support can be added?
At first I came up with lucidrain's video-diffusion-pytorch
https://github.com/lucidrains/video-diffusion-pytorch
But, after some research it seems like zeroscope might be the right model to use
https://huggingface.co/cerspense/zeroscope_v2_576w
Reacted by lin72h
Activity
leejet commentedon Aug 20, 2023
This model appears to be significantly different from stable-diffusion, no plans to support it currently. If there's time in the future, I will consider providing support for it.
ehartford commentedon Aug 20, 2023
I didn't necessarily mean this specific model, more "video" in general.
I think zeroscope would probably be the right place to start.
Sorry for being sloppy.
https://huggingface.co/cerspense/zeroscope_v2_576w
leejet commentedon Aug 21, 2023
It looks like this needs some work, and there are no plans to support it currently. Maybe in the future?
Green-Sky commentedon Nov 21, 2023
stable video diffusion (SVD) models from stability where released!
https://stability.ai/news/stable-video-diffusion-open-ai-video-model
https://huggingface.co/stabilityai/stable-video-diffusion-img2vid / https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt
leejet commentedon Nov 22, 2023
The SVD demo looks quite good. I'll make time in the next few days to study it, starting by running the official code to see its performance.
Amin456789 commentedon Nov 27, 2023
patiently waiting for SVD to release and being quantized!
FSSRepo commentedon Nov 28, 2023
@leejet
It seems to have almost the same architecture as SD 2.1 but includes some temporal consistency blocks called "time_stack." We'll need to see how they work and whether new functions need to be added to ggml. The conversion program works with this model; however, please note that we'll need to implement the vision version of CLIP to generate embeddings from images.
leejet commentedon Nov 28, 2023
I'm currently reviewing the SVD implementation code in comfyui. Perhaps I can learn how to conveniently implement SVD within sd.cpp from this.
FSSRepo commentedon Nov 28, 2023
Amazing! Good luck!!, Unfortunately, my time is limited as I am a student. Otherwise, I would be more than happy to help.
Amin456789 commentedon Nov 28, 2023
Bless u guys! SVD in cpp will be a dream! Good luck to all of u!
Amin456789 commentedon Dec 21, 2023
@leejet any update and progress on svd and inpainting? really excited to try them out in cpp!
leejet commentedon Dec 21, 2023
I've got a basic understanding of the SVD model architecture. Once I merge the #104 and #117, I'll attempt to implement SVD.
Amin456789 commentedon Dec 21, 2023
niceee! so excited, thanks
Jonathhhan commentedon Dec 29, 2023
Hotshot-XL looks interesting, too and works with SDXL models: https://huggingface.co/hotshotco/Hotshot-XL
Amin456789 commentedon Jan 1, 2024
@leejet it will be great if you support fp16 of SVD when it is done:
https://huggingface.co/becausecurious/stable-video-diffusion-img2vid-fp16/tree/main
they are smaller and probably more ram friendly
engineer1109 commentedon Apr 15, 2024
Need as well.
Amin456789 commentedon Jul 15, 2024
@leejet any update on svd please?
mirix commentedon Oct 24, 2024
I don't know if this is even remotely related to the SD architecture, but it would be could to support the new kid on the block:
https://huggingface.co/genmo/mochi-1-preview
https://huggingface.co/Kijai/Mochi_preview_comfy/tree/main
patrickjonesdotca commentedon Dec 9, 2024
Any updates on SVD?
Zctoylm0927 commentedon Dec 10, 2024
Any updates on SVD?
bombless commentedon Dec 18, 2024
There are more img2vid and txt2vid models coming
https://github.com/THUDM/CogVideo
https://huggingface.co/IamCreateAI/Ruyi-Mini-7B
https://github.com/Tencent/HunyuanVideo
delldu commentedon Jan 17, 2025
Good luck for you !!! In my opinion, applying ggml on video(AI) may be nightmare, because it dose not support tensor more than 4d, so conv3d, batchnorm3d etc will make you crazy !!!
stduhpf commentedon Jan 17, 2025
I can confirm
leejet commentedon Sep 14, 2025
Support for the Wan video model has been added.