Luma Dream Machine

A

AnimateDiff

Yuwei Guo|N/A

AnimateDiff is a motion module framework developed by Yuwei Guo that transforms any personalized text-to-image diffusion model into a video generator by inserting learnable temporal attention layers into the existing architecture. Released in July 2023, AnimateDiff introduced a groundbreaking approach by decoupling motion learning from visual appearance learning, allowing users to leverage the vast ecosystem of fine-tuned Stable Diffusion models and LoRA adaptations for video creation without retraining. The core innovation is a plug-and-play motion module that learns general motion patterns from video data and can be inserted into any Stable Diffusion checkpoint to animate its outputs while preserving visual style and quality. The motion module consists of temporal transformer blocks with self-attention across frames, generating temporally coherent sequences with natural object movement. AnimateDiff supports both SD 1.5 and SDXL base models with optimized motion module versions for each architecture. The framework enables generation of animated GIFs and short video loops with customizable frame counts, frame rates, and motion intensities. Users can combine AnimateDiff with ControlNet for pose-guided animation, IP-Adapter for reference-based motion, and various LoRA models for style-specific video generation. Common applications include animated artwork, social media content, game asset animation, product visualization, and creative storytelling. Available under the Apache 2.0 license, AnimateDiff is accessible on Hugging Face, Replicate, and fal.ai, with extensive community support through ComfyUI workflows and Automatic1111 extensions. The framework has become one of the most influential open-source video generation approaches, enabling creators to produce stylized animated content with unprecedented flexibility.

C

CogVideoX

Tsinghua & ZhipuAI|5B

CogVideoX is an open-source video generation model jointly developed by Tsinghua University and ZhipuAI that utilizes an expert transformer architecture to produce high-quality videos from text descriptions. Released in August 2024, CogVideoX represents a significant advancement in open-source video generation, offering capabilities that approach proprietary models while remaining freely available for research. Built on a 5 billion parameter transformer architecture that processes text and visual tokens through specialized expert layers, it enables efficient computation while maintaining high output quality. CogVideoX employs a 3D causal VAE for video encoding and decoding, capturing both spatial and temporal information in a unified latent space, resulting in videos with smooth motion transitions and consistent visual coherence. The model supports variable-length video generation and multiple resolution outputs, providing flexibility for different use cases. CogVideoX demonstrates strong performance in generating videos with accurate motion dynamics, scene transitions, and visual storytelling elements, handling both simple prompts and complex narrative scenarios. The training approach incorporates progressive resolution scaling and temporal consistency losses that maintain stable generation quality across different durations. Available under the Apache 2.0 license on Hugging Face, CogVideoX can be accessed through fal.ai and Replicate, and can be run locally with sufficient GPU resources. The model has been well-received in the research community as a strong open-source baseline for video generation, enabling academic studies and commercial applications that require transparent, modifiable video generation capabilities without proprietary API constraints.

C

CogVideoX-5B

Tsinghua & ZhipuAI|5B

CogVideoX-5B is a 5-billion parameter open-source video generation model developed jointly by Tsinghua University and ZhipuAI that produces high-quality, temporally consistent videos from text descriptions and image inputs. Built on a 3D VAE (Variational Autoencoder) combined with a Diffusion Transformer architecture, CogVideoX-5B processes spatial and temporal dimensions jointly, enabling the generation of videos with smooth motion, consistent object appearances, and coherent scene dynamics across frames. The model supports both text-to-video generation where users describe desired scenes in natural language and image-to-video generation where a static image serves as the first frame and the model animates it with appropriate motion. CogVideoX-5B can generate videos of up to 6 seconds at 480x720 resolution with 8 frames per second, producing content suitable for social media clips, concept visualization, and creative prototyping. The 3D VAE compresses video data into a compact latent space that preserves temporal coherence, while the Diffusion Transformer generates content with strong semantic understanding of motion, physics, and spatial relationships. As one of the most capable open-source video generation models available, CogVideoX-5B achieves competitive quality with proprietary alternatives while remaining freely accessible for research and development. Released under the Apache 2.0 license, the model is available on Hugging Face and integrates with the Diffusers library for straightforward deployment. Key applications include generating short-form video content, creating animated product demonstrations, producing visual concept previews for film and advertising pre-production, and prototyping motion graphics without manual animation.

G

Gemini Omni Flash

Status unknown

Google DeepMind|undisclosed

Gemini Omni Flash is Google DeepMind's groundbreaking multimodal AI model that generates physics-aware video with synchronized audio from any combination of text, images, video, and audio inputs. Announced at Google I/O 2026, it represents a paradigm shift from traditional text-to-video models by enabling conversational, iterative video editing — users can refine scenes through natural language without regenerating from scratch. The model maintains character consistency and scene memory across multiple editing rounds, preserves identity and voice throughout sequences, and understands real-world physics including gravity, collisions, and material properties. Omni Flash supports cinematic camera controls (dolly zoom, over-shoulder shots, tracking), accurate text rendering with word-by-word animation, multi-input synthesis (combining videos, images, audio, and storyboards), and style transfer across artistic mediums including anime, claymation, and watercolor. Built on Gemini's training data, it carries significantly more world knowledge than standalone video models like Veo, enabling it to visualize complex concepts from quantum computing to historical events without exhaustive prompting. Available through the Gemini app, Google Flow, and Google AI Studio, it produces clips up to 10 seconds with invisible SynthID watermarking for content authenticity.