PowerPaint
PowerPaint is a versatile open-source inpainting model developed by researchers at Tsinghua University and HKUST under the Tencent ARC umbrella, introducing the innovative concept of learnable task prompts that enable multiple inpainting functions within a single unified model.
Key Highlights
Learnable Task Prompts
An innovative mechanism that learns specialized prompt embeddings for different inpainting tasks, managing multiple tasks with a single model
Unified Multi-Task Model
Combines object removal, text-guided inpainting, shape-guided insertion and outpainting tasks in a single model
Benchmark-Leading Performance
Superior performance demonstrating the ability to match or exceed specialized single-task models across multiple inpainting benchmarks
Stable Diffusion Based
Built on Stable Diffusion, compatible with the existing SD ecosystem and accessible with open-source model weights
About
PowerPaint is a versatile AI inpainting model developed by researchers at Tsinghua University and HKUST that introduces the concept of learnable task prompts, enabling multiple inpainting functions within a single unified model. PowerPaint successfully performs diverse inpainting tasks including object removal, text-guided content generation, shape-guided inpainting, and image outpainting, all under one model architecture, eliminating the need for separate specialized models for each task. This multi-task approach establishes a new paradigm in the image editing domain.
The model's technical innovation centers on learnable task vectors (learnable task prompts) that enable behavioral specialization within shared weights. Each inpainting task — object removal, content addition, shape-based editing, and outpainting — has specially optimized task vectors that enable the model to exhibit different behaviors using the same underlying weights. This approach eliminates the traditional paradigm requiring separate models for each task and provides significant resource efficiency. Built upon a Stable Diffusion backbone, the architecture is enriched with a ControlNet-like control mechanism for precise task guidance. Task vectors are automatically learned during training and activated based on the user's selected task during inference, providing a seamless multi-task experience.
PowerPaint supports four distinct and complementary inpainting modes. In text-guided object addition mode, users describe the desired content through text prompts and the model generates contextually appropriate content in the masked region that blends naturally with surroundings. Object removal mode cleanly fills the masked area with surrounding texture-consistent content while preserving image integrity. Shape-guided inpainting mode preserves the mask shape while generating content within it for controlled editing. Outpainting mode creates natural extensions beyond the image boundaries for canvas expansion. The ability to switch freely between these four modes makes PowerPaint an exceptionally versatile and practical tool for diverse editing scenarios.
Application scenarios span a broad range of professional and creative workflows, serving different professional groups. Photographers can use the same tool for both removing unwanted objects and adding creative content elements without switching applications. Graphic designers can perform rapid visual generation for mockups and concept visuals during the ideation phase. In the e-commerce sector, product image background cleanup and alternative background generation can be accomplished in a single streamlined workflow. Digital artists can utilize the outpainting mode for canvas extension and content variations to expand their creative compositions. Social media content creators can leverage outpainting to adapt visuals for different platform dimensions and aspect ratio requirements.
In the academic landscape, PowerPaint is recognized as a successful demonstration of the multi-task inpainting approach and is cited as a reference in image editing research. In benchmark evaluations, it achieves competitive and sometimes superior results when compared against separately optimized task-specific models across standard datasets. Its object removal quality rivals specialized models such as LaMa and MAT in head-to-head comparisons. For text-guided content generation, it delivers performance comparable to Stable Diffusion Inpainting while offering the additional task modes that single-purpose models cannot provide, giving it a distinct versatility advantage.
The model is PyTorch-based and publicly available through Hugging Face for open access. A Gradio-based demo interface and Diffusers library integration facilitate easy experimentation and production deployment across different environments. GPU requirements are similar to standard Stable Diffusion models, with 8GB or more VRAM recommended for optimal performance. PowerPaint is regarded as a reference model in the multi-task inpainting domain and is recognized as an important work shaping the future direction of image editing technologies. Its unified model approach continues to inspire future research in multi-task visual processing and generation systems.
Use Cases
Smart Object Removal
Seamlessly removing objects from images with natural-looking background using context-aware removal prompt
Creative Content Insertion
Adding new objects and elements to images guided by text description and mask shape
Image Extension
Extending images in any direction consistently using the outpainting task prompt
Research and Development
Using as a baseline model and comparison point in multi-task inpainting research
Pros & Cons
Pros
- Multi-task inpainting — object removal, addition, replacement, and outpainting
- Optimized results for each task with task-adaptive prompt encoding
- Precise control with text-guided inpainting
- Open-source research project
Cons
- Not offered as a commercial product
- High GPU requirements
- In research stage — no stable release
- Insufficient documentation
Technical Details
Parameters
N/A
Architecture
Stable Diffusion based with task-specific learnable prompt tokens
Training Data
Custom curated dataset with task-specific annotations for different inpainting modes
License
Apache 2.0
Features
- Learnable Task Prompt (LTP) Mechanism
- Context-Aware Object Removal
- Text-Guided Inpainting
- Shape-Guided Object Insertion
- Image Outpainting
- Stable Diffusion Architecture Base
Benchmark Results
| Metric | Value | Compared To | Source |
|---|---|---|---|
| FID Score (Object Removal) | 8.73 | SD Inpainting: 12.6 | PowerPaint Paper (ECCV 2024) |
| CLIP Score (Text-guided) | 27.4 | SD Inpainting: 25.8 | PowerPaint Paper (ECCV 2024) |
| Desteklenen Görevler | Removal, Fill, Shape-guided, Outpainting | — | PowerPaint GitHub |
| Inference Süresi (512x512) | ~4s (50 steps, A100) | — | PowerPaint GitHub |
Available Platforms
Frequently Asked Questions
Related Models
Adobe Generative Fill
Adobe Generative Fill is a generative AI feature integrated directly into Adobe Photoshop, powered by Adobe's proprietary Firefly image generation model. Introduced in 2023, it enables users to add, modify, or remove content in images using natural language text prompts within the familiar Photoshop interface. The feature works by selecting a region with any Photoshop selection tool, typing a descriptive prompt in the contextual task bar, and receiving three AI-generated variations within seconds. Generated content is placed on a separate layer, preserving Photoshop's non-destructive editing workflow that professionals rely on. A key differentiator is Firefly's training data approach, which uses exclusively licensed Adobe Stock imagery, openly licensed content, and public domain materials, providing commercial safety and IP indemnification that competing solutions cannot match. Generative Fill automatically maintains coherence with surrounding color, lighting, perspective, and texture for seamless blending. The companion Generative Expand feature enables extending images beyond their original canvas boundaries. Professional applications span advertising campaign iteration, photography post-production, real estate staging, product photography background replacement, fashion color modification, and editorial visual preparation. The feature is accessible through Photoshop's Creative Cloud subscription with a monthly generative credits system, and also available through Adobe Express and the web-based Firefly application. Content Credentials metadata indicates when AI was used, supporting transparency standards. Adobe Generative Fill represents the most commercially safe and professionally integrated approach to AI-powered image editing available today.
DALL-E Inpainting
DALL-E Inpainting is OpenAI's proprietary image editing capability that allows users to modify specific regions of existing images through natural language prompts, available through both the DALL-E web interface and the OpenAI API. Building on the DALL-E image generation architecture, the inpainting feature enables users to select rectangular or custom-shaped regions of an image and describe what should appear in the masked area, with the AI generating contextually appropriate content that blends with the surrounding image. The system understands complex spatial relationships, lighting conditions, and artistic styles to produce edits that maintain visual coherence with the original image. Key capabilities include adding new objects to scenes, replacing backgrounds, modifying clothing or accessories on people, changing weather conditions or time of day in landscapes, and removing unwanted elements. The API provides programmatic access for building automated editing pipelines and integrating inpainting into custom applications, with options for controlling output resolution and the number of generated variations. Unlike open-source alternatives, DALL-E Inpainting operates entirely in the cloud with no local GPU requirements, making it accessible to users without specialized hardware. The model benefits from OpenAI's continuous improvements and safety filters that prevent generation of harmful content. Commercial usage is permitted under OpenAI's terms of service, with generated images belonging to the user. While it requires a paid API subscription or credits-based usage, its ease of integration, consistent quality, and the backing of OpenAI's infrastructure make it a reliable choice for developers and businesses requiring scalable AI-powered image editing capabilities.
FLUX Fill
FLUX Fill is the specialized inpainting and outpainting model within the FLUX model family developed by Black Forest Labs, designed for professional-grade region editing, content filling, and image extension. Built on the 12-billion parameter Diffusion Transformer architecture that powers all FLUX models, FLUX Fill takes an input image along with a binary mask indicating the region to be modified and generates seamlessly blended content that matches the surrounding context in style, lighting, perspective, and detail level. The model excels at both inpainting tasks where masked areas within an image are filled with contextually appropriate content and outpainting tasks where image boundaries are extended to create larger compositions. FLUX Fill leverages the superior prompt adherence of the FLUX architecture, allowing users to guide the generation with text descriptions of what should appear in the masked region, providing precise creative control over the output. The model handles complex scenarios including filling regions that span multiple materials and textures, maintaining structural continuity of architectural elements, and generating photorealistic human features in masked face areas. As a proprietary model, FLUX Fill is accessible through Black Forest Labs' API and partner platforms including Replicate and fal.ai, with usage-based pricing. Professional photographers use FLUX Fill for removing unwanted elements and extending compositions, e-commerce teams employ it for product background replacement, digital artists leverage it for creative compositing, and marketing professionals use it for adapting images to different aspect ratios and formats without losing content quality.
GPT Image 1
GPT Image 1 is OpenAI's latest image generation model that integrates natively within the GPT architecture, combining language understanding with visual generation in a unified autoregressive framework. Unlike diffusion-based competitors, GPT Image 1 generates images token by token through an autoregressive process similar to text generation, enabling a conversational interface where users iteratively refine outputs through dialogue. The model excels at text rendering within images, producing legible and accurately placed typography that has historically been a weakness of diffusion models. It supports both generation from text descriptions and editing through natural language instructions, allowing users to upload images and describe desired modifications. GPT Image 1 understands complex compositional prompts with multiple subjects, spatial relationships, and specific attributes, producing coherent scenes accurately reflecting described elements. It handles diverse styles from photorealism to illustration, painting, graphic design, and technical diagrams. Editing capabilities include inpainting, style transformation, background replacement, object addition or removal, and color adjustment, all through conversational input. The model is accessible through the OpenAI API for application integration and through ChatGPT for consumer use. Safety systems prevent harmful content generation. Generated images belong to the user with full commercial rights under OpenAI's terms. GPT Image 1 represents a significant step toward multimodal AI systems seamlessly blending language and visual capabilities, making AI image creation more intuitive through natural conversation.
Quick Info
Stable Diffusion based with task-specific learnable prompt tokens