【マルチモーダル】Text to Image #まとめ編 #01

Index

Text to Image

Text から Image を生成する Vision-Language のマルチモーダルなタスク.

Diffusion Model を用いた手法をまとめる.

• Diffusion Model

  • yhayato1320.hatenablog.com
    
    

• Text-to-Image

  • yhayato1320.hatenablog.com

GLIDE / 2021

• GLIDE
  • yhayato1320.hatenablog.com

RDM / 2022

テキスト入力から画像を生成する際に、外部データベースにある「テキストに合致する画像群」を元に画像生成するという手法.

• Semi-Parametric Neural Image Synthesis
  • [2022]
  • arxiv.org

DreamBooth / 2022

特定の被写体の画像数枚を拡散モデルに学習させることで、特定の被写体を生成する方法を提案.

特定の被写体をモデルに埋め込みながら、拡散モデルが持つ事前知識を忘れないようにすることで目的を達成.

• DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation

  • [2022]
  • arxiv.org
    
    

• 【DL輪読会】DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation

  • 【DL輪読会】DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation from Deep Learning JP
    www.slideshare.net

Composable-Diffusion / 2022

• Compositional Visual Generation with Composable Diffusion Models

  • [2022]
  • arxiv.org
  • energy-based-model.github.io
    
    

• 拡散モデルを組み合わせた合成画像生成

  • ai-scholar.tech

ControlNets / 2023

拡散モデルを humanpose など様々な条件で制御できるようにした.

学習する条件部NNはゼロ重みで初期化した層で入力と出力を挟み、 固定の学習済み拡散モデルUNetの復号部に加える.

学習時プロンプトは半分の確率でドロップアウトし、条件部利用を促進させる.

• Adding Conditional Control to Text-to-Image Diffusion Models

  • [2023]
  • https://github.com/lllyasviel/ControlNet/raw/main/github_page/control.pdf
  • github.com
    
    

• huggingface.co

  • huggingface

T2I-Adapter / 2023

• T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models

  • [2023]
  • arxiv.org
    
    

• huggingface.co

  • huggingface

Fair Diffusion / 2023

• Fair Diffusion: Instructing Text-to-Image Generation Models on Fairness
  • [2023]
  • arxiv.org

Hybrid Diffusion Model / HDM / 2023

• Controlled and Conditional Text to Image Generation with Diffusion Prior
  • [2023]
  • arxiv.org

Directed Diffusion / 2023

• Directed Diffusion: Direct Control of Object Placement through Attention Guidance
  • [2023]
  • arxiv.org

X&Fuse / 2023

• X&Fuse: Fusing Visual Information in Text-to-Image Generation
  • [2023]
  • arxiv.org

VPD / 2023

• Unleashing Text-to-Image Diffusion Models for Visual Perception
  • [2023]
  • arxiv.org
  • github.com
  • vpd.ivg-research.xyz

Word-As-Image / 2023

• Word-As-Image for Semantic Typography
  • [2023]
  • arxiv.org

ODISE / 2023

• Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models
  • [2023]
  • https://arxiv.org/abs/2303.04803
  • arxiv.org
  • huggingface.co

Text-to-Image Model Editing method / TIME / 2023

• Editing Implicit Assumptions in Text-to-Image Diffusion Models
  • [2023]
  • arxiv.org
  • time-diffusion.github.io
  • github.com

HiPer / 2023

• Highly Personalized Text Embedding for Image Manipulation by Stable Diffusion
  • [2023]
  • arxiv.org

P+ / 2023

• P+: Extended Textual Conditioning in Text-to-Image Generation
  • [2023]
  • arxiv.org
  • prompt-plus.github.io

DS-Fusion / 2023

ロゴの生成.

• DS-Fusion: Artistic Typography via Discriminated and Stylized Diffusion
  • [2023]
  • arxiv.org
  • ds-fusion.github.io

GlueGen / 2023

• GlueGen: Plug and Play Multi-modal Encoders for X-to-image Generation
  • [2023]
  • arxiv.org

MagicFusion / 2023

• MagicFusion: Boosting Text-to-Image Generation Performance by Fusing Diffusion Models
  • [2023]
  • arxiv.org
  • magicfusion.github.io

Anti-DreamBooth / 2023

• Anti-DreamBooth: Protecting users from personalized text-to-image synthesis
  • [2023]
  • arxiv.org
  • anti-dreambooth.github.io
  • github.com

Diffusion Classifier / 2023

• Your Diffusion Model is Secretly a Zero-Shot Classifier
  • [2023]
  • arxiv.org
  • diffusion-classifier.github.io

Forget-Me-Not / 2023

• Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models
  • [2023]
  • arxiv.org

SuTI / 2023

• Subject-driven Text-to-Image Generation via Apprenticeship Learning
  • [2023 Google]
  • arxiv.org

Diffusion SpaceTime Attn / 2023

• Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesis
  • [2023]
  • arxiv.org
  • github.com

Continual Diffusion / 2023

• Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRA
  • [2023]
  • arxiv.org
  • jamessealesmith.github.io

RAPHAEL / 2023

• RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Paths
  • [2023]
  • arxiv.org

タスク

Image Editing

• Image Editing
  • yhayato1320.hatenablog.com

Text-to-3D

3DFuse / 2023

• Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation
  • [2023]
  • arxiv.org
  • ku-cvlab.github.io

工夫・テクニック

SVDiff / 2023

Fine Turning のテクニック.

• SVDiff: Compact Parameter Space for Diffusion Fine-Tuning
  • [2023]
  • arxiv.org

Local Prompt Mixing / 2023

• Localizing Object-level Shape Variations with Text-to-Image Diffusion Models
  • [2023]
  • arxiv.org
  • orpatashnik.github.io

Ablating Concepts / 2023

• Ablating Concepts in Text-to-Image Diffusion Models
  • [2023]
  • arxiv.org
  • www.cs.cmu.edu

Discriminative Class Tokens / 2023

• Discriminative Class Tokens for Text-to-Image Diffusion Models
  • [2023]
  • arxiv.org

Layout Guidance / 2023

• Training-Free Layout Control with Cross-Attention Guidance
  • [2023]
  • arxiv.org
  • silent-chen.github.io
  • github.com

参考

• Text-to-Image Diffusion Models are Zero-Shot Classifiers
  • [2023]
  • arxiv.org

Web サイト

• DiffusionによるText2Imageの系譜と生成画像が動き出すまで
  • speakerdeck.com