DiffX: Guide Your Layout to Cross-Modal Generative Modeling

Diffusion models have made significant strides in language-driven and layout-driven image generation. However, most diffusion models are limited to visible RGB image generation. In fact, human perception of the world is enriched by diverse viewpoints, such as chromatic contrast, thermal illumination, and depth information. In this paper, we introduce a novel diffusion model for general layout-guided cross-modal generation, called DiffX. Notably, DiffX presents a simple yet effective cross-modal generative modeling pipeline, which conducts diffusion and denoising processes in the modality-shared latent space. Moreover, we introduce the Joint-Modality Embedder (JME) to enhance interaction between layout and text conditions by incorporating a gated attention mechanism. Meanwhile, the advanced Long-CLIP is employed for long caption embedding for user instruction. To facilitate the user-instructed generative training, we construct the cross-modal image datasets with detailed text captions assisted by the Large-Multimodal Model (LMM). Through extensive experiments, DiffX demonstrates robustness in cross-modal generation across three ``RGB+X'' datasets: FLIR, MFNet, and COME15K, guided by various layout conditions. It also shows the potential for the adaptive generation of ``RGB+X+Y+Z'' images or more diverse modalities on COME15K and MCXFace datasets. Our code and constructed cross-modal image datasets are available at https://github.com/zeyuwang-zju/DiffX.