Lightweight and Efficient Diffusion Model Based on Feature Enhancement

• 1、North China University of Technology, School of Electrical and Control Engineering, Beijing 100144, China

Abstract：Diffusion models have made significant progress in the field of image generation, but there is still room for improvement in the quality of image generation within latent space. U-DiT, an efficient diffusion model that combines U-Net and Transformer architectures, excels in image generation tasks. However, its feature representation capabilities require further enhancement. This paper introduces a feature enhancement method based on inter-channel dependencies named Channel-Correlation Adaptive Recalibration (CCAR). By incorporating CCAR before the self-attention layers within the Transformers blocks, it adaptively adjusts channel-wise weights, complementing the downsampled self-attention mechanism of U-DiT, whiLightweight and Efficient Diffusion Model Based on Feature Enhancementch emphasizes global low-frequency information. This approach optimizes the channel dimensions of feature maps, enhancing the model\'s ability to capture details and structures, thus improving its expression of task-critical features. Moreover, the residual connection design of CCAR avoids additional parameter overhead, enhancing feature extraction capabilities while ensuring the lightweight nature of the model. Experiments on the ImageNet dataset show that the improved U-DiT model reduces the Fréchet Inception Distance (FID) from 10.08 to 8.08 and increases the Inception Score (IS) from 112.44 to 121.33, demonstrating significantly better image generation quality compared to the original model. This study offers new insights into the enhancement of diffusion models and highlights the potential of modeling inter-channel dependencies to boost image generation performance.

Keywords： Artificial Intelligence；Diffusion Models；Transformer；Channel Attention；Image Generation