Multimodal target fusion

• 1、School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing, 100876
• 2、School of School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing, 100876

Abstract：The traditional target detection algorithm and the target detection framework of conventional convolutional neural network only use a single modal target image for detection, which is difficult to adapt to complex environments and poor lighting conditions, and the detection accuracy is low. To address this issue, a multi-modal object detection network based on Transformer is proposed, which integrates visible light images of textures with thermal infrared images. The two modalities complement each other to reconstruct the target image and improve detection accuracy. In order to fully extract the features of different modal targets, a variant feature extraction network module based on the Tranformer architecture was designed, and the YOLOv5 network was used to perform object detection analysis on the fusion results. This module can adaptively extract complementary feature information from different modalities and use it for input feature fusion to reconstruct the network and obtain results. The experiment shows that the proposed fusion network can improve the quality of fused images, thereby enhancing the accuracy of detection results.

Keywords： Transfomer Multimodal object detection Feature extraction network Infrared and visible light images Adaptive feature extraction