Rethinking Self-Attention for Multispectral Object Detection - Informatique, Biologie Intégrative et Systèmes Complexes Accéder directement au contenu
Article Dans Une Revue IEEE Transactions on Intelligent Transportation Systems Année : 2024

Rethinking Self-Attention for Multispectral Object Detection

Résumé

Data from different modalities, such as infrared and visible images, can offer complementary information, and integrating such information can significantly enhance the capabilities of a system to perceive and recognize its surroundings. Thus, multi-modal object detection has widespread applications, particularly in challenging weather conditions like low-light scenarios. The core of multi-modal fusion lies in developing a reasonable fusion strategy, which can fully exploit the complementary features of different modalities while preventing a significant increase in model complexity. To this end, this paper proposes a novel lightweight cross-fusion module named Channel-Patch Cross Fusion (CPCF), which leverages Channel-wise Cross-Attention (CCA), Patch-wise Cross-Attention (PCA) and Adaptive Gating (AG) to encourage mutual rectification among different modalities. This process simultaneously explores commonalities across modalities while maintaining the uniqueness of each modality. Furthermore, we design a versatile intermediate fusion framework that can leverage CPCF to enhance the performance of multi-modal object detection. The proposed method is extensively evaluated on multiple public multi-modal datasets, namely FLIR, LLVIP, and DroneVehicle. The experiments indicate that our method yields consistent performance gains across various benchmarks and can be extended to different types of detectors, further demonstrating its robustness and generalizability. Our codes are available at \url{https://github.com/Superjie13/CPCF\_Multispectral}.
Fichier principal
Vignette du fichier
IEEE_ITS_2023_Rethinking_Self_Attention_for_Multispectral_Object_Detection.pdf (4.41 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04620359 , version 1 (21-06-2024)

Identifiants

Citer

Sijie Hu, Fabien Bonardi, Samia Bouchafa, Helmut Prendinger, Désiré Sidibé. Rethinking Self-Attention for Multispectral Object Detection. IEEE Transactions on Intelligent Transportation Systems, inPress, pp.1-12. ⟨10.1109/TITS.2024.3412417⟩. ⟨hal-04620359⟩
0 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Mastodon Facebook X LinkedIn More