Debiasing Method for Multimodal Foundation Models Based on Cross-Modal Feature Decoupling

• 1、School of Computer Science and Technology , Beijing Jiaotong University

Abstract：Machine learning models have been proven to inherit biases from their training datasets, especially the multimodal foundation models trained on large-scale, unprocessed datasets crawled from the In-ternet. These biases will be amplified and propagated to downstream tasks after fine-tuning. This study proposes a general method for debiasing multimodal foundation models by decoupling the bi-as attribute features and the target attribute features in the multimodal embedding through orthogo-nal projection during the fine-tuning process. This enables the model to learn a fair representation that does not contain bias information. Moreover, the fairness research on multimodal foundation models often involves multiple bias attributes. We demonstrate the effectiveness of our method, which can decouple multiplDebiasing Method for Multimodal Foundation Models Based on Cross-Modal Feature Decouplinge bias attributes simultaneously in a single fine-tuning with only a small amount of computing resources.

Keywords： multimodal fairness feature decoupling multi-bias debiasing