Multimodal image-text model fine-tuning method based on fine-grained auxiliary tasks

• 1、School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing 100000

Abstract：With the development of vision-language multimodal models, their ability to recognize coarse-grained entities has been significantly improved. However, the current model's ability to process fine-grained information is relatively weak. Currently, many studies have improved the fine-grained capability of models by introducing fine-grained tasks. However, most studies have overlooked the important role of introducing fine-grained tasks during the fine-tuning phase. Therefore, this article introduces the fine-grained task Masked Language Modeling into the fine-tuning stage of multimodal models, using it as an auxiliary task to enhance the fine-grained capability of the model. Finally, the experimental results on vlm-probing, food-500 cap and other datasets in this article show that using Masked Language Modeling as auxiliary tasks can improve the fine-grained ability of multimodal models and enhance the image-text retrieval performance on datasets with rich descriptions. Meanwhile, the model's image-text retrieval performance will not decrease on general datasets.

Keywords： Intelligence Science and Technology, Multimodal Image-Text Models, Fine-tuning, Fine-grained Tasks