Small data set recognition network based on self-supervised Transformer

• 1、School of Cyberspace Security, Beijing University of Posts and Telecommunications,Beijing 100876

Abstract：Vision Transformer (ViT), an architectural network distinct from convolutional neural networks, has multiple advantages including design simplicity, robustness, and achieving sota on many vision tasks. However, compared with the convolutional neural network, ViT lacks an inductive bias, so a large amount of data set pre-training is required to learn the inductive bias from it, making it difficult to train from scratch on a small data set. The purpose of this paper is to design a robust training scheme for small-scale datasets. Take a two-stage approach. In the first stage, a self-supervised learning scheme is designed to train on a small data set and learn inductive biases from it as initial weights. In the second stage, the ViT image segmentation stage is optimized and fine-tuned with a small dataset using the initialized weights on the optimized ViT model. Through extensive experiments on a variety of public small data sets, it is proved that the method proposed in this paper has better performance compared with existing algorithms.

Keywords： computer application technology image recognition ViT self-supervised learning small dataset