A federated learning method based on two-layer knowledge distillation

• 1、School of Cyberspace Security, Beijing University of Posts and Telecommunications

Abstract：Federated learning, as a machine learning training method designed to protect user privacy, can complete the training of global models without users sharing data. However, federated learning will face the problem of data heterogeneity in practical applications, that is, the data in the client is usually non-uniformly distributed, which will lead to a significant reduction in the accuracy of the model trained by federated learning, which will cause difficulties in the practical application of federated learning. In this paper, the server and client in federated learning are optimized by knowledge distillation, so that the global aggregation in the server and the local training in the client can learn more global knowledge, reduce the knowledge forgetting phenomenon caused by data heterogeneity, and improve the overall effect of federated learning from the perspective of knowledge transfer. At the same time, this paper simulates the scenario of data heterogeneity through experiments, and compares the proposed federated learning method based on double-layer knowledge distillation with the current mainstream federated learning methods, and the experimental results show that the model trained by the proposed method has obvious advantages in accuracy and convergence speed, which provides a new idea for the use of federated learning in the data heterogeneous scenario.

Keywords： Data Security vs. Computer Security Privacy Protection Federated Learning Knowledge Distillation