Anomaly Detection for Large Model Training Based on Log and Traffic Analysis

• 1、School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, Beijing 100876, China

Abstract：In large-scale language model (LLM) training cluster networks, the rapid growth of model parameters significantly increases computational requirements. This demand has driven the expansion of distributed systems, while also bringing challenges in terms of performance and network reliability. As network traffic becomes increasingly complex, potential bottlenecks, communication delays, and packet loss rates begin to have adverse effects on training. To address these issues, this paper proposes a traffic and log aware anomaly detection framework for real-time monitoring and automatic identification of network anomalies. Combining in-depth analysis of network traffic patterns during LLM training with advanced monitoring techniques. By summarizing and extracting traffic features from distributed training tasks, Transformer traffic prediction and log based anomaly analysis tools are integrated. Compared with existing solutions, it can effectively detect and alleviate network problems, identify issues such as slow nodes, link delays, network jitter, and optical module failures, which can significantly affect performance. Therefore, detecting and addressing these issues is crucial for maintaining efficiency. In addition, this method supports real-time fault localization and rapid resolution without interrupting the current training, thereby improving reliability and robustness. Real-time detection and resolution sets it apart from existing methods and provides stronger flexibility during the training process.

Keywords： Fundamentals of Computer Science and Technology Large model network flow Anomaly detection 3D Parallel