Enhancing Inference Efficiency in High-Concurrency Recommendation Models through Operator Scheduling

• 1、School of Computer Science, Beijing University of Posts and Telecommunications, Beijing 100876

Abstract：Recommendation systems are crucial in industries such as e-commerce and digital media. However, as the number of users grows, these systems face significant challenges in processing high-concurrency requests under strict latency constraints. Current scheduling strategies in service frameworks primarily utilize GPU computing power to accelerate the processing of individual requests. Yet, these strategies lack optimization for the concurrent execution of operators and struggle to manage complex requests with varying commercial values. This results in operator interference, inefficient resource utilization and increased inference latency. We propose OpStream, an innovative operator scheduling system that optimizes the inference performance of recommendation models in high-concurrency scenarios. OpStream integrates a holistic scheduling strategy that dynamically allocates operators to multiple CUDA streams based on real-time GPU load monitoring and operator prioritization. By incorporating operator analysis and stream load monitoring, OpStream enhances GPU resource utilization and reduces operator interference, leading to substantial performance improvements. Our experimental results show that OpStream achieves up to 62.9% speedup in inference latency and reduces inference costs by 49.4% in complex request scenarios.?

Keywords： Recommendation System Operator Scheduling Parallel Computing