Advances in Artificial Intelligence and Security | 2021
Asynchronous Compatible D-ML System for Edge Devices
Abstract
The parameter server system (Parameter Server, PS) is the most widely used distributed machine learning system. Its core module is the communication module, which mainly includes three parts: communication content, communication frequency, and communication pace. The communication scheme of the existing PS system will have problems such as waste of computing power resources and a large proportion of communication overhead. This paper hopes to design and implement a distributed set that saves computing power and has a smaller communication overhead based on the communication module of the original PS system. Machine learning system, this article mainly has the following contributions: First, through the analysis and experiment of Bulk Synchronous Parallel (BSP) communication pace, based on the superiority of BSP algorithm level, it overcomes the waste of computing power of BSP at the system level, and designs and implements a new set of Communication pace, the new communication pace under the same training task can save more computing power and achieve higher model accuracy. Secondly, this article combines the quantization compression technology with the parameter server system, and uses the quantization technology to compress and decompress the communication content in the parameter server system, so that the volume of the communication content during the operation of the parameter server system is significantly reduced, thereby reducing the communication overhead and improving Speedup ratio of the system. Finally, based on the analysis and experiment of the change trend of the model accuracy rate, this paper designs and implements a dynamic communication frequency adjustment scheme based on the change trend of the model accuracy rate, which reduces the overall communication frequency during the operation of the parameter server system and reduces the parameter server. T Experiments show that the system designed in this paper is superior to the existing mainstream solutions in terms of communication pace, communication content, and communication frequency. When training the same model, this system compared with the existing parameter server system in model convergence time, model There is a noticeable improvement in the highest accuracy.