Adaptive Worker Grouping for Communication-Efficient and Straggler-Tolerant Distributed SGD

TY - CHAP

T1 - Adaptive Worker Grouping for Communication-Efficient and Straggler-Tolerant Distributed SGD

AU - Zhu, Feng

AU - Zhang, Jingjing

AU - Simeone, Osvaldo

AU - Wang, Xin

N1 - Funding Information: The work of Feng Zhu and Jingjing Zhang has been supported by National Natural Science Foundation of China Grant No. 62101134. Osvaldo Simeone has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 Research and Innovation Programme (Grant Agreement No. 725731). Xin Wang has been supported by the Innovation Program of Shanghai Municipal Science and Technology Commission Grant 20JC1416400, and the National Natural Science Foundation of China Grant No. 62071126. Publisher Copyright: © 2022 IEEE.

PY - 2022

Y1 - 2022

N2 - Wall-clock convergence time and communication load are key performance metrics for the distributed implementation of stochastic gradient descent (SGD) in parameter server settings. Communication-adaptive distributed Adam (CADA) has been recently proposed as a way to reduce communication load via the adaptive selection of workers. CADA is subject to performance degradation in terms of wall-clock convergence time in the presence of stragglers. This paper proposes a novel scheme named grouping-based CADA (G-CADA) that retains the advantages of CADA in reducing the communication load, while increasing the robustness to stragglers at the cost of additional storage at the workers. G-CADA partitions the workers into groups of workers that are assigned the same data shards. Groups are scheduled adaptively at each iteration, and the server only waits for the fastest worker in each selected group. We provide analysis and experimental results to elaborate the significant gains on the wall-clock time, as well as communication load and computation load, of G-CADA over other benchmark schemes.

AB - Wall-clock convergence time and communication load are key performance metrics for the distributed implementation of stochastic gradient descent (SGD) in parameter server settings. Communication-adaptive distributed Adam (CADA) has been recently proposed as a way to reduce communication load via the adaptive selection of workers. CADA is subject to performance degradation in terms of wall-clock convergence time in the presence of stragglers. This paper proposes a novel scheme named grouping-based CADA (G-CADA) that retains the advantages of CADA in reducing the communication load, while increasing the robustness to stragglers at the cost of additional storage at the workers. G-CADA partitions the workers into groups of workers that are assigned the same data shards. Groups are scheduled adaptively at each iteration, and the server only waits for the fastest worker in each selected group. We provide analysis and experimental results to elaborate the significant gains on the wall-clock time, as well as communication load and computation load, of G-CADA over other benchmark schemes.

KW - Adaptive selection

KW - coding

KW - distributed learning

KW - grouping

KW - stochastic gradient descent (SGD)

UR - http://www.scopus.com/inward/record.url?scp=85136272913&partnerID=8YFLogxK

U2 - 10.1109/ISIT50566.2022.9834752

DO - 10.1109/ISIT50566.2022.9834752

M3 - Conference paper

AN - SCOPUS:85136272913

T3 - IEEE International Symposium on Information Theory - Proceedings

SP - 2996

EP - 3000

BT - 2022 IEEE International Symposium on Information Theory, ISIT 2022

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2022 IEEE International Symposium on Information Theory, ISIT 2022

Y2 - 26 June 2022 through 1 July 2022

ER -