Blog

April 27, 2023

Beyond supervised learning, we accelerate large-scale monotonevariational inequality problems with applications such as training GANs indistributed settings. We propose quantized generalized extra-gradient(Q-GenX) family of algorithms with the optimal rate of convergence andachieve noticeable speedups when training GANs on multiple GPUs withoutperformance degradation.

Publication

Distributed extra-gradient with optimal complexity and communication guarantees

May 1, 2023

Ali Ramezani-Kebrya, Kimon Antonakopoulos, Igor Krawczuk, Justin Deschenaux, Volkan Cevher

Paper abstract

We consider monotone variational inequality (VI) problems in multi-GPU settings where multiple processors/workers/clients have access to local stochastic dual vectors. This setting includes a broad range of important problems from distributed convex minimization to min-max and games. Extra-gradient, which is a de facto algorithm for monotone VI problems, has not been designed to be communication efficient. To this end, we propose a quantized generalized extra-gradient (Q-GenX),which is an unbiased and adaptive compression method tailored to solve VIs. We provide an adaptive step-size rule, which adapts to the respective noise profiles at hand and achieve a fast rate of O(1/T) under relative noise, and an order optimal O(1/√T) under absolute noise and show distributed training accelerates convergence. Finally, we validate our theoretical results by providing real-world experiments and training generative adversarial networks on multiple GPUs.