ResearchTrend.AI
  • Papers
  • Communities
  • Events
  • Blog
  • Pricing
Papers
Communities
Social Events
Terms and Conditions
Pricing
Parameter LabParameter LabTwitterGitHubLinkedInBlueskyYoutube

© 2025 ResearchTrend.AI, All rights reserved.

  1. Home
  2. Papers
  3. 2301.01313
11
36

Decentralized Gradient Tracking with Local Steps

3 January 2023
Yue Liu
Tao R. Lin
Anastasia Koloskova
Sebastian U. Stich
ArXivPDFHTML
Abstract

Gradient tracking (GT) is an algorithm designed for solving decentralized optimization problems over a network (such as training a machine learning model). A key feature of GT is a tracking mechanism that allows to overcome data heterogeneity between nodes. We develop a novel decentralized tracking mechanism, KKK-GT, that enables communication-efficient local updates in GT while inheriting the data-independence property of GT. We prove a convergence rate for KKK-GT on smooth non-convex functions and prove that it reduces the communication overhead asymptotically by a linear factor KKK, where KKK denotes the number of local steps. We illustrate the robustness and effectiveness of this heterogeneity correction on convex and non-convex benchmark problems and on a non-convex neural network training task with the MNIST dataset.

View on arXiv
Comments on this paper