Publication Date
Spring 2026
Degree Type
Master's Project
Degree Name
Master of Science in Computer Science (MSCS)
Department
Computer Science
First Advisor
Genya Ishigaki
Second Advisor
Fabio Di Troia
Third Advisor
Jelena Gligorijevic
Keywords
Reinforcement Learning, Pipeline Parallelism, Gradient Compression, Distributed Training
Abstract
On-policy deep reinforcement learning is a family of reinforcement learning algorithms in which an agent interacts with an environment to optimize a reward objective by alternating between a rollout phase, where it collects experiences using the current policy, and a training phase, where its policy, parameterized by a neural network, is updated using the collected experiences. In resource-constrained environments, where an entire policy model cannot fit on a single device, pipeline parallelism can be used to split the neural network model across multiple devices. However, it substantially increases communication overhead, since we need to transfer both forward activations and backward gradients between devices for each training epoch. In this project, we present a technique that accumulates gradients and only transmits high-magnitude gradients during backward propagation. Furthermore, we introduce an exponential buffer decay mechanism to mitigate gradient staleness, which prevents outdated accumulated gradients from making harmful updates to the policy. Our experiment shows that the proposed communication-efficient pipeline parallelism reduces communication load by 35\%, while maintaining comparable training performance to the full gradient updates.
Recommended Citation
Ahmed, Faiz Sameer, "Communication-Efficient Pipeline Parallelism for On-policy Reinforcement Learning" (2026). Master's Projects. 1799.
DOI: https://doi.org/10.31979/etd.y2yu-6szj
https://scholarworks.sjsu.edu/etd_projects/1799
