Distributed training profiler for analyzing compute, communication, memory, and scaling bottlenecks in ML training systems.
A distributed training simulator for data-parallel learning, all-reduce, and communication vs computation trade-offs.
A 6-week masterclass in Distributed Deep Learning Infrastructure from scratch. Covers the $\alpha-\beta$ communication model, custom Ring All-Reduce primitives, Parameter Servers (Hogwild!), 1F1B Pipeline Parallelism, ZeRO/FSDP memory sharding, gradient compression with error feedback, and fault-tolerant elastic training simulators.
Datacenter Computing UPB 2024-2025 - RDMA Client-Server Library
Add a description, image, and links to the all-reduce topic page so that developers can more easily learn about it.
To associate your repository with the all-reduce topic, visit your repo's landing page and select "manage topics."