Global-batch load balance almost free lunch to improve your MoE LLM training
The Mixture-of-Experts (MoE) architecture, which is a technique for scaling model parameters, and describes how an MoE layer consists of a router and a group of experts.
Monday
20 January
