Alibaba· QwenStory of the weekResearch paper

Global-batch load balance almost free lunch to improve your MoE LLM training

The post discusses the Mixture-of-Experts (MoE) architecture, which is a technique for scaling model parameters, and describes how an MoE layer consists of a router and a group of experts.

From the source

The Mixture-of-Experts (MoEs) architecture has become a popular model-parameter-scale-up technique. Typically, one MoE layer consists of a router (often parameterized as one single Linear layer) and a group of experts (for transformer-based models, each expert is one feedforward layer).

qwenlm.github.io