# Alibaba — Global-batch load balance almost free lunch to improve your MoE LLM training

- Company: Alibaba (alibaba.com)
- Announced: 2025-01-20T16:00:03+00:00
- Category: research-paper
- Subject: Qwen
- Source: https://qwenlm.github.io/blog/global-load-balance/
- Record: https://forck.live/items/2299-global-batch-load-balance-almost-free-lunch-to-improve-your-moe-llm-training

The post discusses the Mixture-of-Experts (MoE) architecture, which is a technique for scaling model parameters, and describes how an MoE layer consists of a router and a group of experts.

## Evidence

Verbatim from https://qwenlm.github.io/blog/global-load-balance/:

> The Mixture-of-Experts (MoEs) architecture has become a popular model-parameter-scale-up technique. Typically, one MoE layer consists of a router (often parameterized as one single Linear layer) and a group of experts (for transformer-based models, each expert is one feedforward layer).

---

Record: https://forck.live/items/2299-global-batch-load-balance-almost-free-lunch-to-improve-your-moe-llm-training
Catalogue: https://forck.live/llms.txt
Feed: https://forck.live/feed.md