# Hugging Face — Mixture of Experts (MoEs) in Transformers

- Company: Hugging Face (huggingface.co)
- Announced: 2026-02-26T00:00:00+00:00
- Subject: Platform
- Source: https://huggingface.co/blog/moe-transformers
- Record: https://forck.live/items/1542-mixture-of-experts-moes-in-transformers

This blog post introduces Mixture of Experts (MoEs) in Transformers, explaining how sparse feed-forward layers replace dense layers to improve compute efficiency, and details the engineering work in the transformers library to support MoEs, including a weight loading refactor, expert backend, and expert parallelism.

## Evidence

Verbatim from https://huggingface.co/blog/moe-transformers:

> Making MoEs first-class citizens in transformers means redesigning parts of the loading pipeline, execution model, and distributed abstractions, not just adding new model classes.

---

Record: https://forck.live/items/1542-mixture-of-experts-moes-in-transformers
Catalogue: https://forck.live/llms.txt
Feed: https://forck.live/feed.md