# Amazon — Your Experts Have Partners. There's HOPE for Keeping Them Together

- Company: Amazon (amazon.com)
- Announced: 2026-11-09T12:34:29+00:00
- Category: research-paper
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://huggingface.co/blog/amazon/higher-order-pruning-experts
- Record: https://forck.live/items/33400-your-experts-have-partners-there-s-hope-for-keeping-them-together
- Models affected: Qwen3.5-122B

Amazon's new paper derives HOPE, a second-order expert-pruning method that accounts for pairwise interactions between experts in Mixture-of-Experts language models. Existing pruning methods score each expert independently, assuming its contribution is independent of all others, but the authors say experts form stable cooperative groups that get consistently co-activated together. Ignoring those interactions leads to suboptimal pruning, especially at aggressive rates, because deleting one expert can dismantle a cooperative group. At 50% pruning, HOPE achieves the best average rank of any method, with gains of up to +6.1% on agentic coding tasks over the next-best pruning method. The authors illustrate the memory problem with Qwen3.5-122B, which has 256 experts in each of its 48 MoE layers and uses only 8 experts per layer for any given token, with expert MLPs taking up 95% of its ~250 GB parameter memory footprint. HOPE defines a tractable surrogate that upper-bounds the dominant component of the pruning error, and minimizing it over a calibration set becomes a quadratic program. The post explains how HOPE works, where it outperforms existing methods, and how to apply it to other models.

## Evidence

Verbatim from https://huggingface.co/blog/amazon/higher-order-pruning-experts:

> Higher-Order Pruning of Experts in Mixture-of-Experts Language Models
> 
> Introduction
> 
> Mixture-of-Experts (MoE) models have enabled state-of-the-art LLMs to be scaled to hundreds of billions of parameters, but they come with a fundamental and costly caveat: every expert MLP must occupy memory at inference time, even though the vast majority are idle for any given token. For example, Qwen3.5-122B has 256 experts in each of its 48 MoE layers, and 95% of its ~250 GB parameter memory footprint is taken up entirely by these expert MLPs alone. However, for any given token, only 8 experts in each layer are actually used.
> 
> Expert pruning is one of the most direct fixes to this memory bottleneck. That is, permanently delete the least useful experts. Pruning avoids the latency issues which make dynamic loading/off-loading of experts impractical; it does not require any training; and it composes with quantization . Given these attractive benefits and the simplicity of the idea, it's easy to see why expert pruning has become a go-to approach for shrinking MoEs. The difficulty lies in deciding which experts are safest to remove.

---

Record: https://forck.live/items/33400-your-experts-have-partners-there-s-hope-for-keeping-them-together
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
