From the source
Amazon's new paper derives HOPE, a second-order expert-pruning method that accounts for pairwise interactions between experts in Mixture-of-Experts language models.
- Existing pruning methods score each expert independently, assuming its contribution is independent of all others, but the authors say experts form stable cooperative groups that get consistently co-activated together.
- Ignoring those interactions leads to suboptimal pruning, especially at aggressive rates, because deleting one expert can dismantle a cooperative group.
- At 50% pruning, HOPE achieves the best average rank of any method, with gains of up to +6.1% on agentic coding tasks over the next-best pruning method.
- The authors illustrate the memory problem with Qwen3.5-122B, which has 256 experts in each of its 48 MoE layers and uses only 8 experts per layer for any given token, with expert MLPs taking up 95% of its ~250 GB parameter memory footprint.
- HOPE defines a tractable surrogate that upper-bounds the dominant component of the pruning error, and minimizing it over a calibration set becomes a quadratic program.
- The post explains how HOPE works, where it outperforms existing methods, and how to apply it to other models.






