From the source
- A developer has added expert streaming to Transformers so a 180B-parameter MoE model can run on a 32GB MacBook, getting within 6% of Strata on token generation.
- The existing Accelerate path copies a layer's whole expert bank to the GPU on every token, so a model uses 8 or 10 experts but moves all of them.
- The fix turns the expert cache into a slot pool: one preallocated device tensor shared by all layers, plus a per-layer table mapping expert ids to slots.
- Remapping the router's expert ids to slots makes the pool look like a regular expert bank, so grouped_mm runs on it unchanged and no custom expert matmul is needed.
- On OLMoE-1B-7B, which fits in memory, the first version with a Python loop over one expert at a time reached 10.1 tok/s against about 32 for vanilla Transformers; the slot pool brought it to 34.6 tok/s against 38.1 for vanilla.
- The option is exposed through from_pretrained in a Transformers pull request, and the code, model package and Metal kernels were built with AI.






