# Hugging Face — 180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers

- Company: Hugging Face (huggingface.co)
- Announced: 2026-11-18
- Category: capability-change
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://huggingface.co/blog/moe-expert-streaming
- Record: https://forck.live/items/38074-180b-parameters-32gb-of-ram-pushing-moe-inference-to-the-limit-with-transformers
- Subject: Platform
- Models affected: Qwen3.8-Flash-Next, OLMoE-1B-7B, Qwen3.6-35B-A3B

A developer has added expert streaming to Transformers so a 180B-parameter MoE model can run on a 32GB MacBook, getting within 6% of Strata on token generation. The existing Accelerate path copies a layer's whole expert bank to the GPU on every token, so a model uses 8 or 10 experts but moves all of them. The fix turns the expert cache into a slot pool: one preallocated device tensor shared by all layers, plus a per-layer table mapping expert ids to slots. Remapping the router's expert ids to slots makes the pool look like a regular expert bank, so grouped_mm runs on it unchanged and no custom expert matmul is needed. On OLMoE-1B-7B, which fits in memory, the first version with a Python loop over one expert at a time reached 10.1 tok/s against about 32 for vanilla Transformers; the slot pool brought it to 34.6 tok/s against 38.1 for vanilla. The option is exposed through from_pretrained in a Transformers pull request, and the code, model package and Metal kernels were built with AI.

## Evidence

Verbatim from https://huggingface.co/blog/moe-expert-streaming:

> Streaming MoE experts directly from SSD gets within 6% of Strata on token generation.
> 
> I've been trying to rely less on online models and run bigger ones locally. At home I have a PC with two RTX 4090s (let's call it Orthrus - the two-headed hound of Greek mythology), and for work I have a MacBook M5 with 32G of unified memory.
> 
> Orthrus runs on Windows 11, because my kids play games all the time. I had a dual boot with Ubuntu, but their games ended up taking most of the two terabytes of disk.
> 
> I am also a huge fan of RMD (rooms moving development): one hour in the kitchen sipping my coffee, two hours at my office desk, then the sofa. A fixed desktop tower is not for me.
> 
> My first experiment was to run Qwen 3.8 27B with llama.app on Orthrus and use it from my laptop through pi . That model does the coding tasks, an online model like Opus does the orchestration, and the second RTX runs a FIM model for auto-completion in Neovim.
> 
> When I talked about this on social networks, I got very strong feedback: use Strata with Qwen3.8-Flash-Next .
> 
> What is Strata?
> 
> Strata is an inference engine for MoE models like Qwen. Only the hot experts stay in VRAM, the rest goes to RAM, or even to disk.

---

Record: https://forck.live/items/38074-180b-parameters-32gb-of-ram-pushing-moe-inference-to-the-limit-with-transformers
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
