Apple· 1h agoLocal and Distributed Large LLM Inference with MLX and RDMA on Mac Studio
Running Muse Spark Open on a cluster of four Mac Studios.
Apple researchers demonstrate running the nearly trillion-parameter Muse Spark Open model on a cluster of four Mac Studios using MLX and RDMA over Thunderbolt, achieving over 36 tokens per second.
The M5 Ultra Mac Studio shows up to 4.6x faster time to first token and 1.6x faster generation speed compared to the M3 Ultra, with peak power consumption of about 300W.