The Inference Frontier: from 100 to 10,000 tokens per second — Sean Lie, Cerebras CTO
About this episode
From the show’s notesFrom pushing inference beyond 4,000 tokens per second to working with OpenAI on a new generation of ultra-fast AI infrastructure, Cerebras is betting that speed doesn’t just make models faster, it makes entirely new kinds of AI possible. In this episode, Cerebras co-founder and CTO Sean Lie joins swyx and Vibhu fresh off Hot Chips to unpack CS4, preview CS5, and explain why 100–200 tokens per second may soon feel like “batch mode.”
We go deep on Cerebras’ wafer-scale architecture, its partnership with OpenAI, and the emerging hardware stack for frontier inference. Sean breaks down OpenAI’s Jalapeño chip, why AI-first chip design could transform the semiconductor industry, what NVIDIA, Groq, AMD, Etched, and other challengers are getting right and wrong, and why the future of AI infrastructure will increasingly be heterogeneous and disaggregated. We also discuss model-hardware co-design, the path toward 10,000-token-per-second inference, new approaches to memory and 3D packaging, and the growing strategic competition between the US and China across both models and semiconductors.
Read the show’s notes in full
We discuss: • Why Cerebras believes 100–200 tokens per second is becoming the new “batch mode” • CS4 and how Cerebras is pushing inference beyond 4,000 tokens per second • CS5 and the path toward 10,000 TPS on medium-sized models • Running frontier models at up to 5,000 tokens per second • Why Cerebras is effectively sold out of its current capacity • How OpenAI is using ultra-fast inference internally for incident response and critical research • Why faster inference can enable more reasoning loops and more capable agents • OpenAI’s Jalapeño chip and why Sean sees AI-first chip design as the future • How Cerebras and OpenAI could combine CS5 and Jalapeño into a new inference stack • Sean’s critique of Groq and the limitations of SRAM architectures for frontier-scale models • Why inference is breaking apart into specialized workloads like prefill, decode, attention, and expert routing • Why future data centers may increasingly be designed like one giant computer • How models designed around NVIDIA GPUs leave performance gains on the table for alternative architectures • Why hardware-model co-design could unlock massive additional gains • NVIDIA, AMD, TPU, and Trainium — and where traditional accelerator architectures are heading • Sean’s take on Etched and what next-generation AI hardware startups should actually innovate on • Why memory bandwidth, 3D integration, power, and cooling are becoming central bottlenecks • The rise of Chinese open models and China’s increasingly independent AI hardware ecosystem
—
Sean Lie • Cerebras: cerebras.ai • X:
