Why Inference Matters 12x More Than Training | YC Paper Club
About this episode
From the show’s notesThis week’s Paper Club is all about inference.
For every FLOP spent training a model, roughly six go to serving it each year — and getting that part right has produced over $220 billion in market cap in the last decade. But inference is much more than calling model.forward. Every token has to move through a complicated stack of routing, caching, scheduling, kernels, memory, and hardware, all while balancing speed, cost, and quality.
Read the show’s notes in full
We brought together researchers and builders from Together AI, Ollama, Luminal, and SGLang & RadixArk to take us through that stack from end to end. We’ll see how a token gets from an API request to a GPU and back, why the same open model can perform differently depending on who serves it, and how new approaches to compilers and caching are making inference faster and dramatically more efficient.
Join us for the next Paper Club on November 4th, where we'll talk AI for biology: events.ycombinator.com/yc-paper…
Chapters:
Apply to Y Combinator: ycombinator.com/apply
Work at a startup:
ycombinator.com/jobs
