From the source
- Exa optimized its embedding infrastructure on AWS Neuron hardware, reporting roughly a 1.5× throughput improvement over its initial Inferentia2 implementation.
- The work began with a backfill: moving to a better document model meant re-embedding 30 billion high-priority documents, which temporarily doubled the embedding load and prompted a re-evaluation of hardware choices.
- Exa says the unit economics of embedding underlie everything it does, and at the scale of tens or hundreds of billions of documents any per-document cost improvement compounds.
- The optimizations included weight layout changes, autocast, a drop-in attention kernel, a custom fused SwiGLU MLP kernel to eliminate HBM stalls, and an attention kernel rewritten around the model's exact shapes to cut FLOPS spent on padding.
- Exa says the downcast outputs met its correctness gate of 0.999 cosine similarity against the reference.
- The workload runs on a single NeuronCore, with Kubernetes pods packed onto larger instances so no cores sit idle.






