# Amazon — Exa boosts embedding throughput by 50% to index billions of documents on AWS AI chips

- Company: Amazon (amazon.com)
- Announced: 2026-11-18T23:58:22+00:00
- Category: infrastructure-release
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://aws.amazon.com/blogs/machine-learning/exa-boosts-embedding-throughput-by-50-to-index-billions-of-documents-on-aws-ai-chips/
- Record: https://forck.live/items/38351-exa-boosts-embedding-throughput-by-50-to-index-billions-of-documents-on-aws-ai
- Subject: Bedrock / Nova

Exa optimized its embedding infrastructure on AWS Neuron hardware, reporting roughly a 1.5× throughput improvement over its initial Inferentia2 implementation. The work began with a backfill: moving to a better document model meant re-embedding 30 billion high-priority documents, which temporarily doubled the embedding load and prompted a re-evaluation of hardware choices. Exa says the unit economics of embedding underlie everything it does, and at the scale of tens or hundreds of billions of documents any per-document cost improvement compounds. The optimizations included weight layout changes, autocast, a drop-in attention kernel, a custom fused SwiGLU MLP kernel to eliminate HBM stalls, and an attention kernel rewritten around the model's exact shapes to cut FLOPS spent on padding. Exa says the downcast outputs met its correctness gate of 0.999 cosine similarity against the reference. The workload runs on a single NeuronCore, with Kubernetes pods packed onto larger instances so no cores sit idle.

## Evidence

Verbatim from https://aws.amazon.com/blogs/machine-learning/exa-boosts-embedding-throughput-by-50-to-index-billions-of-documents-on-aws-ai-chips/:

> At Exa , we are building one of the few true web indexes by aggregating and serving knowledge from across the web, along with private and synthetic sources. We optimize the index for semantic search, grounding agents and humans in true, high-quality data. We use multi-tower semantic machine learning (ML) models to embed web data and query data into the same vector space, so users can express complex queries in natural language alongside traditional keyword search. The keystone technology behind this is an in-house vector database. It must ingest as much high-quality web data as possible and embed all of it. The unit economics of embedding therefore underlie everything we do. At the scale of tens or hundreds of billions of documents, finding the hardware that delivers the best performance per dollar isn’t a tuning detail. Any per-document cost improvement compounds into a significant number.
> 
> This post details how we optimized our embedding infrastructure for cheaper, more flexible inference workloads on AWS Neuron hardware ( AWS Inferentia2 and AWS Trainium2 ), and how an agentic development workflow drove the optimization.

---

Record: https://forck.live/items/38351-exa-boosts-embedding-throughput-by-50-to-index-billions-of-documents-on-aws-ai
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
