From the source
Hugging Face has published a Space that records which popular Transformers models on the Hub run unmodified on AWS Trainium, after the TorchNeuron backend made that possible.
- Each row reports the result, the number of Trainium cores the model is sharded over, the wall time including compilation, and peak device memory.
- Tests load the model on a trn2 instance over the smallest number of cores that hold its weights, generate 32 tokens and check them, comparing against a CPU run on a single core and using a small judge model for fluent English when sharded.
- Only models fitting on 4 Trainium2 devices, at most 16 cores, are tested, and a PASS says nothing about throughput or longer generations.
- The Space lists the Neuron stack, packages and run.py that produced each result, and users can skip graph compilation by syncing cached graphs from the public NEFF cache bucket.
- Coverage is limited to text-generation and image-text-to-text models for now, with plans to add diffusers, sentence transformers and other tasks; all results come from the public aws-neuron/torch-neuronx-model-support dataset.
