# Hugging Face — Fast Inference on Large Language Models: BLOOMZ on Habana Gaudi2 Accelerator

- Company: Hugging Face (huggingface.co)
- Announced: 2023-03-28
- Category: not stated
- Coverage: not counted
- Announcement: no
- Group: routine
- Source: https://huggingface.co/blog/habana-gaudi-2-bloom
- Record: https://forck.live/items/2080-fast-inference-on-large-language-models-bloomz-on-habana-gaudi2-accelerator
- Subject: Platform
- Models affected: BLOOMZ, BLOOMZ-7B

To deploy large language models like BLOOMZ on Habana Gaudi2 accelerators using Optimum Habana, and presents benchmarks showing that Gaudi2 achieves lower inference latency than Nvidia A100 80GB GPUs for BLOOMZ models.

## Evidence

Verbatim from https://huggingface.co/blog/habana-gaudi-2-bloom:

> Gaudi2 is 2.89x faster than A100 for BLOOMZ-7B!

---

Record: https://forck.live/items/2080-fast-inference-on-large-language-models-bloomz-on-habana-gaudi2-accelerator
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md