# TII — Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance

- Company: TII (tii.ae)
- Announced: 2025-05-20T12:00:00+00:00
- Category: not stated
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://falcon-lm.github.io/blog/falcon-h1/
- Record: https://forck.live/items/18415-falcon-h1-a-family-of-hybrid-head-language-models-redefining-efficiency-and
- Subject: Falcon LLM

Falcon CHAT Hugging Face Paper Github DEMO DISCORD Introduction Today, we are proud to introduce the Falcon-H1 series, a collection of six open-source models ranging from 0.5B to 34B parameters, each available in both base and instruction-tuned variants. At the core of these models lies a hybrid architecture that combines the strengths of the classical Transformer-based attention mechanism with the State Space Model (SSM), known for its superior long-context memory and computational efficiency. This architectural innovation is further enhanced by fundamental advancements in training dynamics and data utilization, enabling Falcon-H1 models to deliver uncompromised performance that rivals the top Transformer-based models across all covered size tiers. In this release, we feature six open-weight models: 0.5B, 1.5B, 1.5B-Deep, 3B, 7B, and 34B, along with their instruct versions. All our open-source models are with a permissive license based on Apache 2.0. Key Features of Falcon-H1 Hybrid Architecture (Attention + SSM): We combine attention and Mamba-2 heads in parallel within our hybrid mixer block. …

---

Record: https://forck.live/items/18415-falcon-h1-a-family-of-hybrid-head-language-models-redefining-efficiency-and
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md