# Hugging Face — Transformers Model Integration Contracts

- Company: Hugging Face (huggingface.co)
- Announced: 2026-11-03
- Category: developer-tool-release
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://huggingface.co/blog/transformers-mic
- Record: https://forck.live/items/30893-transformers-model-integration-contracts
- Subject: Platform
- Models affected: T5, UMT5, SmolLM

Hugging Face is proposing transformers-mic, a standard for packaging small-model regression tests, after a UMT5 attention bug sat in Transformers' main branch for almost two months. Transformers supports over 750 model types and runs 88,000+ CPU tests on main plus 200,000+ GPU tests daily, so full integration tests cannot run on every pull request. A Model Integration Contract pairs a shrunk fixture built from a real checkpoint's configuration with a runnable usage path and human-reviewed baselines, and the contract holds if the usage path still reproduces those baselines. In a CPU study the experimental UMT5 fixture used 66 MB of weights against the source checkpoint's 1,227 MB, and fixtures also let very large models that do not fit standard GPU runners be tested on CPU. Replaying frozen MICs over five months of commits, the UMT5 fixture flagged the regression from its July 30, 2026 introduction until its September 28 fix, with label log-probabilities off by 1.22 and training loss by about 0.20. The contract used SDPA, the default attention path, and running only the eager attention variant missed the bug. On a proof branch the check failed with the same differences, and the selected T5 contract's runner step took 13.4 seconds on CPU excluding setup. Contracts are published in huggingface-mic, pinned to a frozen revision with the package versions used to record baselines, and pull requests select MICs based on the files a patch changes.

## Evidence

Verbatim from https://huggingface.co/blog/transformers-mic:

> On July 30, 2026, Transformers enabled PyTorch's scaled dot-product attention (SDPA) for T5. As a side effect, UMT5's decoder could attend to future tokens. During training, it could peek at the answer it was supposed to predict. Nothing crashed, and the bug stayed in main for almost two months.
> 
> When we later replayed those commits, a small UMT5 model running on CPU flagged it on every one of them. This post is about that model and the tests built around it.
> 
> Why bugs reach main
> 
> Transformers supports over 750 model types, and its test suite is big: 88,000+ CPU tests on main, plus 200,000+ GPU tests in a daily run. Model tests are only part of it: we also test pipelines, training, distributed execution, quantization and attention kernels.
> 
> The integration tests load real checkpoints and exercise the model's API on real hardware. Running all of that on every PR would mean keeping a lot of GPUs busy all day long, which isn't practical. By the time a scheduled run finds a regression, it may already be in main.
> 
> Our approach is to shrink the model instead.

---

Record: https://forck.live/items/30893-transformers-model-integration-contracts
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md