# Hugging Face — How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications

- Company: Hugging Face (huggingface.co)
- Announced: 2026-10-21T17:35:36+00:00
- Category: not stated
- Coverage: not counted
- Announcement: no
- Group: routine
- Source: https://huggingface.co/blog/Dharma-AI/vocabx-vocabulary-specialization
- Record: https://forck.live/items/26060-how-specializing-the-vocabulary-of-slms-can-cut-inference-cost-by-77-in-real
- Subject: Platform
- Models affected: Qwen3-4B

Dharma AI says it cut measured inference cost by 77% in a production Portuguese essay grader by specializing the tokenizer vocabulary of a Qwen3-4B model, without changing the base model. The grader, which returns a JSON assessment with a grade and comment per competency, inherited a tokenizer built for many languages and text types, so the text it writes most was split into more tokens than needed. The company rebuilt that vocabulary with a method it calls VocabX, keeping the same base model, data and training recipe, and says the numbers come from a paper still under review and may change. The waste has two parts: fragmentation, where repeated JSON envelopes, competency names and evaluation phrases cost several tokens where one would do, and excess, where the model scores a vocabulary of 151,669 entries at every generated token. Because decode is memory-bound, a generated token cost 7.4 times a token read in the case measured, though the ratio varies with hardware, batch size and context length. VocabX controls vocabulary size in two steps, and the output layer in the grader went from about 152,000 to about 65,000 rows.

## Evidence

Verbatim from https://huggingface.co/blog/Dharma-AI/vocabx-vocabulary-specialization:

> In the case we measured, the grader, a generated token costs 7.4 times a token read, but the ratio varies with the hardware, the problem, the batch size, the context length and other factors, and it can be larger than that.

---

Record: https://forck.live/items/26060-how-specializing-the-vocabulary-of-slms-can-cut-inference-cost-by-77-in-real
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md