From the source
Dharma AI says it cut measured inference cost by 77% in a production Portuguese essay grader by specializing the tokenizer vocabulary of a Qwen3-4B model, without changing the base model.
- The grader, which returns a JSON assessment with a grade and comment per competency, inherited a tokenizer built for many languages and text types, so the text it writes most was split into more tokens than needed.
- The company rebuilt that vocabulary with a method it calls VocabX, keeping the same base model, data and training recipe, and says the numbers come from a paper still under review and may change.
- The waste has two parts: fragmentation, where repeated JSON envelopes, competency names and evaluation phrases cost several tokens where one would do, and excess, where the model scores a vocabulary of 151,669 entries at every generated token.
- Because decode is memory-bound, a generated token cost 7.4 times a token read in the case measured, though the ratio varies with hardware, batch size and context length.
- VocabX controls vocabulary size in two steps, and the output layer in the grader went from about 152,000 to about 65,000 rows.
