From the source
From the source
Ai2 compares token-level predictions of Olmo Hybrid and Olmo 3 transformers.
Ai2 published research comparing its 7B transformer Olmo 3 and hybrid model Olmo Hybrid at the token level.
The hybrid predicts content words such as nouns, verbs, and adjectives better than the transformer, but its advantage nearly disappears on tokens that repeat verbatim from earlier in the input, where the transformer's attention mechanism excels.
From the source
Our results show that the hybrid's advantage is real across many tokens—but not all. Olmo Hybrid is strongest on tokens that carry meaning, such as nouns, verbs, and adjectives, and on tokens that can only be predicted by following what's going on—like which person a pronoun refers to. But the hybrid's advantage almost disappears on tokens that simply repeat something already in the input – a word or phrase reproduced verbatim from earlier – where the answer is sitting right there to be looked up.
allenai.org
