# Baseten — Why do models forget facts that they iteratively learn?

- Company: Baseten (baseten.co)
- Announced: not stated by the source (first seen 2026-11-06T18:26:26.917531+00:00)
- Category: research-paper
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://www.baseten.co/blog/why-do-models-forget-facts-that-they-iteratively-learn
- Record: https://forck.live/items/32764-why-do-models-forget-facts-that-they-iteratively-learn
- Subject: Baseten
- Models affected: talkie-1930-13b-it, Qwen3

Baseten's Base Labs reran its continual-learning experiment with real sports facts to separate interference between facts from general model damage. The earlier paper wrote invented facts that conflicted with world knowledge into Qwen3 models, and repeated learning of those facts damaged the model's general ability, making it hard to tell how much forgetting came from interference. The new run uses talkie-1930-13b-it, a 13B model trained only on text published before 1931, and writes twenty sports results from 2011-2019, facts that are new to the model but ordinary. Recall rose from 0% at base to 91% immediately after each fact's write and fell to 33% after all twenty writes, with late facts surviving and early ones lost. Losses were often abrupt: the 2011 Wimbledon women's champion was recalled perfectly until the 2019 US Open women's champion was written three facts later, then dropped to zero. Overwritten facts were not erased and could be relearned faster, and activation patching showed the overwrites happen largely in later-layer MLP weights. Code, data and checkpoints are released at basetenlabs/talkie-facts-replication.

## Evidence

Verbatim from https://www.baseten.co/blog/why-do-models-forget-facts-that-they-iteratively-learn:

> A core thesis of ours at Base Labs is to try and follow research threads to their end, much as a PI at a University may pursue a research agenda over a longer period of time. In doing so, we hope to avoid the ‘conference cycle’ of simply publishing a result and moving on.
> 
> In this piece we detail a deliberately narrow extension of our paper Can a Language Model Continually Learn Facts in its Weights? ( O'Neill et al., 2026 ). There we studied continual learning in the iterative regime, writing a stream of invented facts (that conflicted with the world knowledge of the model) one at a time into various Qwen3 models. For each new fact we train a new LoRA adapter, and merge this into the model weights before we learn the next fact (e.g. fact 1 LoRA, merge, fact 2 LoRA etc.). There we found that the model was better at answering questions about facts when exposed to the fact in some kind of varied way during training (e.g. the fact paraphrased or written in different ways). Separately, we found that facts that the model had behaviorally forgotten were still present in its log-probs.
> 
> This paper left two major questions answered.

---

Record: https://forck.live/items/32764-why-do-models-forget-facts-that-they-iteratively-learn
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
