# Apple — IDEA Prune: An Integrated Enlarge-and-Prune Pipeline in Generative Language Model Pretraining

- Company: Apple (apple.com)
- Announced: 2026-08-26
- Category: research-paper
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://machinelearning.apple.com/research/idea-prune-pipeline
- Record: https://forck.live/items/10307-idea-prune-an-integrated-enlarge-and-prune-pipeline-in-generative-language
- Subject: Machine Learning Research

Apple researchers propose an integrated enlarge-and-prune pipeline for generative language model pretraining that combines enlarged model training, pruning, and recovery under a single cosine annealing learning rate schedule, complemented by a novel iterative structured pruning method. Experiments compressing 2.8B models to 1.3B with up to 2T tokens show the approach mitigates knowledge loss and achieves superior pruned model performance.

## Evidence

Verbatim from https://machinelearning.apple.com/research/idea-prune-pipeline:

> We propose an integrated enlarge-and-prune pipeline, which combines enlarge model training, pruning, and recovery under a single cosine annealing learning rate schedule. This approach is further complemented by a novel iterative structured pruning method for gradual parameter removal. The proposed method helps to mitigate the knowledge loss caused by the rising learning rate in naive enlarge-and-prune pipelines and enable effective redistribution of model capacity among surviving neurons, facilitating smooth compression and enhanced performance.

---

Record: https://forck.live/items/10307-idea-prune-an-integrated-enlarge-and-prune-pipeline-in-generative-language
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md