From the source
From the source
Integrated enlarge-and-prune pipeline improves pruned model performance in LLM pretraining.
Apple researchers propose an integrated enlarge-and-prune pipeline for generative language model pretraining that combines enlarged model training, pruning, and recovery under a single cosine annealing learning rate schedule, complemented by a novel iterative structured pruning method.
Experiments compressing 2.8B models to 1.3B with up to 2T tokens show the approach mitigates knowledge loss and achieves superior pruned model performance.
From the source
We propose an integrated enlarge-and-prune pipeline, which combines enlarge model training, pruning, and recovery under a single cosine annealing learning rate schedule. This approach is further complemented by a novel iterative structured pruning method for gradual parameter removal. The proposed method helps to mitigate the knowledge loss caused by the rising learning rate in naive enlarge-and-prune pipelines and enable effective redistribution of model capacity among surviving neurons, facilitating smooth compression and enhanced performance.
machinelearning.apple.com