IT之家 · ITHome
Google Research introduces Sleep Framework for continual learning in language models
Google Research introduced Sleep Framework, which splits language-model operation into an awake phase for receiving information and a sleep phase for consolidating memories. The approach aims to absorb new knowledge while retaining existing capabilities, using synthetic examples and reinforcement learning to improve reasoning. The study reports better results than several baselines on translation, long-context understanding and mathematics tasks.
What to know
- 01
Memory consolidation combines parameter expansion, mixture-of-experts memory blocks and knowledge distillation to transfer short-term information into long-term parameters.
- 02
Reported results include handling contexts of up to about 10 million tokens and near-zero performance decline during multistage continual learning on previously unseen languages.
Why it matters
The research addresses how language models can update their knowledge after deployment while limiting losses in existing capabilities.
AI-assisted brief. Read the source for full details. Editorial & sourcing policy
