Emergent and Predictable Memorization in Large Language Models - arxiv.org  ## Metadata - Author: **arxiv.org** - Full Title: Emergent and Predictable Memorization in Large Language Models - Category: #articles - Tags: #ai #llm - URL: https://arxiv.org/abs/2304.11158 ## Highlights - The paper "Emergent and Predictable Memorization in Large Language Models" by Stella Biderman et al. studies the problem of memorization in large language models and proposes a method to predict which sequences will be memorized before full training of the model, based on extrapolation of memorization behavior from lower-compute trial runs, and provides novel insights on the distribution of memorization scores across models and data. Key insights and lessons learned from the paper: Memorization is a key concern for deploying large language models safely, particularly for sensitive datapoints such as PII. Intermediate checkpoints are better predictors of memorization behavior than smaller fully-trained models. Memorization scores follow a power-law distribution across models and data, with some datapoints being more prone to memorization than others. Fine-tuning can mitigate memorization to some extent, but not completely.