AI · 1h ago
LLM Parameters Store ~3.6 Bits Each, New Study Finds
A paper from Meta, DeepMind, Cornell, and NVIDIA quantifies LLM memory capacity at about 3.6 bits per parameter in BF16 format. Using random bit strings, researchers isolated memorization from generalization, revealing a phase transition when data exceeds model capacity. The findings also explain double descent and highlight privacy risks for rare training data.
Meridian48 take
The 3.6-bit figure is a useful benchmark, but real-world models mix memorization and generalization, so the practical implications for scaling and privacy may be more nuanced.
llm-memoryinformation-theory