Version 4

Narrative

Timeline

• 2016-01-01: Zhang et al. demonstrate that standard neural networks can memorize completely random labels on CIFAR-10 and ImageNet, invalidating data-independent generalization bounds. [18]
• 2019-01-01: Nagarajan and Kolter show empirically that spectral-norm bounds scale in the wrong direction, and prove formally in an overparameterized linear setting that uniform convergence is provably insufficient to explain gradient descent generalization. [17][44][45]
• 2019-06-01: Feldman publishes 'Does Learning Require Memorization? A Short Tale about a Long Tail,' arguing memorization of tail examples is causally necessary for learning from long-tailed distributions — reframing Zhang et al.'s finding as evidence that memorization is part of learning, not evidence of theory failure. [6][7][8][11][13][12]
• 2022-01-01: A NeurIPS 2022 paper claims PAC-Bayes compression bounds can be made tight enough to actually explain generalization in neural networks, directly challenging the narrative that all known bounds are vacuous. [27][28][29]
• 2024-01-01: NeurIPS 2024 paper on symmetries in overparameterized neural networks using a mean-field view offers a new structural lens on why overparameterization does not prevent generalization. [46][47]
• 2025-01-01: CMU PhD blog post argues classical generalization theory is more predictive for foundation models than for conventional deep networks, implying the theory-failure narrative may not apply uniformly across architectures and scales. [30]
• 2025-01-01: NeurIPS 2025 poster 'Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networks' (Montanari and Urbani) presents results separating generalization dynamics from overfitting dynamics in the large two-layer regime; paper subsequently receives OpenReview listing, Semantic Scholar indexing, and a YouTube talk, confirming broad academic uptake. [1][4][24][2][3][25][5]
• 2026-04-24: Jamie Simon and Daniel Kunin (UC Berkeley) appear on Imbue's podcast arguing that a scientific theory of deep learning is achievable, marking the first direct public advocacy by the learning mechanics authors. [21]
• 2026-04-25: LawrenceC publishes a critical review of Simon et al.'s 'learning mechanics' manifesto on the Alignment Forum, welcoming its ambition while doubting it will deliver a comprehensive or broadly useful theory. [19]
• 2026-04-26: LawrenceC publishes 'The paper that killed deep learning theory,' providing detailed technical and historical context for why Zhang et al. 2016 was so devastating to the classical generalization-bound paradigm. [18]
• 2026-04-27: LawrenceC publishes 'The other paper that killed deep learning theory,' narrating Nagarajan and Kolter 2019 as the definitive proof that uniform convergence cannot explain neural network generalization; the post is crossposted to LessWrong and drives renewed interest in the original paper. [17][20]
• 2026-04-01: YouTube video 'There Will Be a Scientific Theory of Deep Learning' (Apr 2026) is published, likely corresponding to Simon and Kunin's learning mechanics presentation. [22]
• 2026-04-01: Alignment Forum quick paper review of 'There Will Be a Scientific Theory of Deep Learning' appears, representing a second community voice engaging directly with the learning mechanics paper. [23]
• 2026-04-30: OpenReview submission 'Is Memorization Actually Necessary for Generalization?' appears, representing a formal challenge to Feldman's affirmative claim and deepening the memorization-causation debate. [9]

Perspectives

Tensions

• Can learning mechanics, which focuses on average-case training dynamics and coarse aggregate statistics, ever constitute a comprehensive theory of deep learning — or is it structurally limited to explaining some aspects while leaving others (especially what specific algorithms individual networks learn) permanently outside its scope? [19][21][22][23][14]
• Is memorization causally necessary for generalization on long-tailed distributions, or is it merely correlated? Feldman's affirmative argument is now contested by a direct OpenReview submission ('Is Memorization Actually Necessary for Generalization?'), and the outcome reshapes how Zhang et al.'s 2016 result should be interpreted: either memorization signals theory failure, or it signals that theory was measuring the wrong objective. [18][6][9][10][7][8]
• Nearly a decade after Nagarajan and Kolter precisely diagnosed the failure of uniform convergence, no satisfactory algorithm- and data-dependent generalization theory has emerged. The dynamical decoupling result for two-layer networks and the algorithmic stability cluster are the newest candidates with the right structural properties, but neither has yet been shown to extend to the multi-layer, non-linear regime of modern LLMs. [17][1][4][2][5][31][32][33][37][38]
• PAC-Bayes compression bounds are claimed to be tight enough to explain generalization, while a separate line of work argues tighter bounds always come at a price. Which claim holds, and does either constitute a satisfying theoretical explanation rather than a post-hoc certificate? [27][28][39][29]
• If classical generalization theory is more predictive for foundation models than for conventional deep networks, is the theory-failure diagnosis architecture-specific — and does that change the urgency or direction of the theory-building program? [30]
• The NeurIPS 2025 dynamical decoupling result establishes that generalization and overfitting are separable phenomena in large two-layer networks. Does this separation persist in deeper networks, and if so, does it support or complicate the learning mechanics program's focus on aggregate training dynamics? [1][4][24][2][3][25][5]
• Benign overfitting results show that interpolating classifiers can generalize under certain geometric conditions, but those conditions are typically verified only in simplified settings. Does benign overfitting theory scale to the overparameterized, multi-layer, non-linear regime of modern LLMs? [40][41][42][43]