Version 10

Narrative

Timeline

• 2016-01-01: Zhang et al. demonstrate that standard neural networks can memorize completely random labels on CIFAR-10 and ImageNet, invalidating data-independent generalization bounds. [23]
• 2019-01-01: Nagarajan and Kolter show empirically that spectral-norm bounds scale in the wrong direction, and prove formally in an overparameterized linear setting that uniform convergence is provably insufficient to explain gradient descent generalization. [22][145][146][155][156]
• 2019-06-01: Feldman publishes 'Does Learning Require Memorization? A Short Tale about a Long Tail,' arguing memorization of tail examples is causally necessary for learning from long-tailed distributions. [64][69][157][70][71][72][73][81]
• 2020-07-01: Negrea et al. publish 'In Defense of Uniform Convergence' at ICML 2020, arguing that uniform convergence can be partially recovered through derandomization applied to interpolating classifiers. [85][86][87]
• 2020-11-01: 'Disentangling Feature and Lazy Training in Deep Neural Networks' provides a formal framework for distinguishing when networks operate in the NTK (lazy) regime versus learning new representations (feature learning), directly engaging the axis that learning mechanics treats as the central diagnostic for whether classical theory applies. [138]
• 2021-01-01: Yang et al. establish an exact algebraic gap between generalization error and the tightest possible uniform convergence bound in random feature models, giving Nagarajan-Kolter a precise formal complement. [88][89]
• 2021-01-01: ACM Communications publishes the canonical journal version of Zhang et al.'s rethinking-generalization result, cementing it as a textbook-permanent finding rather than a contested empirical claim. [158]
• 2022-01-01: A NeurIPS 2022 paper claims PAC-Bayes compression bounds can be made tight enough to actually explain generalization in neural networks, directly challenging the narrative that all known bounds are vacuous. [104][105][106]
• 2022-01-01: NeurIPS 2022 paper 'Towards Understanding Grokking: An Effective Theory of Representation Learning' provides an effective-theory account of delayed generalization, framing grokking as a consequence of representation learning dynamics. [143]
• 2022-01-01: Tirumala et al. publish 'Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models' (NeurIPS 2022), finding that memorization in LLMs peaks early in training and subsequently decreases while generalization continues to improve — a temporal separation that echoes grokking's sequential structure in the large-scale pretraining regime. [17]
• 2023-07-01: Wu et al. publish 'The Implicit Regularization of Dynamical Stability in SGD' at ICML 2023, showing that SGD's dynamical stability provides an implicit regularizer that suppresses memorization — bridging the algorithmic stability and implicit regularization research strands. [90][91]
• 2024-01-01: NeurIPS 2024 paper on symmetries in overparameterized neural networks using a mean-field view offers a new structural lens on why overparameterization does not prevent generalization. [159][160]
• 2024-07-01: u-μP: The Unit-Scaled Maximal Update Parametrization (arxiv 2407.17465) refines the μP framework by incorporating unit scaling, representing continued active development of learning mechanics' main practical engineering artifact. [142]
• 2024-10-01: 'Mitigating Memorization in Language Models' (arxiv 2410.02159) proposes engineering techniques to reduce memorization in deployed LLMs — treating memorization as a practical harm to suppress rather than a theoretically necessary property, implicitly opposing Feldman's necessity thesis in the LLM regime. [12][13]
• 2025-01-01: CMU PhD blog post argues classical generalization theory is more predictive for foundation models than for conventional deep networks, implying the theory-failure narrative may not apply uniformly across architectures and scales. [161]
• 2025-01-01: Urbani delivers a seminar at the Stanford Mathematics department on generalization in two-layer neural networks, confirming the Montanari-Urbani result is circulating in pure-math venues beyond ML conferences. [58]
• 2025-01-01: ICLR 2025 paper 'When Memorization Hurts Generalization' argues memorization can actively damage generalization performance — a stronger claim than merely saying memorization is unnecessary. [63][82][83][84]
• 2025-01-01: NeurIPS 2025 Oral 'Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networks' (Montanari and Urbani) presents results separating generalization dynamics from overfitting dynamics in the large two-layer regime; oral status confirmed, placing it in the top ~1-2% of NeurIPS 2025 submissions. [45][46][47][48][49][50][51][53][54][55][56][57]
• 2025-01-01: NeurIPS 2025 Best Paper 'Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Training' — confirmed via the official NeurIPS Best Paper Awards blog post — attributes diffusion models' resistance to memorization to implicit dynamical regularization; this is now the most institutionally recognized result in the training-dynamics cluster. [92][93][94][95][96][97][98][99][100][2][101][102][103][1]
• 2025-01-01: ICML 2025 poster 'Rethinking Benign Overfitting in Two-Layer Neural Networks' revisits the conditions under which interpolating classifiers can generalize in the two-layer setting. [147]
• 2025-01-01: ICLR 2025 poster 'Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data' extends the memorization-generalization debate to LLMs, examining how specific pretraining data drives capability versus memorization. [133][134][135][137][132]
• 2025-01-01: 'Necessary Memorization in Overparameterized Learning under Long-Tailed Mixture Models: Theory and Privacy Implications' provides formal theoretical support for Feldman's necessity claim in a specific mathematical setting, and introduces differential privacy leakage as a downstream consequence of memorization. [59]
• 2025-01-01: NeurIPS 2025 poster 'Understanding the Evolution of the Neural Tangent Kernel' tracks how the NTK changes during training rather than treating it as a fixed linearization, engaging the 'lazy training vs. feature learning' distinction central to the learning mechanics critique of classical theory. [33]
• 2025-01-01: 'Reactivation: Empirical NTK Dynamics Under Task Shifts' empirically tracks NTK dynamics as tasks change, connecting the NTK evolution program to the continual learning setting. [139]
• 2025-01-01: 'Generative Modeling of Weights: Generalization or Memorization?' opens a new subdomain within the memorization debate: whether generative models of neural network weight distributions can generalize to unseen architectures or merely memorize training-set weight configurations. The paper has acquired dissemination across HuggingFace, a project page, and Emergent Mind, indicating growing community attention. [144][162][163][164][165][166][167]
• 2025-01-01: A Workshop on Mechanistic Interpretability appears in the NeurIPS 2025 virtual infrastructure (the URL routes through the NeurIPS 2025 system, though the page title references 2026), marking the field's confirmed presence at NeurIPS alongside the separately-tracked ICML 2026 workshop. [3]
• 2025-01-01: NeurIPS 2025 poster 'A Closer Look at NTK Alignment: Linking Phase Transitions in Deep Image Regression' directly connects NTK alignment dynamics to phase transitions, providing a new bridge between the NTK evolution program and the grokking/phase-transition research strand. [7]
• 2025-01-01: An 'Actionable Interpretability' workshop at ICML 2025 adds a second distinct interpretability-focused workshop slot to the same conference, representing a parallel institutional track to the mechanistic interpretability workshop program. [5]
• 2025-01-01: ICML 2025 poster 'Adaptive kernel predictors from feature-learning infinite limits of neural networks' suggests the infinite-width limit frameworks underlying the NTK program can accommodate feature learning, potentially bridging the lazy-training and feature-learning regimes. [10]
• 2025-02-01: 'Pruning as a Defense: Reducing Memorization in Large Language Models' (arxiv 2502.15796) demonstrates that model pruning can reduce LLM memorization as a practical defense — extending the engineering memorization mitigation cluster to compression-based techniques. [14][15]
• 2025-02-19: Pierfrancesco Urbani (CNRS) delivers an invited talk at the Simons Institute (Berkeley) on 'Generalization and overfitting in two-layer neural networks,' extending the dynamical decoupling result's reach to a major theory venue. [52]
• 2025-07-01: 'Grokking Beyond the Euclidean Norm of Model Parameters' (ICML 2025 poster) challenges the L2-norm-based account of the memorization-to-generalization transition in grokking, suggesting the phase transition involves dynamics not fully captured by weight norm trajectories. [8]
• 2025-07-01: 'Understanding the Evolution of the Neural Tangent Kernel at the Edge of Stability' (arxiv 2507.12837) extends NTK dynamics analysis to the edge-of-stability training regime — where gradient descent operates near sharpness boundaries with large stable learning rates — connecting NTK evolution to the separately active catapult/sharpness-reduction theoretical program. [6]
• 2025-10-01: 'Memorizing Long-tail Data Can Help Generalization Through Composition' (arxiv 2510.16322) proposes that the mechanism by which tail memorization aids generalization is compositional rather than coverage-based, offering a mechanistic refinement of Feldman's thesis. [60][74]
• 2026-01-01: MIT names mechanistic interpretability as its 2026 Breakthrough of the Year, institutionally recognizing circuit-level understanding as a viable alternative program to statistical generalization theory for understanding deep learning. [117][121][122][168]
• 2026-01-01: A dedicated ICML 2026 workshop on mechanistic interpretability is announced by Neel Nanda (DeepMind) on LinkedIn and X/Twitter, with infrastructure confirmed via Buttondown newsletter, official schedule page, OpenReview submission portal, and ICML Blog; and an 'Open problems in mechanistic interpretability: 2026 status report' is published as a GitHub gist. [123][124][126][127][4][128][169][170][129][171][172][116][115]
• 2026-02-01: 'Beyond Explainable AI (XAI): An Overdue Paradigm Shift and Post-XAI Research Directions' (arxiv 2602.24176) argues the XAI paradigm has exhausted itself and a new framework is needed — introducing a third interpretability-adjacent pressure on deep learning theory, distinct from both statistical generalization theory and circuit-level mechanistic interpretability, from the applied AI accountability side. [19]
• 2026-03-01: MIT News covers research on improving AI models' ability to explain their predictions, marking science-communication investment in interpretability research that the statistical generalization theory program has not received. [125]
• 2026-04-01: 'Grokking as Dimensional Phase Transition in Neural Networks' (arxiv 2604.04655) introduces a new theoretical framework for delayed generalization, framing the memorization-to-generalization transition as a phase transition in the dimensionality of learned representations. [9]
• 2026-04-01: 'Training Data Pruning Improves Memorization of Facts' (arxiv 2604.08519) presents a counterintuitive finding: selective reduction of training data can improve how well models memorize individual facts, suggesting data redundancy may suppress rather than reinforce fact memorization. [75][173][174][175]
• 2026-04-24: Jamie Simon and Daniel Kunin (UC Berkeley) appear on Imbue's podcast arguing that a scientific theory of deep learning is achievable, marking the first direct public advocacy by the learning mechanics authors. [29][34]
• 2026-04-25: LawrenceC (confirmed as Lawrence Chan via Google Scholar) publishes a critical review of Simon et al.'s 'learning mechanics' manifesto on the Alignment Forum, welcoming its ambition while doubting it will deliver a comprehensive or broadly useful theory. [24][20]
• 2026-04-26: LawrenceC publishes 'The paper that killed deep learning theory,' providing detailed technical and historical context for why Zhang et al. 2016 was so devastating to the classical generalization-bound paradigm. [23][26]
• 2026-04-27: LawrenceC publishes 'The other paper that killed deep learning theory,' narrating Nagarajan and Kolter 2019 as the definitive proof that uniform convergence cannot explain neural network generalization; the post is crossposted to LessWrong and drives renewed interest in the original paper. [22][25][27]
• 2026-04-28: 'There Will Be a Scientific Theory of Deep Learning' (arxiv 2604.21691) appears as a late-April 2026 preprint offering the first formal academic-paper-level optimistic response to the thread's founding pessimism, its title directly inverting LawrenceC's framing. The paper has accumulated a ResearchGate page indicating cross-platform indexing. [35][37][21]
• 2026-04-30: At least three distinct OpenReview submissions titled 'Is Memorization Actually Necessary for Generalization?' appear, representing independent formal challenges to Feldman's affirmative claim. [65][61][62][66][67]
• 2026-05-01: A LessWrong 'Quick Paper Review' of 'There Will Be a Scientific Theory of Deep Learning' appears on the same platform as LawrenceC's original critique posts, and a Reddit r/MachineLearning thread opens discussion of the preprint — marking its transition from indexed to actively engaged within the communities that originally amplified the pessimistic framing. [42][43]

Perspectives

Tensions

• Can learning mechanics, which focuses on average-case training dynamics and coarse aggregate statistics, ever constitute a comprehensive theory of deep learning — or is it structurally limited to explaining some aspects while leaving others permanently outside its scope? The April 2026 preprint 'There Will Be a Scientific Theory of Deep Learning' argues the question is not closed and has active community engagement in the LessWrong venue where LawrenceC's pessimism originated. The NeurIPS 2025 Best Paper for implicit dynamical regularization provides indirect empirical warrant for the learning mechanics framing, and the NTK edge-of-stability and phase-transition papers provide further external convergence on dynamics-first explanations. Whether these constitute supporting evidence for learning mechanics' feature-learning framing or constitute separate theoretical lines remains unresolved. [24][29][30][31][32][44][35][42][43][33][138][139][142][6][7]
• Is memorization causally necessary, causally harmful, or merely correlated with generalization — and does the answer change across training epochs, architectures, task settings, and data regimes? Feldman's necessity claim is being formalized under long-tailed mixture models with a compositional mechanism account, while 'When Memorization Hurts Generalization' claims the opposite, and applied LLM mitigation work frames memorization as 'undesirable' without engaging the formal necessity arguments. Tirumala et al.'s 'Memorization Without Overfitting' shows temporal separation between memorization and generalization in LLMs, echoing grokking but in pretraining. These positions may not be contradictory if memorization's effects depend on training epoch, regime, and task structure — but no synthesis has yet been attempted. [64][65][61][62][63][66][67][82][83][84][59][60][74][143][9][75][76][144][14][17][15][12][13][16]
• Yang et al. 2021 establish an exact algebraic gap between generalization error and uniform convergence in random feature models. Negrea et al.'s ICML 2020 'In Defense of Uniform Convergence' argues the framework can be salvaged through derandomization applied to interpolating classifiers. Do derandomized uniform convergence arguments fall within the class foreclosed by the exact gap result, or do they constitute a genuine escape hatch that keeps the classical program viable? [22][88][89][145][146][85][86][87]
• The NeurIPS 2025 dynamical decoupling result establishes that generalization and overfitting are separable phenomena in large two-layer networks. Does this separation persist in deeper networks and in transformer architectures, and if so, does it support or complicate the learning mechanics program's focus on aggregate training dynamics? [45][46][48][51][52][53][54][55][56][58]
• If implicit dynamical regularization explains why diffusion models don't memorize (NeurIPS 2025 Best Paper), and Wu et al.'s SGD dynamical stability result provides a parallel mechanism, is there a unified dynamical account of memorization suppression across architectures? Or does the memorization debate remain architecture-regime-specific, with separate mechanisms governing transformers, diffusion models, and two-layer networks? The NTK edge-of-stability paper[6] raises whether the same dynamical regularization account holds in high-sharpness training regimes. [92][93][94][95][100][2][90][136][133][147][139][6]
• PAC-Bayes compression bounds are claimed to be tight enough to explain generalization, while Yang et al.'s exact gap result says uniform convergence-style arguments are provably incapable of capturing the correct quantity. Are PAC-Bayes compression bounds a genuine escape hatch from the Nagarajan-Kolter impossibility, or do they fall within the class of arguments the exact gap result forecloses? [104][105][148][106][88][89]
• Benign overfitting results show that interpolating classifiers can generalize under certain geometric conditions. ICML 2025's 'Rethinking Benign Overfitting in Two-Layer Neural Networks' revisits these conditions — do they hold broadly enough to constitute a useful theory, or do they require fine-tuned assumptions that fail in realistic multi-layer, non-linear settings? [149][150][151][152][147][153][154]
• Is mechanistic interpretability a practical alternative to statistical generalization theory for understanding deep learning, or does its circuit-level focus simply answer a different question? The field now has multi-conference workshop infrastructure (ICML 2026, NeurIPS workshop presence, ICML 2025 'Actionable Interpretability') and a named advocate (Neel Nanda) who spans safety, alignment, and academic ML — but institutional success is not the same as theoretical displacement, and the mechanistic interpretability and generalization theory programs have not yet engaged each other directly. The 'Beyond XAI' paradigm-shift argument introduces a third pressure from the applied accountability side, further fragmenting the interpretability landscape. [117][118][119][120][24][29][123][124][121][4][129][116][115][3][5][19]
• The grokking sub-debate now has at least four competing accounts — norm-based, dimensional phase transition, effective theory of representation learning, and compositional mechanism — that have not been formally reconciled. 'Grokking Beyond the Euclidean Norm' challenges the norm account without proposing a replacement. The new NTK alignment/phase-transition paper[7] provides an external bridge between NTK dynamics and phase transitions that the grokking literature has not yet incorporated. Is grokking a single phenomenon admitting a unified theory, or a family of distinct delayed-generalization transitions that happen to share a surface phenomenology? [8][9][143][60][74][7]
• The NTK evolution program now spans multiple fronts — edge of stability[6], phase transitions[7], adaptive kernel predictors from feature-learning limits[10], empirical task-shift dynamics[139], and physics-journal venues[11] — but has not yet produced a unified account of how these strands relate. Does the NTK evolution program converge toward the feature-learning regime that learning mechanics treats as operative in practice, or does it remain a set of disconnected extensions of the lazy-training regime? The adaptive kernel predictors paper[10] suggests the former is possible but has not yet been demonstrated. [33][138][139][11][10][6][7]