Version 1

The thread is inaugurated by LawrenceC's three-part series on the Alignment Forum (April 25–27, 2026) reconstructing the collapse of classical deep learning theory and appraising the uncertain prospects for its replacement. The series opens with a critical review of Simon et al.'s recent manifesto proposing "learning mechanics" — a theory of training dynamics focused on coarse aggregate statistics and average-case predictions — as the embryo of a proper scientific theory of deep learning.[1] LawrenceC appreciates the paper's clarity but remains skeptical: the main practical output of learning mechanics research so far has been hyperparameter scaling techniques like mu-parameterization, and the field has 'yielded little practical fruit' beyond retrofitting known empirical phenomena.[1] Crucially, learning mechanics explicitly disclaims explaining the specific algorithms learned by particular networks, which LawrenceC argues disqualifies it as a complete theory even if it succeeds on its own terms.[1]

The series then turns historical, reconstructing why theory needed reinvention in the first place. Zhang et al.'s 2016 result — that standard neural networks can memorize completely random labels on CIFAR-10 and ImageNet, achieving near-zero training loss — demolished the classical paradigm.[2] Because the same architecture and training algorithm can either generalize or memorize depending solely on label correctness, any data-independent complexity measure (VC dimension, Rademacher complexity) is incapable of explaining generalization; the hypothesis class is too rich by every classical metric.[2] Regularization techniques including data augmentation, weight decay, and dropout had minimal effect on memorization capacity, further undermining norm-based explanations, and memorization required only 1.5–3.5x more training steps than true-label learning — meaning latent memorization capacity is always present.[2]

The third post covers Nagarajan and Kolter's 2019 result, which LawrenceC frames as the final nail in the coffin of the uniform-convergence approach: spectral-norm bounds scaled in the empirically wrong direction (worsening as training data increased while test error fell), and the paper proved formally in an overparameterized linear setting that uniform convergence bounds must be vacuous — making the entire statistical learning theory apparatus provably insufficient for explaining gradient descent generalization.[3] The constructive diagnosis is structural: SGD produces classifiers microscopically complex near training points but macroscopically simple near unseen data, simultaneously accounting for good generalization and the failure of worst-case bounds.[3] LawrenceC closes with a challenge to the community: 'almost a decade later, we still don't have those results.'

The discourse is currently shaped entirely by one voice writing in a single venue over three consecutive days. LawrenceC's tone is controlled pessimism — acknowledging that a partial theory like learning mechanics may crystallize while doubting it will be comprehensive or broadly useful. The large volume of reactive search results (Wikipedia entries, arXiv abstracts, course slides, items 1927–1958) contain no substantive additional claims and represent no independent perspectives. The central tension the series identifies — that any valid generalization theory must be algorithm- and data-dependent in ways that all existing frameworks have failed to satisfy, nearly a decade after the problem was precisely diagnosed — remains fully open.[3][1]