[Deep Learning Theory Team Seminar] Understanding the mechanisms of fast hyperparameter transfer (by Nikhil Ghosh, Flatiron Institute)

Nikhil Ghosh of the Flatiron Institute presents work on scale-aware hyperparameters, the approach that lets optimal settings found by cheap grid searches on small models carry over to much larger ones with little loss in performance. As model sizes have grown, standard hyperparameter optimization has become prohibitively expensive, which makes understanding when and why this transfer works a practical concern for anyone training at scale. The talk lays out a conceptual framework for reasoning about hyperparameter transfer across scale. In synthetic settings, Ghosh gives quantitative examples on both sides of the ledger: cases where transfer offers a provable computational advantage, and cases where it fails even under μP. To explain the fast transfer seen in practice, he proposes decomposing the optimization trajectory into a width-stable component that fixes the optimal hyperparameters and a width-sensitive component that improves with width while only weakly perturbing that optimum. Empirical evidence for the hypothesis comes from large language model pretraining. The session runs for one hour and is held in hybrid format, with in-person attendance available at the University of Tokyo Hongo campus and remote participation for everyone else.