Lectures

Stat 154/254: Statistical Machine Learning

Lectures marked with \(\star\) are still in progress.

Introduction

• Class introduction
• Key concepts

Unit 1: Regression in a machine learning setting

• Linear regression in matrix notation (review)
• Linear regression in an ML setting
• Basis expansions and splines for linear regression
• L1 and L2 penalization

Unit 2: Classification

• Empirical risk minimization for classification
  • Perceptron algorithm (extra, not covered in class)
• Trading off false positives and negatives

Unit 3: Risk and complexity

• Regret decomposition
• Asymptotics and bias–variance tradeoff in linear regression
• Generalization bounds
  • Generalization error and uniform laws for smooth function classes
  • VC dimension and zero–one loss
• Cross validation
  • Held-out data and cross-validation
  • Simulated bias–variance tradeoff example
  • Cross–validation cautionary note

Unit 4: Trees and weak learners

• Classification and regression trees
• Bagging weak learners
• Boosting weak learners

Unit 5: Neural networks and optimization

This unit will mostly not have its own notes, and will instead be based on readings from Prince (2023), which is freely available online.

• Black-box optimization
• Shallow and deep neural networks (Prince (2023) chapters 3 and 4)
• Optimization and generalization in neural networks (Prince (2023) chapters, 6, 9)
• Architecture as structure (Prince (2023) chapters 10, 11, and 12)
• Simulation-based inference and TabPFN

Prerequisites

Mathematical statistics, probability, linear algebra, multivariate calculus, and Python are all prerequisites.

For assistance with mathematics prerequisites, the Student Learning Center provides additional assistance.

A summary of the required linear algebra skills can be found here.

References

Prince, Simon J. D. 2023. Understanding Deep Learning. The MIT Press. http://udlbook.com.