Lab 7 Exercise - Proxy Losses, Logistic Regression and ROC Curves

Author

Josh Davis/Lucas Schwengber

Learning goals

By the end of this lab you should be able to:

  1. State the Bayes classifier for a weighted zero-one loss: predict 1 when \(p(\boldsymbol{x}) = \mathbb{P}_{\,}\left(y = 1 \mid \boldsymbol{x}\right) > c_{\text{FP}} / (c_{\text{FP}} + c_{\text{FN}})\).
  2. Explain why we minimize a proxy loss instead of the zero-one loss.
  3. Explain why proxy losses such as the log loss are minimized by \(p(\boldsymbol{x}) = \mathbb{E}_{\vphantom{}}\left[y \mid \boldsymbol{x}\right]\), and that logistic regression is ERM with the log loss.
  4. Explain how LDA estimates \(p(\boldsymbol{x})\) generatively, and when it differs from logistic regression.
  5. Choose a threshold from the costs of the two kinds of mistakes.
  6. Use an ROC curve to detect a classifier that looks accurate but ignores a rare class.
Part Learning goals
1. Zero-one loss vs. proxy losses 1, 2, 3
2. Logistic regression vs. LDA 4
3. Thresholds and costs 1, 5
4. ROC curves and rare classes 6
ImportantHow to read this lab

Everything you have to do is in a red TODO box like this one. Text outside the boxes is background you need to understand the TODOs.

Each TODO says in words what to do, followed by functions that may be useful. In your notebook, put each TODO under a markdown heading with its number, for example ### TODO 1.2, and write your answers to the questions in a markdown cell under their TODO.

Setup

ImportantTODO (Setup)

TODO 0.1. Create a new notebook, and copy and run the code below. It downloads simulator.py and imports everything you need.

import urllib.request
import numpy as np
import matplotlib.pyplot as plt
from scipy.special import expit
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.dummy import DummyClassifier
from sklearn.inspection import DecisionBoundaryDisplay
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, roc_curve, roc_auc_score
from sklearn.model_selection import KFold, cross_val_predict, cross_val_score

base_url = "https://stat154.berkeley.edu/fall-2026/datasets/classification_lab"
urllib.request.urlretrieve(f"{base_url}/simulator.py", "simulator.py")

from simulator import PracticeWorld

As in Lab 4, you work in simulated practice worlds with two features, where we know the true probability \(p(\boldsymbol{x})\). PracticeWorld(scenario, prior=0.5) creates a world in which \(\mathbb{P}_{\,}\left(y = 1\right)\) equals prior:

  • "shared": class 0 is \(N(\boldsymbol{\mu}_0, \boldsymbol{\Sigma})\) and class 1 is \(N(\boldsymbol{\mu}_1, \boldsymbol{\Sigma})\);
  • "far_cluster": the same, except that 10% of class 1 comes from a small cluster far away.

If world = PracticeWorld(...), then world.simulate(n, seed) draws X, y; world.true_prob(X) gives \(p(\boldsymbol{x})\); world.true_risk(model) gives the true log loss of a fitted model (loss="error" for its misclassification rate); world.bayes_risk() gives the smallest possible risk; and world.plot_true_boundary(ax, level=0.5) draws \(\{\boldsymbol{x}: p(\boldsymbol{x}) = \text{level}\}\) on a plot of the two features.

To draw where a fitted model’s probability equals t, use DecisionBoundaryDisplay.from_estimator(model, X, response_method="predict_proba", plot_method="contour", levels=[t], ax=ax).

Throughout the lab:

  • use LogisticRegression(C=np.inf), i.e. logistic regression without the default ridge penalty;
  • whenever you compare models or choose a hyperparameter, use 5-fold CV with KFold(5, shuffle=True, random_state=0), since in real life we do not have world.true_risk.

1 Zero-one loss vs. proxy losses

Logistic regression fits \(q(\boldsymbol{x}) = \mathrm{Expit}\left(\beta_0 + \boldsymbol{x}^\intercal\boldsymbol{\beta}\right)\) by minimizing the training log loss. Here you perturb its fitted coefficients and compare how the zero-one loss and the proxy losses respond.

ImportantTODO (Part 1)

TODO 1.1. Create the "shared" world, simulate 200 observations with seed 0, fit logistic regression, and print \(\hat\beta_0\) and \(\hat{\boldsymbol{\beta}}\).

TODO 1.2. For each \(\delta\) in np.linspace(-3, 3, 601), compute \(q(\boldsymbol{x}_n) = \mathrm{Expit}\left(\hat\beta_0 + \delta + \boldsymbol{x}_n^\intercal\hat{\boldsymbol{\beta}}\right)\) on the training set and three training losses: the zero-one loss of \(\mathrm{I}\left(q> 0.5\right)\), the log loss, and the squared loss \(\frac1N \sum_n (q(\boldsymbol{x}_n) - y_n)^2\). Plot them against \(\delta\).

May be useful: expit, @, np.clip(q, 1e-12, 1 - 1e-12) before taking logs.

TODO 1.3. For each \(s\) in np.linspace(0.1, 4, 400), compute \(q(\boldsymbol{x}_n) = \mathrm{Expit}\left(s(\hat\beta_0 + \boldsymbol{x}_n^\intercal\hat{\boldsymbol{\beta}})\right)\) and plot the training zero-one loss and log loss against \(s\).

TODO 1.4. At a single \(\boldsymbol{x}\) with \(p = p(\boldsymbol{x}) = 0.8\), the expected losses of a prediction \(q\) are \(-p \log q- (1 - p) \log(1 - q)\) (log), \(p(1 - q)^2 + (1 - p)q^2\) (squared), and \(p\) if \(q\le 0.5\), \(1 - p\) otherwise (zero-one). Plot them against \(q\) in np.linspace(0.005, 0.995, 500).

TODO 1.5. Answer in a markdown cell:

  1. Why can we not minimize the training zero-one loss with gradient descent (TODO 1.2), and what does TODO 1.3 show that the zero-one loss cannot see?
  2. In TODO 1.4, which \(q\) minimize each expected loss? Why is it a problem, if the costs of mistakes change, that the zero-one loss does not pin down \(p\)?

2 Logistic regression vs. LDA

LDA assumes each class is Gaussian with a common covariance and applies Bayes’ rule. Its log odds are linear, exactly like logistic regression, but its coefficients come from the class proportions, class means and pooled covariance rather than from minimizing the log loss (ESL Section 4.4.5).

ImportantTODO (Part 2)

TODO 2.1. On the data from Part 1, fit LDA. Print the intercepts and coefficients of both models, plot the data with the true, LDA and logistic regression boundaries, and print each model’s 5-fold CV log loss and true risk.

May be useful: LinearDiscriminantAnalysis(), cross_val_score with scoring="neg_log_loss" (it returns negative log losses).

TODO 2.2. Repeat TODO 2.1 (without printing the coefficients) on 500 observations from the "far_cluster" world with seed 0, and also print the CV and true misclassification rates.

May be useful: cross_val_score with scoring="accuracy", .true_risk(model, loss="error").

TODO 2.3. Answer in a markdown cell:

  1. In TODO 2.1 the two models have the same form. Why are their coefficients not identical?
  2. In the "far_cluster" world, which model is better? Why does the far cluster move the LDA boundary, but barely affect logistic regression?

3 Thresholds and costs

Suppose class 1 is a disease affecting 10% of patients, and a false negative costs 9 times as much as a false positive: \(c_{\text{FN}} = 9\), \(c_{\text{FP}} = 1\). The average cost of a classifier on \(N\) observations is \((c_{\text{FP}} \cdot \text{FP} + c_{\text{FN}} \cdot \text{FN}) / N\), and the Bayes classifier predicts 1 when \(p(\boldsymbol{x}) > c_{\text{FP}} / (c_{\text{FP}} + c_{\text{FN}})\) (Homework 2, Problem 1). The threshold of \(\mathrm{I}\left(q(\boldsymbol{x}) > t\right)\) is a hyperparameter, so we choose it with CV.

ImportantTODO (Part 3)

TODO 3.1. Create the "shared" world with prior=0.1, and simulate a training set of 2,000 observations with seed 0 and a test set of 20,000 with seed 1. Fit logistic regression on the training set, and compute its 5-fold CV (out-of-fold) probabilities on the training set and its probabilities on the test set.

May be useful: cross_val_predict with method="predict_proba" (the second column is the probability of class 1).

TODO 3.2. For each \(t\) in [0.02, 0.05, 0.1, 0.2, 0.3, 0.5, 0.7], compute from the CV probabilities the TPR \(= \text{TP}/(\text{TP} + \text{FN})\), the FPR \(= \text{FP}/(\text{FP} + \text{TN})\), the accuracy and the average cost. Choose the threshold with the lowest CV cost.

May be useful: confusion_matrix(y_true, y_pred).ravel(), which returns tn, fp, fn, tp.

TODO 3.3. On the test set, compute the accuracy and average cost of logistic regression with your chosen threshold and with the threshold 0.5, and of the Bayes classifier, which thresholds world.true_prob(X_test) at \(c_{\text{FP}} / (c_{\text{FP}} + c_{\text{FN}})\).

TODO 3.4. Answer in a markdown cell:

  1. Which threshold has the highest accuracy, and which the lowest cost? Why are they different?
  2. How does your chosen threshold compare with \(c_{\text{FP}} / (c_{\text{FP}} + c_{\text{FN}})\), and how close does its test cost get to the Bayes classifier’s? Why?

4 ROC curves and rare classes

The ROC curve plots the TPR against the FPR of \(\mathrm{I}\left(q(\boldsymbol{x}) > t\right)\) for every threshold \(t\).

ImportantTODO (Part 4)

TODO 4.1. For each prior in [0.5, 0.2, 0.05, 0.02], create the "shared" world, simulate 2,000 training observations with seed 0 and 20,000 test observations with seed 1, and fit logistic regression. On the test set, compute the accuracy, TPR and FPR at threshold 0.5, the accuracy of always predicting 0, and the AUC.

May be useful: roc_auc_score(y_true, q).

TODO 4.2. On one plot, draw the test ROC curves for prior=0.5 and prior=0.02, and mark the point at threshold 0.5 on each. For prior=0.02, add the ROC curve of DummyClassifier(strategy="prior") and a marker at \((0, 0)\), the classifier that always predicts 0.

May be useful: roc_curve(y_true, q), which returns fpr, tpr, thresholds.

TODO 4.3. Answer in a markdown cell:

  1. As class 1 becomes rarer, what happens to the accuracy and the TPR at threshold 0.5? Would accuracy alone warn you that something is wrong?
  2. How would you use an ROC curve, together with a classifier’s operating point, to tell a classifier that ignores the rare class (like logistic regression at threshold 0.5 with prior=0.02) from one with no signal at all (like the dummy classifier)?

Submission

ImportantTODO (Submission)

Upload your notebook, with every TODO in Parts 1-4 done and every question answered, on Gradescope by Tuesday at 9 PM. The grading will be based on completion.

LLM usage policy

You may use auto-completion for filling out small code snippets (affect around 2-3 lines of code) and documenting your code. Small code snippets are things like “add a label for the axis of the plot”, “change the color of the points”, “add an argument for the number of interactions in the fit”. These are small code operations which do not involve significant planning nor automated changes to the logic of the code without your supervision. All edits must be verified and accepted manually, and you should make sure you know what the code is doing.

Things that are explicitly not allowed are major edits to the code (affect > 10 lines) and ones which involve automating huge parts of the development of the logic structure of your analysis. For example, “write a function to extract features from the data”, or “run a test set validation loop and report the best model”.

Delegate to language models only things which you know exactly how you would have done it yourself or would be comfortable googling/asking for help from someone who already took the course/knows how to code, but who cannot do the learning for you.