Lab 8 Exercise - Classification Competition

Author

Erez Buchweitz/Lucas Schwengber/Josh Davis

Learning goals

By the end of this lab you should be able to:

  1. Explain why accuracy is misleading when one class is rare, and what the \(F_1\) score measures instead.

  2. Estimate the test \(F_1\) of a model with cross validation, and use it to select the method, its hyperparameters and its threshold. Understand that the \(F_1\) is not the loss the models are trained with: each method is fit as usual, and the \(F_1\) is only used to compare and tune them.

  3. Explain why, with imbalanced classes, the default fit with threshold 0.5 usually gives a poor \(F_1\), and fix it by tuning the threshold or the class weights.

  4. Explain how the main hyperparameters of each tree-based method control its bias and variance.

  5. Discuss the trade-offs involved in using all the classification methods used in the lab and explain how these methods compare in a realistic task.

  6. Explain why the method you selected beat the others, in terms of the boundaries each method can fit and of bias and variance.

1 Prediction setup

Your task is predict whether a patient is diabetic (yes or no) based on various recorded metrics which are given as features. You are given three files:

You may download and read them like this:

import pandas as pd

X_train = pd.read_parquet("X_train.parquet")
y_train = pd.read_parquet("y_train.parquet")

There are two data sets, (X_train, y_train) and (X_test, y_test), which are exchangeable. However, I have hidden y_test from you. Your task is to make the best predictions of y_test possible based on the training data and on X_test.

2 Allowed methods to use

You are allowed to use all classification methods which you have learned in the lectures or labs. That is: logistic regression, linear discriminant analysis and tree-based methods (which include bagging and boosting).

Moreover, your predictions should be obtained by training and doing model selection only on the data you are provided with from the lab.

Any violations of the above policies may result in zero grade.

3 Task list

Besides trying to obtain the best set of final predictions, you should compare all the methods you have learned on this task, and be able to say why the one you chose works best.

ImportantTODO (Methods comparison)

TODO 1. Fit every allowed method at least once on the training data: logistic regression, LDA, a decision tree, a random forest and gradient boosting. For each one, estimate its \(F_1\) score.

May be useful: cross_val_score or cross_val_predict with scoring="f1", f1_score, LogisticRegression, LinearDiscriminantAnalysis, DecisionTreeClassifier, RandomForestClassifier, GradientBoostingClassifier.

TODO 2. Select your best model among model classes and hyperparameter selections in term of the expected test \(F_1\)-score. Use it to generate your final predictions.

May be useful: GridSearchCV, cross_val_predict with method="predict_proba" to try several thresholds from one set of out-of-fold probabilities.

TODO 3. In a brief markdown cell, explain why you think the selected model performed better than the other ones.

4 How to submit predictions

You will submit an .npy file containing a numpy array (one-dimensional), whose length is the same as the number of rows in X_test, that will comprise your binary predictions of y_test. Your prediction should be either value 1 if you think that patient is diabetic, or 0 otherwise. You can save an array to file like this:

import numpy as np

np.save(f"pred_{codename}.npy", pred)

The name of the file should be "pred_{codename}.npy" where {codename} is a secret codename of your group’s choosing, that you will be able to use later to identify your submission in the leaderboard. The file, alongside your code, should be submitted to GradeScope.

5 How your predictions will be scored

Your predictions will be compared against the real y_test using the \(F_1\) score:

\[ F_1(\boldsymbol{y}, \hat{\boldsymbol{y}}) = \frac{2}{\frac{1}{\text{precision}}+\frac{1}{\text{recall}}}, \; \; \; \; \text{recall} = \frac{|\{n: \boldsymbol{y}_n=\hat{\boldsymbol{y}}_n=1\}|}{|\{n: \boldsymbol{y}_n=1\}|}, \; \; \; \; \text{precision} = \frac{|\{n: \boldsymbol{y}_n=\hat{\boldsymbol{y}}_n=1\}|}{|\{n: \hat{\boldsymbol{y}}_n=1\}|} \]

A leaderboard will be published with every group’s codename alongside its \(F_1\) score on the test set, ranked from high to low. No other identifying information will be shared, so no other student will be able to know your rank on the leaderboard. The leaderboard is for your own benefit! The group with the highest test \(F_1\) score is the winner.

Submission

ImportantTODO (Submission)

Upload the two files on Gradescope by Tuesday Nov. 3rd at 9 PM:

  1. your code
  2. pred_{codename}.npy as described earlier.

The grading will be based on completion of the task.

LLM usage policy

You may use auto-completion for filling out small code snippets (affect around 2-3 lines of code) and documenting your code. Small code snippets are things like “add a label for the axis of the plot”, “change the color of the points”, “add an argument for the number of interactions in the fit”. These are small code operations which do not involve significant planning nor automated changes to the logic of the code without your supervision. All edits must be verified and accepted manually, and you should make sure you know what the code is doing.

Things that are explicitly not allowed are major edits to the code (affect > 10 lines) and ones which involve automating huge parts of the development of the logic structure of your analysis. For example, “write a function to extract features from the data”, or “run a test set validation loop and report the best model”.

Delegate to language models only things which you know exactly how you would have done it yourself or would be comfortable googling/asking for help from someone who already took the course/knows how to code, but who cannot do the learning for you.