Lab 1A: Data, Models, and Decisions

Let us pretend that we are tasked with designing a system to make admission decisions at the University of Pennsylvania (Penn). In order to design this system we need to acquire data, choose a model, and train it to make admission decisions.

To do so we leverage data on the performance of past students to construct an AI model that predicts the Penn GPA of students based on their High School GPA and their SAT scores. We then use this AI to predict the future performance of applicants based on their High School GPA and SAT scores.

Environment setup¶

Before we beign with the lab proper, we import some necessary Python packages. Python is a very pithy programming language. It can do very little on its own. Most of the functionalities that we need require that we load specific packages. For this particular lab we load Numpy, Pytorch, and matplotlib. These packages are used to load data, process data, and make plots, respectively.

There is nothing to understand in the code above. This is just knowledge that you need to have. Like what is the command in a text editor to write in boldface.

Data¶

This file contains data that we have available to make admission decisions. For a collection of former students we have access to their high school (HS) grade point average (GPA), their Scholastic Assessment Test (SAT) scores, their gender and their Penn GPA. There are a total of 600 students in this file.

Task 1. Load the GPA and SAT scrore data and plot: (i) The Penn GPA as a function of the High School GPA. (ii) The Penn GPA as a function of the SAT score.

Loading Data¶

You can corroborate that the numbers above correspond the first 5 rows of the GPA and SAT scrore data.

Penn GPA as a Function of High School GPA¶

Continuing with Task 1 we now proceed to plot Penn GPAs as a function of High School GPAs. This is done by calling a command to plot the data.

¶

In this figure the horizontal axis is the high school GPA and the vertical axis is the Penn GPA. This plot shows that high school GPA is predictive of Penn GPA. Although there is significant variation we can see that higher high school GPA corresponds with higher Penn GPA. This indicates that high school GPAs are useful information for admission decisions as they can predict with some accuracy the GPA that an admitted student may attain at Penn.

Penn GPA as a Function of SAT Scores¶

To finalize Task 1 we plot Penn GPAs as a function of SAT scores. This is more or less the same code that wrote for plotting Penn GPAs with respect to High School GPAs. It may be good exercise to do this part without looking at this solution but looking at the previous one instead.

¶

In this figure the horizontal axis is the SAT score and the vertical axis is the Penn GPA. The predictive power here is weaker than the predictive power of High School GPA. There ir, nonetheless, useful information here that we can leverage to design an AI that predicts Penn GPA of admission candidates.

System¶

To make admission decisions we interpret Penn as a system that takes as inputs high school GPA, SAT scores and gender information of a student and produces as an output the Penn GPA of the corresponding graduate. If we define the input data as a vector $x = [\text{HS GPA}; \text{SAT}; \text{Gender}]$ and we denote the output as $y = \text{Penn GPA}$ we can represent this system as the function,

\begin{equation}\tag{1} \text{Penn GPA} ~=~ y ~=~ P(x) ~=~ P \left[ \begin{array}{l} \text{HS GPA} \\ \text{SAT} \\ \text{Gender} \end{array} \right] \ . \end{equation}

Notice that this is a poor representation of Penn. Incoming students are much more than their gender, High School GPA, and SAT scores. Penn graduates are much more than their Penn GPAs and the institution itself does much more to a high school graduate than transforming their High School GPA and SAT scores into a Penn GPA. This is just one aspect of the whole system on which we are choosing to focus. The distinction between what a system is and what an engineer chooses to say that a system is warrants some discussion that we undertake in Section 4 of the lab assgignment.

Model and Decisions¶

To make admission decisions we design an AI model that predicts the Penn GPA of prospective students.

To make matters simpler let us begin by ignoring SAT scores and gender and attempt predictions based on high school GPAs. This means that the system above is replaced by the system

\begin{equation}\tag{2} \text{Penn GPA} ~=~ y ~=~ P(x) ~=~ P ( \, \text{HS GPA} \, ) . \end{equation}

The function $P(x)$ is the true effect of Penn on scholastic accomplishment. This is information that becomes available after the fact. When a student graduates Penn, we have access to their high school GPA $x$ and their Penn GPA $y$.

Penn GPA predictions are to be made prior the fact. Before a student attends Penn we want to estimate their Penn GPA based on their high school GPA $x$. We choose to postulate a linear relationship and make predictions of the form

\begin{equation}\tag{3} \hat{y} = \alpha x. \end{equation}

In this equation, $\hat{y}$ is a prediction of the true Penn GPA $y$ that will be available after the fact. The coefficient $\alpha$ is to be determined with the goal of making predictions $\hat{y}$ close to to actual Penn GPA $y$. We can then use Penn GPA predictions to make admission decisions.

Least Squares Estimation¶

To determine a proper value for the coefficient $\alpha$ in (3) we utilize the data we have available on the scholastic performance of past students. Use $N$ to denote the total number of available data points. Introduce a subindex $i$ to differentiate past students so that the pair $(x_i, y_i)$ denotes the high school GPA and Penn GPA os student $i$. For these students we can make GPA predictions $\hat{y}_i = \alpha x_i$. For a given coefficient $\alpha$ we define the mean squared error (MSE),

\begin{equation}\tag{4} \text{MSE}(\alpha) ~=~ \frac{1}{N} \sum_{i=1}^{N} \Big( y_i – \hat{y}_i \Big)^2 ~=~ \frac{1}{N} \sum_{i=1}^{N} \Big( y_i – \alpha x_i \Big)^2 . \end{equation}

The mean squared error $\text{MSE}(\alpha)$ measures the predictive power of coefficient $\alpha$. The quantity $( y_i – \hat{y}_i )^2$ is always nonnegative and indicates good the predicted GPA $\hat{y}_i$ is to the true GPA $y_i$. The MSE averages this metric over all students. It follows that a natural choice for $\alpha$ is the value that makes the MSE smallest. We therefore define the optimal coefficient

\begin{equation}\tag{5} \alpha^* ~=~ \text{argmin}_\alpha \frac{1}{2} \text{MSE}(\alpha) ~=~ \text{argmin}_\alpha \frac{1}{2N} \sum_{i=1}^{N} \Big( y_i – \alpha x_i \Big)^2 , \end{equation}

and proceed to make Penn GPA predictions as $\hat{y} = \alpha^* x$. This GPA predictor is called the linear minimum mean squared error (MMSE) prediction. This is because the predictor is the linear function that minimizes the MSE.

Task 2 Prove that the MMSE estimator coefficient $\alpha^*$ defined in (5) is given by the expression

\begin{equation}\tag{6} \alpha^* = \sum_{i=1}^{N} x_i y_i ~\bigg/~ \sum_{i=1}^{N} x_i^2 . \end{equation}

Compute $\alpha^*$ for the data loaded in Task 1. Plot the Penn GPA with respect to HS GPA and superimpose the prediction line $\hat{y} = \alpha^* x$.

In Task 2 we make predictions of the Penn GPA of students that have graduated Penn. Predicting Penn GPAs of past students is unnecessary given that we already know their true GPAs. Our motivation for solving this unnecessary problem is to determine the coefficient $\alpha^*$ that we can use to make predictions $\hat{y} = \alpha^* x$ of students that have not yet attended Penn — for which $x$ is available but $y$ is not. The effectiveness of this prediction depends on the extent to which the past is a good representation of the future.

It is germane to emphasize that in Task 2 we are using something we know — the GPA of former students — to answer a new question — the GPA of a prospective student. However primitive, this is a form of intelligence.

Root Mean Squared Error¶

We evaluate the merit of $\alpha^*$ with the root mean squared error (RMSE)

\begin{equation}\tag{7} \text{MSE}(\alpha) ~=~ \sqrt { \text{MSE}(\alpha) } ~=~ \Bigg [\, \frac{1}{N} \sum_{i=1}^{N} \Big( y_i – \alpha x_i \Big)^2 \, \Bigg]^{1/2}. \end{equation}

The reason we use the RMSE to evaluate the merit of $\alpha^*$ instead of the MSE is that the RMSE has the same units as our target variable. It is easier to interpret than the MSE. Since the difference between the two is just a square root function, the coefficient $\alpha^*$ that minimizes the MSE also minimizes the RMSE.

Task 3. Compute the RMSE of $\alpha^*$ and comment on the quality of the Penn GPA predictions.

The RMSE is 0.094. We can think of this number as the accuracy of our Penn GPA predictions. This number seems to imply that our Penn GPA predictions are quite accurate because Penn GPAs can range from 0 to 4. However, the actual range of Penn GPAs observed in the dataset is between 3.3 and 4.0. In a variable whose range spans 0.7 units, a prediction error of 0.094 is not very accurate. This is apparent in the Figure we prepared in response to Task 2 where the line of predicted Penn GPAs is a rough estimate of observed Penn GPAs.

Linear Regression¶

We consider now a more complete model in which the Penn GPA is deemed to depend on the high school GPA and the SAT score. We therefore define the input vector $x = [x_1; x_2] = [\text{HS GPA}; \text{SAT}; \text{Gender}]$

\begin{equation}\tag{8} \text{Penn GPA} ~=~ y ~=~ P(x) ~=~ P \left[ \begin{array}{l} x_1 \\ x_2 \\ \end{array} \right] \ . ~=~ P \left[ \begin{array}{l} \text{HS GPA} \\ \text{SAT} \\ \end{array} \right] \ . \end{equation}

To make Penn GPA predictions we postulate an input-output relationship of the form

\begin{equation}\tag{9} \hat{y} ~=~ w_1 x_1 + w_2 x_2 ~=~ w^T x . \end{equation}

As is the case of (3), $\hat{y}$ in (9) is a prediction of the true Penn GPA $y$ that will be available after the fact. The coefficient $w=[w_1; w_2]$ is to be determined with the goal of making predictions $\hat{y}$ close to to actual Penn GPA $y$.

Task 4 Define and compute the MMSE estimator coefficients $w^*$, that would extend the MMSE definition in (5). Show that this coefficient is given by the expression

\begin{equation}\tag{10} w^* = \Bigg[ \sum_{i=1}^{N} x_i x_i^T \Bigg]^{-1} \sum_{i=1}^{N} x_i y_i . \end{equation}

In matrix form, this is equivalent to

\begin{equation} \Bigg[ x^T x \Bigg]^{-1} \cdot x^Ty \end{equation}

Compute $w^*$ for the data loaded in Task 1. Compute the RMSE of $w^*$ and comment on the quality of the Penn GPA predictions.