Chapter 8 - Linear Regression

Learning Outcomes

  • Define the explanatory variable as the independent variable (predictor), and the response variable as the dependent variable (predicted).
  • Plot the explanatory variable (\(x\)) on the x-axis and the response variable (\(y\)) on the y-axis, and fit a linear regression model $y = \beta_0 + \beta_1 x$ where \(\beta_0\) is the intercept, and \(\beta_1\) is the slope.
    • Note that the point estimates (estimated from observed data) for \(\beta_0\) and \(\beta_1\) are \(b_0\) and \(b_1\), respectively.
  • When describing the association between two numerical variables, evaluate
    • direction: positive (\(x \uparrow, y \uparrow\)), negative (\(x \downarrow, y \uparrow\))
    • form: linear or not
    • strength: determined by the scatter around the underlying relationship
  • Define correlation as the association between two numerical variables.
    • Note that a relationship that is nonlinear is simply called an association.
  • Note that correlation coefficient (\(r\), also called Pearson’s \(r\)) the following properties:
    • the magnitude (absolute value) of the correlation coefficient measures the strength of the linear association between two numerical variables
    • the sign of the correlation coefficient indicates the direction of association
    • the correlation coefficient is always between -1 and 1, inclusive, with -1 indicating perfect negative linear association, +1 indicating perfect positive linear association, and 0 indicating no relationship
    • the correlation coefficient is unitless
    • since the correlation coefficient is unitless, it is not affected by changes in the center or scale of either variable (such as unit conversions)
    • the correlation of X with Y is the same as of Y with X
    • the correlation coefficient is sensitive to outliers
  • Recall that correlation does not imply causation.
  • Define residual (\(e\)) as the difference between the observed (\(y\)) and predicted (\(\hat{y}\)) values of the response variable. $e_i = y_i - \hat{y}_i$
  • Define the least squares line as the line that minimizes the sum of the squared residuals, and list conditions necessary for fitting such line:
    1. linearity
    2. nearly normal residuals
    3. constant variability
  • Define an indicator variable as a binary explanatory variable (with two levels).
  • Calculate the estimate for the slope (\(b_1\)) as $b_1 = R\frac{s_y}{s_x}$, where \(r\) is the correlation coefficient, \(s_y\) is the standard deviation of the response variable, and \(s_x\) is the standard deviation of the explanatory variable.
  • Interpret the slope as
    • “For each unit increase in \(x\), we would expect \(y\) to increase/decrease on average by \(|b_1|\) units” when \(x\) is numerical.
    • “The average increase/decrease in the response variable when between the baseline level and the other level of the explanatory variable is \(|b_1|\).” when \(x\) is categorical.
    • Note that whether the response variable increases or decreases is determined by the sign of \(b_1\).
  • Note that the least squares line always passes through the average of the response and explanatory variables (\(\bar{x},\bar{y}\)).
  • Use the above property to calculate the estimate for the slope (\(b_0\)) as $b_0 = \bar{y} - b_1 \bar{x}$, where \(b_1\) is the slope, \(\bar{y}\) is the average of the response variable, and \(\bar{x}\) is the average of explanatory variable.
  • Interpret the intercept as
    • “When \(x = 0\), we would expect \(y\) to equal, on average, \(b_0\).” when \(x\) is numerical.
    • “The expected average value of the response variable for the reference level of the explanatory variable is \(b_0\).” when \(x\) is categorical.
  • Predict the value of the response variable for a given value of the explanatory variable, \(x^\star\), by plugging in \(x^\star\) in the in the linear model: $\hat{y} = b_0 + b_1 x^\star$
    • Only predict for values of \(x^\star\) that are in the range of the observed data.
    • Do not extrapolate beyond the range of the data, unless you are confident that the linear pattern continues.
  • Define \(R^2\) as the percentage of the variability in the response variable explained by the the explanatory variable.
    • For a good model, we would like this number to be as close to 100% as possible.
    • This value is calculated as the square of the correlation coefficient, and is between 0 and 1, inclusive.
  • Define a leverage point as a point that lies away from the center of the data in the horizontal direction.
  • Define an influential point as a point that influences (changes) the slope of the regression line.
    • This is usually a leverage point that is away from the trajectory of the rest of the data.
  • Do not remove outliers from an analysis without good reason.
  • Be cautious about using a categorical explanatory variable when one of the levels has very few observations, as these may act as influential points.
  • Determine whether an explanatory variable is a significant predictor for the response variable using the \(t\)-test and the associated p-value in the regression output.
  • Set the null hypothesis testing for the significance of the predictor as $H_0: \beta_1 = 0$, and recognize that the standard software output yields the p-value for the two-sided alternative hypothesis.
    • Note that \(\beta_1 = 0\) means the regression line is horizontal, hence suggesting that there is no relationship between the explanatory and the response variables.
  • Calculate the T score for the hypothesis test as $T_{df}=\frac { b_{ 1 }-{ null\quad value } }{ SE_{ b_{ 1 } } }$ with \(df = n - 2\).
    • Note that the T score has \(n - 2\) degrees of freedom since we lose one degree of freedom for each parameter we estimate, and in this case we estimate the intercept and the slope.
  • Note that a hypothesis test for the intercept is often irrelevant since it’s usually out of the range of the data, and hence it is usually an extrapolation.
  • Calculate a confidence interval for the slope as $b_1 \pm t^\star_{df} SE_{b_1}$ where \(df = n - 2\) and $t^\star_{df}$ is the critical score associated with the given confidence level at the desired degrees of freedom.
    • Note that the standard error of the slope estimate $SE_{b_1}$ can be found on the regression output.

Supplemental Readings

Videos