Endogeneity Goes Agains Assumption Gauss Markov

Ordinary Least Squares (OLS) is the almost common estimation method for linear models—and that's true for a skillful reason. As long equally your model satisfies the OLS assumptions for linear regression, yous tin can rest like shooting fish in a barrel knowing that you're getting the all-time possible estimates.

Regression is a powerful assay that tin analyze multiple variables simultaneously to answer complex research questions. However, if you don't satisfy the OLS assumptions, you lot might not be able to trust the results.

In this postal service, I cover the OLS linear regression assumptions, why they're essential, and help you make up one's mind whether your model satisfies the assumptions.

What Does OLS Judge and What are Good Estimates?

First, a bit of context.

Regression analysis is similar other inferential methodologies. Our goal is to draw a random sample from a population and utilize information technology to estimate the properties of that population.

In regression assay, the coefficients in the regression equation are estimates of the actual population parameters. We want these coefficient estimates to exist the all-time possible estimates!

Suppose you request an guess—say for the toll of a service that you are considering. How would you define a reasonable gauge?

1. The estimates should tend to be right on target. They should not be systematically too high or too low. In other words, they should be unbiased or correct on average.
2. Recognizing that estimates are almost never exactly correct, yous want to minimize the discrepancy between the estimated value and actual value. Large differences are bad!

These two properties are exactly what we need for our coefficient estimates!

When your linear regression model satisfies the OLS assumptions, the procedure generates unbiased coefficient estimates that tend to be relatively shut to the truthful population values (minimum variance). In fact, the Gauss-Markov theorem states that OLS produces estimates that are better than estimates from all other linear model estimation methods when the assumptions hold truthful.

For more information about the implications of this theorem on OLS estimates, read my postal service: The Gauss-Markov Theorem and Blueish OLS Coefficient Estimates.

The Seven Classical OLS Assumptions

Like many statistical analyses, ordinary least squares (OLS) regression has underlying assumptions. When these classical assumptions for linear regression are true, ordinary least squares produces the best estimates. Still, if some of these assumptions are not truthful, yous might demand to employ remedial measures or employ other estimation methods to improve the results.

Many of these assumptions describe backdrop of the error term. Unfortunately, the error term is a population value that we'll never know. Instead, we'll use the next best affair that is available—the residuals. Residuals are the sample estimate of the error for each observation.

Residuals = Observed value – the fitted value

When it comes to checking OLS assumptions, assessing the residuals is crucial!

There are seven classical OLS assumptions for linear regression. The first six are mandatory to produce the best estimates. While the quality of the estimates does not depend on the seventh assumption, analysts often evaluate it for other important reasons that I'll comprehend.

OLS Assumption 1: The regression model is linear in the coefficients and the error term

This supposition addresses the functional form of the model. In statistics, a regression model is linear when all terms in the model are either the constant or a parameter multiplied by an independent variable. You lot build the model equation only by adding the terms together. These rules constrain the model to one blazon:

In the equation, the betas (βs) are the parameters that OLS estimates. Epsilon (ε) is the random fault.

In fact, the defining characteristic of linear regression is this functional grade of the parameters rather than the ability to model curvature. Linear models can model curvature by including nonlinear variables such equally polynomials and transforming exponential functions.

To satisfy this supposition, the correctly specified model must fit the linear blueprint.

Related posts: The Difference Betwixt Linear and Nonlinear Regression and How to Specify a Regression Model

OLS Supposition two: The fault term has a population hateful of zero

The error term accounts for the variation in the dependent variable that the independent variables do not explain. Random chance should determine the values of the error term. For your model to exist unbiased, the average value of the error term must equal zero.

Suppose the boilerplate error is +7. This non-zero average error indicates that our model systematically underpredicts the observed values. Statisticians refer to systematic error like this as bias, and it signifies that our model is inadequate because it is not correct on average.

Stated another way, we want the expected value of the error to equal zero. If the expected value is +7 rather than zero, part of the fault term is predictable, and we should add that information to the regression model itself. We want merely random error left for the error term.

Yous don't demand to worry about this assumption when you lot include the constant in your regression model because it forces the mean of the residuals to equal zero. For more than data most this assumption, read my post almost the regression abiding.

OLS Assumption 3: All contained variables are uncorrelated with the error term

If an independent variable is correlated with the error term, we can use the independent variable to predict the error term, which violates the notion that the fault term represents unpredictable random error. Nosotros need to discover a fashion to incorporate that data into the regression model itself.

This assumption is besides referred to as exogeneity. When this type of correlation exists, there is endogeneity. Violations of this supposition can occur because there is simultaneity between the independent and dependent variables, omitted variable bias, or measurement error in the independent variables.

Violating this supposition biases the coefficient estimate. To empathize why this bias occurs, go on in mind that the mistake term e'er explains some of the variability in the dependent variable. Still, when an independent variable correlates with the error term, OLS incorrectly attributes some of the variance that the fault term actually explains to the independent variable instead. For more than information near violating this assumption, read my post virtually confounding variables and omitted variable bias.

Related mail: What are Independent and Dependent Variables?

OLS Assumption 4: Observations of the error term are uncorrelated with each other

I ascertainment of the error term should not predict the adjacent observation. For instance, if the error for one observation is positive and that systematically increases the probability that the following error is positive, that is a positive correlation. If the subsequent error is more probable to have the opposite sign, that is a negative correlation. This problem is known both as serial correlation and autocorrelation. Serial correlation is nigh probable to occur in time series models.

For example, if sales are unexpectedly high on one solar day, then they are likely to be higher than average on the next day. This blazon of correlation isn't an unreasonable expectation for some discipline areas, such every bit inflation rates, Gdp, unemployment, and so on.

Appraise this assumption by graphing the residuals in the order that the data were collected. You want to see randomness in the plot. In the graph for a sales model, in that location is a cyclical design with a positive correlation.

As I've explained, if you have information that allows you lot to predict the mistake term for an ascertainment, you lot must comprise that information into the model itself. To resolve this issue, you might need to add together an independent variable to the model that captures this information. Analysts commonly use distributed lag models, which utilize both current values of the dependent variable and past values of independent variables.

For the sales model above, we need to add variables that explains the cyclical pattern.

Series correlation reduces the precision of OLS estimates. Analysts tin also utilize time series assay for time dependent furnishings.

An alternative method for identifying autocorrelation in the residuals is to assess the autocorrelation role, which is a standard tool in time serial analysis.

Related post: Introduction to Time Series Analysis

OLS Assumption 5: The mistake term has a constant variance (no heteroscedasticity)

The variance of the errors should be consistent for all observations. In other words, the variance does not modify for each observation or for a range of observations. This preferred condition is known as homoscedasticity (same scatter). If the variance changes, nosotros refer to that every bit heteroscedasticity (different scatter).

The easiest way to check this supposition is to create a residuals versus fitted value plot. On this type of graph, heteroscedasticity appears as a cone shape where the spread of the residuals increases in i direction. In the graph below, the spread of the residuals increases every bit the fitted value increases.

Heteroscedasticity reduces the precision of the estimates in OLS linear regression.

Related post: Heteroscedasticity in Regression Analysis

Note: When assumption iv (no autocorrelation) and v (homoscedasticity) are both true, statisticians say that the mistake term is independent and identically distributed (IID) and refer to them as spherical errors.

OLS Assumption 6: No independent variable is a perfect linear function of other explanatory variables

Perfect correlation occurs when two variables have a Pearson's correlation coefficient of +1 or -1. When one of the variables changes, the other variable also changes by a completely stock-still proportion. The two variables motion in unison.

Perfect correlation suggests that two variables are different forms of the same variable. For example, games won and games lost have a perfect negative correlation (-i). The temperature in Fahrenheit and Celsius accept a perfect positive correlation (+ane).

Ordinary least squares cannot distinguish one variable from the other when they are perfectly correlated. If you specify a model that contains independent variables with perfect correlation, your statistical software can't fit the model, and it volition display an error bulletin. Y'all must remove one of the variables from the model to proceed.

Perfect correlation is a prove stopper. However, your statistical software can fit OLS regression models with imperfect but strong relationships between the independent variables. If these correlations are loftier enough, they tin can cause bug. Statisticians refer to this condition as multicollinearity, and it reduces the precision of the estimates in OLS linear regression.

Related post: Multicollinearity in Regression Assay: Issues, Detection, and Solutions

OLS Supposition 7: The fault term is ordinarily distributed (optional)

OLS does not require that the error term follows a normal distribution to produce unbiased estimates with the minimum variance. However, satisfying this assumption allows you to perform statistical hypothesis testing and generate reliable confidence intervals and prediction intervals.

The easiest way to make up one's mind whether the residuals follow a normal distribution is to appraise a normal probability plot. If the residuals follow the straight line on this blazon of graph, they are normally distributed. They look good on the plot beneath!

If you lot demand to obtain p-values for the coefficient estimates and the overall test of significance, check this assumption!

Why You lot Should Care Almost the Classical OLS Assumptions

In a nutshell, your linear model should produce residuals that have a hateful of zero, have a constant variance, and are non correlated with themselves or other variables.

If these assumptions hold true, the OLS procedure creates the all-time possible estimates. In statistics, estimators that produce unbiased estimates that have the smallest variance are referred to as being "efficient." Efficiency is a statistical concept that compares the quality of the estimates calculated past different procedures while holding the sample size abiding. OLS is the near efficient linear regression estimator when the assumptions hold truthful.

Some other benefit of satisfying these assumptions is that equally the sample size increases to infinity, the coefficient estimates converge on the actual population parameters.

If your error term also follows the normal distribution, you can safely utilise hypothesis testing to make up one's mind whether the independent variables and the unabridged model are statistically significant. Yous can also produce reliable confidence intervals and prediction intervals.

Knowing that you're maximizing the value of your data by using the nearly efficient methodology to obtain the best possible estimates should set your mind at ease. It's worthwhile checking these OLS assumptions! The best way to assess them is by using remainder plots. To learn how to do this, read my post about using balance plots!

If you're learning regression and like the approach I use in my blog, check out my eBook!

jonesthavence.blogspot.com

Source: https://statisticsbyjim.com/regression/ols-linear-regression-assumptions/

Share this post