CHAPTER 10: MULTICOLLINEARITY: WHAT HAPPENS IF THE REGRESSORS ARE CORRELATED?

ü  The Nature of Multicollinearity

Multicollinearity – the existence of a “perfect”, or exact, linear relationship among some or all explanatory variables for a regression model.

Why does the classical linear regression model assume that there is no multicollinearity among the X’s?

If multicollinearity is perfect, the regression coefficients of the X variables are indeterminate and their standard errors are infinite. If multicollinearity is less than perfect, the regression coefficients, although determinate, possess large standard errors, which means the coefficient cannot be estimated with great passion or accuracy.

Sources of Multicollinearity

1.      The data collection method employed.

2.      Constraints on the model or in the population being sampled.

3.      Model specification.

4.      An overdetermined model.

ü  Practical Consequences of Multicollinearity

1.      Although BLUE, the OLS estimators have large variances and covariances, making precise estimation difficult.

2.      Because of consequence 1, the confidence intervals tend to be much wider, leading to the acceptance of the “zero null hypothesis”.

3.      Also because of consequence 1, the t ratio of one or more coefficients tend to be statistically insignificant.

4.      Although the t ratio of one or more coefficients is statistically insignificant, R², the overall measure of goodness of fit, can be very high.

5.      The OLS estimators and their standard errors can be sensitive to small changes in the data.

ü  Large Variances and Covariances of OLS Estimators

var(β₂) =

                  ∑X_2i(1 – r₂₃)

var(β₃) =

                  ∑X_3i(1 – r₂₃)

cov(β₂, β₃) =         -r₂₃

                        (1 – r₂₃)   ∑X_2i∑X_3i

ü  Detection of Multicollinearity

1.      High R² but few significant t ratios.

2.      High pair-wise correlations among regressors.

3.      Examinations of partial correlations.

4.      Auxiliary regressions.

5.      Eigenvalues and condition index.

CI =  Maximum eigenvalue =    k

         Minimum eigenvalue

6.      Tolerance and variance inflation factor

ü  Remedial Measures

·         Do Nothing

·         Rule-of-Thumb Procedures

1)      A priori information.

2)      Combining cross-sectional and time series data.

3)      Dropping a variable(s) and specification bias.

4)      Transformation of variables.

5)      Additional or new data.

6)      Reducing collinearity in polynomial regressions.

7)      Other methods of remedying multicollinearity.

CHAPTER 9: DUMMY VARIABLE REGRESSION MODELS

ü  The Nature of Dummy Variable

Dummy variables – such variables are thus essentially a device to classify data into mutually exclusive categories such as male or female.

ü  Caution in the Use of Dummy Variables

1.      If a qualitative variable has m categories, introduce only (m – 1) dummy variables.

For each qualitative regressor the number of dummy variables introduced must be one less than the categories of that variable.

2.      The category for which no dummy variable is assigned is known as the base, benchmark, control, comparison, reference, or omitted category.

3.      The intercept value (β₁) represents the mean value of the benchmark category.

4.      The coefficients attached to the dummy variables are known as the differential intercept coefficients because they tell how much the value of the intercept that receives the value of 1 differs from the intercept coefficient of the benchmark category.

5.      If a qualitative variable has more than one category, the choice of the benchmark category is strictly up to the researcher.

6.      There is a way to circumvent this trap by introducing as many dummy variables as the number of categories of that variable, provided we do not introduce the intercept in such a model.

7.      Which is a better method of introducing a dummy variable: (1) introduce a dummy for each category and omit the intercept term or (2) include the intercept term and introduce only (m – 1) dummies.

ü  The Dummy Variable Alternative to the Chow Test

Four possibilities:

1.      Both the intercept and the slope coefficients are the same in the two regressions. This, the case of coincident regressions.

2.      Only the intercepts in the two regressions are different but the slopes are the same. This is the case of parallel regressions.

3.      The intercepts in the two regressions are the same, but the slopes are different. This is the situation of concurrent regressions.

4.      Both the intercepts and slopes in the two regressions are different. This is the case of dissimilar regressions.

ü  The Use of Dummy Variables in Seasonal Analysis

Note that to avoid the dummy variable trap, we are assigning a dummy to each quarter of the year, but omitting the intercept term.

ü  Topics for Further Study

Several topics related to dummy variables are discussed in the literature that are rather advanced, including random, or varying, parameters models, switching regression models and disequilibrium models.

CHAPTER 8: MULTIPLE REGRESSION ANALYSIS: THE PROBLEM OF INFERENCE

ü  Hypothesis Testing in Multiple Regression: General Comments

1.      Testing hypotheses about an individual partial regression coefficient.

2.      Testing the overall significance of the estimated multiple regression model, that is, finding out all the partial slope coefficients are simultaneously equal to zero.

3.      Testing that two or more coefficients are equal to one another.

4.      Testing that the partial regression coefficients satisfy certain restrictions.

5.      Testing the stability of the estimated regression model over time in different cross-sectional units.

6.      Testing the functional form of regression models.

ü  Testing the Overall Significance of a Multiple Regression: the F-test

Decision Rule: Given the k-variable regression model:

Y_i = β₁ + β₂X_2i + β₃X_3i + ……….+ β_kX_ki + u_i

To test the hypothesis

H₀: β₂ = β₃ = ………. = β_k = 0

H₁: Not all slope coefficients are simultaneously zero

compute

F = ESS/df   =   ESS/(k – 1)

      RSS/df        RSS/(n – k)

If F>F_α(k – 1, n – k), reject H₀: otherwise you do not reject it, where F_α(k – 1, n – k) is the critical F value at the α level of significance and (k – 1) numerator df and (n – k) denominator df. Alternatively, if the p value of F is sufficiently low, one can reject H₀.

ü  Testing the Overall Significance of a Multiple Regression in terms of R²

Decision Rule: Testing the overall significance of a regression in terms of R². Alternative but equivalent test.

Given the k-variable regression model

Y_i = β₁ + β₂X_2i + β₃X_3i + ……….+ β_kX_ki + u_i

To test the hypothesis

H₀: β₂ = β₃ = ………. = β_k = 0

versus

H₁: Not all slope coefficients are simultaneously zero

compute

F = R²/ (k – 1)

            (1 – R²)/(n – k)

If F>F_α(k – 1, n – k), reject H0; otherwise you may accept H₀ where F_α(k – 1 , n – k) is the critical F value at the α level of significance and (k – 1) numerator df and (n – k) denominator df. Alternatively, if the p value of F obtained is sufficiently low, reject H₀.

ü  The “Incremental” or “Marginal” Contribution of an Explanatory Variable

F = Q₂/df

      Q₄/df

   = ESS_new – ESS_old/ number of new regressors

      RSS_new/df( =n – number of parameters in the new model)

·         When to Add a New Variable

It can be shown that R² will increase if the t value of the coefficient of the newly added variable is larger than 1 in absolute value. R² will increase with the addition of an extra explanatory variable only if the F(= t²) value of that variable exceeds 1.

·         When to Add a Group of Variables

If adding (dropping) a group of variables to the model gives an F value greater (less) than 1, R² will increase (decrease).

ü  Testing the Equality of Two Regression Coefficients

ü  Testing for Structural or Parameter Stability of Regression Models: The Chow Test

Assumptions of Chow Test:

1.      u_1t      N(0,      ) and u_2t       N(0,      )

2.      The two error terms u1t and u2t are independently distributed.

The mechanics of the Chow test are as follows:

1.      Estimate regression which is appropriate if there is no parameter instability, and obtain RSS₃ with df = (n1 + n2 – k), where k is the number of parameters estimated, 2 in the present case.

2.      Estimate and obtain its residual sum of squares RSS₁ with df = (n1 – k).

3.      Estimate and obtain its residual sum of squares, RSS₂, with df = (n2 – k).

4.      Since the two sets of samples are deemed independent, we can add RSS₁ and RSS₂ to obtain what may be called unrestricted residual sum of squares (RSS_UR), that is, obtain:

RSS_UR = RSS₁ + RSS₂   with df = (n1 + n2 – 2k)

5.      There is no structural change, then the RSS_R and RSS_UR should not be statistically independent.

6.      Therefore, we do not reject the null hypothesis of parameter stability if the computed F value in an application does not exceed the critical F value obtained from the F value at the chosen level of significance.

There are some caveats about the Chow test that must be kept in mind:

1.      The assumptions underlying the test must be fulfilled.

2.      The Chow test will tell us only if the two regressions are different, without telling us whether the difference is on account of the intercepts, or the slopes, or both.

3.      The Chow test assumes that we know the point(s) of structural break.

ü  Testing the Functional Form of Regression Choosing Between Linear and Log-Linear Regression Models

The MacKinnon, White, Davidson Test involves the following steps

Step 1: Estimate the linear model and obtain the estimated Y values.

Step 2: Estimate the log-linear model and obtain the estimated ln Y values.

Step 3: Obtain z₁ = (lnYf – lnf)

Step 4: Regress Y on X’s and Z₁ obtained in Step 3.

Step 5: Obtain z₂ = (antilog of lnf – Yf)

Step 6: Regress log of Y on the logs of X’s and z₂.

CHAPTER 7: MULTIPLE REGRESSION ANALYSIS: THE PROBLEM OF ESTIMATION

ü  The Three-Variable Model Notation and Assumptions

Y_i = β₁ + β₂X2_i + β₃X_3i + u_i

β2 and β3 are called the partial regression coefficients.

Assumptions:

1.      Zero mean value of u_i or

       E(u_i  X_2i, X_3i) = 0 for each i

2.      No serial correlation, or

cov(u_i, u_j) = 0         i   j

3.      Homoscedasticity, or

var(u_i) = 

4.      Zero covariance between u_i and each X variable, or

cov(u_i, X_2i) = cov(u_i, X_3i) = 0

5.      No specification bias, or

The model is correctly specified

6.      No exact collinearity between the X variables, or

No exact linear relationship between X₂ and X₃.

ü  Interpretation of Multiple Regression Equation

E(Y_i  X_2i, X_3i) = β₁ + β₂X_2i + β_3iX_3i

The conditional mean or expected value of Y conditional upon the given or fixed values of X₂ and X₃.

ü  The Meaning of Partial Regression Coefficients

β₂ and β₃ are known as partial regression or partial slope coefficients. β₂ measures the change in the mean value of Y, E(Y), per unit change in X₂, holding the value of X₃ constant. Put differently, it gives the “direct” or “net” effect of a unit change in X₂ on the mean value of Y, net of any effect that X₃ may have on mean Y. Likewise, β₃ measures the change in the mean value of Y per unit change in X₃, holding the value of X₂ constant. That is, it gives the “direct” or “net” effect of a unit change in X₃ on the mean value of Y, net of any effect that X₂ may have on mean Y.

ü  OLS and ML Estimation of the Partial Regression Coefficients

ü  OLS Estimators

OLS  estimator of the population intercept β₁

β₁ = Y – β₂X₂ – β₃X₃

OLS  estimator of the population intercept β₂

β₂ = (∑Y_iX_2i)(∑X_3i) – (∑Y_iX_3i)(∑X_2iX_3i)

                    (∑X_2i)(∑X_3i) – (∑X_2i3i)²

OLS  estimator of the population intercept β₃

β₃ = (∑_YiX_3i)(∑X_2i) – (∑Y_iX_2i)(∑X_2iX_3i)

                     (∑X_2i)(∑X_3i) – (∑X_2i3i)²

Notation:

1)      Equations of β₂ and β₃ are symmetrical in nature because one can be obtained from the other by interchanging the roles of X₂ and X₃.

2)      The denominators of these two equations are identical.

3)      The three-variable case is a natural extension of the two-variable case.

ü  Variances and Standard Errors of OLS Estimators

var(β₁) =  1   +     X₂∑X_3i + X₃∑X_2i - 2X₂X₃∑X_2iX_3i

                   n               ∑X_2i∑X_3i – (∑X_2iX_3i)2

se(β₁) =   +   var(β₁)

var(β₂) =                ∑X_3i

                  (∑X_2i)(∑X_3i) – (∑X_2iX_3i)2

or equivalently,

var(β₂) =     

                 ∑X_2i(1 – r₂₃)

var(β₃) =         ∑X_2i

                 (∑X_2i)(∑X_3i) – (∑X_2iX_3i)2

se(β₂) =   +    var(β₂)

se(β₃) =   +    var(β₃)

cov(β₂, β₃) =        -r₂₃

                        (1 – r₂₃)  X_2i   X_3i

ü  Properties of OLS Estimators

1)      The three-variable regression line passes through the means Y, X₂ and X₃.

2)      The mean value of the estimated Y_i(= Y_i) is equal to the mean value of the actual Y_i.

3)      ∑u_i = u =0

4)      The residuals u_i are uncorrelated with X_2i and X_3i, that is, ∑u_iX_2i = ∑u_iX_3i = 0

5)      The residuals u_i are uncorrelated with Y_i, that is, ∑u_iY_i = 0

6)      The correlation coefficient between X₂ and X₃ increases toward 1, the variances of β₂ and β₃ increase for given values of        and ∑X_2i or ∑X_3i.

ü  The Multiple Coefficient of Determination R² and the Multiple Coefficient of Correlation R

r² = measure the goodness of fit of the regression equation; that is, it gives the proportion or percentage, of the total variation in the dependent variable Y explained by the explanatory variable X.

R² = RSS

        TSS

R² = β₂∑Y_iX_2i + β₃Y_iX_3i

        ∑Y_i²

ü  Simple Regression in the Context of Multiple Regression: Introduction to Specification Bias

Observe several things about this regression compared to the “true” multiple regression.

1.      In absolute terms, the PGNP coefficient has increased.

2.      The standard errors are different.

3.      The intercept values are different.

4.      The r² values are dramatically different.

ü  Interpretation of Simple and Partial Correlation Coefficients

1.      Even if r₁₂ = 0, r₁₂₃ will not be zero unless r₁₃ or r₂₃ or both are zero.

2.      If r₁₂ = 0 and r₁₃ and r₂₃ are nonzero and are of the same sign, r₁₂₃ will be negative, whereas if they are of the opposite signs, it will be positive.

3.      In terms of r₁₂₃ and r₁₂ need not have the same sign.

4.      In the two-variable case we have seen that r² lies between 0 and 1.

5.      The coefficient of partial determination may be interpreted as the proportion of the variation in Y not explained by the variable X₃ that has been explained by the inclusion of X₃ into the model.