Linear Models (OLS, Ridge, and Elastic Net Regressions)

Linear Models for Regression

Linear models are a cornerstone in predictive modeling, known for their simplicity, interpretability, and computational efficiency. They serve as an excellent starting point for understanding the relationships between features and the target variable, especially in the context of regression tasks like house price prediction.

In this post, we’ll delve into two widely used linear regression techniques: Ordinary Least Squares (OLS) and Regularized Linear Models (specifically, Ridge and Elastic Net Regressions).

You can find the complete code in my GitHub repository.

Contents

1. Ordinary Least Squares (OLS) Regression
2. Ridge and Elastic Net Regression for Improved Predictions
3. Comparison of OLS Coefficients and Feature Importance from Ridge and Elastic Net Regression Models
4. Cross-Validation Techniques

1. Ordinary Least Squares (OLS) Regression

Ordinary Least Squares (OLS) is the most straightforward and commonly used linear regression method. It operates on the principle of minimizing the sum of the squared differences between the observed and predicted values, known as residuals.

The goal of OLS is to find the best-fitting line that reduces these residuals to the smallest possible sum, thereby providing the most accurate predictions within the linear model’s assumptions.

By fitting a linear equation to the data, OLS allows us to quantify the impact of each variable, offering a clear and interpretable model where the coefficients represent the expected change in the target variable (house price) for each unit change in the predictor.

However, while OLS is powerful and easy to interpret, it has limitations, particularly when dealing with multicollinearity (high correlation between independent variables) or when the model is prone to overfitting due to a large number of features.

Now that we’ve covered the basics of OLS, let’s look at how it performs in practice.

Model Performance

The Ordinary Least Squares (OLS) regression model was evaluated using multiple metrics through cross-validation.

Mean Absolute Error (MAE): 18,647 (+/- 2,643)

• MAE measures the average absolute difference between the predicted and actual values.
• A lower MAE indicates better model performance. An MAE of $18,647 suggests that, on average, the model’s predictions are off by this amount. The relatively small standard deviation (+/- $2,643) indicates that the model’s performance is consistent across the different folds of the cross-validation.

Mean Squared Error (MSE): 1,377,330,606 (+/- 982,155,554)

• MSE measures the average squared difference between the predicted and actual values, penalizing larger errors more severely.
• The high MSE value reflects the impact of some large prediction errors, and the large standard deviation indicates variability in performance across the cross-validation folds.

Root Mean Squared Error (RMSE): 36,411 (+/- 14,357)

• RMSE is the square root of MSE and provides an error metric in the same units as the target variable (house prices).
• An RMSE of $36,411 indicates the typical magnitude of error in the predictions.
• The substantial standard deviation (+/- $14,357) suggests that the model’s performance varies significantly depending on the data split, highlighting potential issues with model stability.

Mean Absolute Percentage Error (MAPE): 11.036% (+/- 0.977%)

• MAPE expresses prediction accuracy as a percentage, indicating that the model’s predictions deviate by about 11% from the actual values on average.
• This is a fairly low error rate, suggesting that the model is generally accurate, with a small standard deviation indicating consistent performance.

Median Absolute Error (MedAE): 11,607 (+/- 1,154)

• MedAE provides the median of absolute errors, offering a robust measure against outliers.
• The lower MedAE compared to MAE suggests that the majority of the model’s predictions are closer to the actual values, though a few large errors may be skewing the MAE upward. The small standard deviation also indicates stable performance across folds.

R-squared (R2): 0.780 (+/- 0.178)

• R2 measures the proportion of variance in the target variable that is explained by the model.
• R2 of 0.780 suggests that the model explains about 78% of the variability in house prices, which is relatively good.
• However, the high standard deviation (+/- 0.178) indicates that the model’s explanatory power is inconsistent across different subsets of the data, pointing to potential issues with overfitting or sensitivity to specific features.

Coefficient Analysis

The OLS model’s coefficients provide insight into the impact of individual features on house prices:

Top 5 Positive Coefficients:

1. TimeSinceRemodel (644,439): The high positive coefficient for TimeSinceRemodel suggests that older remodels are valued more, potentially due to their historical charm or durability.
2. RoofMatl_WdShngl (498,064)
3. RoofMatl_Membran (482,169)
4. RoofMatl_Roll (448,296)
5. RoofMatl_Meta (446,629)
   These roofing materials significantly increase house prices, indicating that high-quality or unique roofing options are valued by buyers.

Top 5 Negative Coefficients:

1. Condition2_PosN (-232,583): This condition has a large negative impact on house prices, potentially indicating a less desirable location or neighborhood characteristic.
2. PoolQC_missing (-126,523): The lack of information about pool quality, or the absence of a pool, negatively impacts house prices, suggesting that pools are a valued feature.
3. GarageQual_Po (-102,573): Poor garage quality significantly reduces house prices, reflecting the importance of functional and well-maintained garages.
4. Condition2_RRAe (-97,157)
5. MiscFeature_TenC (-96,741):
   These features also have a negative impact on house prices, indicating that certain conditions and miscellaneous features detract from the property’s overall value.

Conclusion

The OLS regression model demonstrates reasonable predictive power with an R2 of 0.780 and a MAPE of 11.036%, indicating that it captures a substantial portion of the variance in house prices.

However, the model’s performance varies significantly across different validation folds, as evidenced by the large standard deviations in MSE, RMSE, and R2.

This suggests that the model may be sensitive to specific data subsets or features, and further refinement or regularization might be needed to improve its stability and generalizability.

The coefficient analysis highlights the importance of certain features, such as remodeling time and roofing material, in influencing house prices, while also pointing out the negative impact of poor conditions and missing features like pool quality.

2. Ridge and Elastic Net Regression for Improved Predictions

In an effort to improve the predictive performance of our house price model, I implemented two advanced regularization techniques: Ridge Regression and Elastic Net Regression.

These methods aim to address overfitting and improve model generalization by introducing penalty terms that shrink the coefficients of less important features.

Ridge Regression

Ridge Regression applies L2 regularization, which adds a penalty proportional to the square of the coefficients to the loss function. This penalization helps reduce the impact of less important features, thereby improving the model’s ability to generalize to new data.

Elastic Net Regression

Elastic Net Regression combines both L1 and L2 regularization. This approach allows Elastic Net to perform both feature selection (like Lasso) and coefficient shrinkage (like Ridge), making it particularly effective in handling correlated features.

Model Performance

	Linear Regression	Ridge Regression	Elastic Net Regression
Mean Absolute Error (MAE)	18,647 (+/- 2,643)	16,753 (+/- 2554)	16,613 (+/- 2,387)
Mean Squared Error (MSE)	1,377,330,606 (+/- 982,155,554)	853,323,765 (+/- 614,762,731)	834,494,733 (+/- 611,226,242)
Root Mean Squared Error (RMSE)	36,411 (+/- 14,357)	28,730 (+/- 10,565)	28,388 (+/- 10,703)
Mean Absolute Percentage Error (MAPE)	11.036% (+/- 0.977%)	9.799% (+/- 1.089%)	9.860% (+/- 1.127%)
Median Absolute Error (MedAE)	11,607 (+/- 1,154)	11,458 (+/- 1,770)	11,318 (+/- 1,634)
R-squared (R2)	0.780 (+/- 0.178)	0.867 (+/- 0.073)	0.870 (+/- 0.074)

Both Ridge and Elastic Net Regression demonstrated improved performance over the baseline OLS model, particularly in terms of reducing the Mean Squared Error (MSE) and enhancing the R-squared value.

The reduction in RMSE and MAPE across both models indicates more accurate and stable predictions, with Ridge Regression slightly edging out Elastic Net in terms of MAPE and RMSE.

R2 Improvement: The R-squared values for Ridge (0.867) and Elastic Net (0.870) show a substantial improvement over the OLS model, meaning that these models explain a larger portion of the variance in house prices. This suggests that the regularization techniques successfully mitigated overfitting and improved the model’s generalizability.

Error Metrics: The lower MAE and MedAE values for both Ridge and Elastic Net indicate that the typical prediction error has decreased, leading to more reliable and consistent predictions. The Elastic Net model achieved a slightly lower MAE and RMSE compared to Ridge, suggesting that the combination of L1 and L2 regularization is beneficial in capturing the underlying data patterns.

RMSE and MSE: The RMSE, both directly calculated and derived from MSE, is lower in the Elastic Net model, pointing to its superior ability in handling outliers and large deviations in the dataset. This is likely due to the model’s ability to perform feature selection while maintaining stability across the remaining features.

Elastic Net’s Advantage

The marginally better performance of the Elastic Net model highlights its flexibility in balancing between Ridge and Lasso’s strengths.

By adjusting the l1_ratio, Elastic Net effectively penalizes redundant or irrelevant features while preserving the predictive power of more important ones, making it particularly useful in datasets with multicollinearity or where some features are more influential than others.

Feature Importance

The results from the Ridge and Elastic Net regression models provide valuable insights into which features most strongly influence house prices. While both models prioritize similar features, the differences in their importance rankings highlight the unique strengths of each regularization technique.

Top 10 Features by Ridge Importance

Ridge Regression applies L2 regularization, which tends to distribute the influence more evenly across the features, particularly when dealing with correlated variables. The top features identified by Ridge Regression are:

1. TotalSF_OverallQual (168,28): This interaction feature, combining total square footage and overall quality, is the most important predictor in the Ridge model. It underscores how the size and quality of a house are critical in determining its price.
2. GrLivArea_TotRmsAbvGrd (9,395): This feature captures the interaction between above-ground living area and the total number of rooms, emphasizing that more and larger rooms add significant value to a house.
3. TotalSF_HouseAge (9,210): The combination of total square footage with the age of the house suggests that newer, larger homes are particularly valuable.
4. BsmtQual_Gd (8,872): Good basement quality is a strong positive indicator of house price, reflecting the importance of a well-finished basement.
5. Neighborhood_CollgCr (7,411): Living in the College Creek neighborhood is associated with higher house prices, likely due to desirable location factors such as proximity to amenities or reputation.
6. KitchenQual_Gd (7,103): A good-quality kitchen significantly boosts house prices, highlighting the importance of this key area of the home.
7. OverallQual (6,879): The general quality rating of the house remains a crucial factor, as higher-quality homes tend to command higher prices.
8. Condition1_Norm (6,780): This feature indicates normal proximity to various conditions, suggesting that homes in standard conditions (e.g., not near negative influences) are more valuable.
9. BsmtExposure_Gd (6,304): Good basement exposure, such as having walkout basements or large windows, contributes positively to house value.
10. GarageArea_GarageCars (6,182): The interaction between garage area and the number of cars it can accommodate shows that garage size and capacity are important in determining house prices.

Top 10 Features by Elastic Net Importance

Elastic Net Regression, which combines L1 and L2 regularization, balances the strengths of Ridge and Lasso by shrinking some coefficients more aggressively and allowing for feature selection. The top features identified by Elastic Net are:

1. TotalSF_OverallQual (21,457): Like in Ridge Regression, this feature is the most important in Elastic Net, but with a higher coefficient, indicating that Elastic Net places even more emphasis on the interaction of size and quality.
2. TotalSF_HouseAge (12,610): Elastic Net also identifies this as a key feature, but with a slightly higher importance than in Ridge, emphasizing the value of newer, larger homes.
3. GrLivArea_TotRmsAbvGrd (12,448): The importance of the living area and room count is even more pronounced in Elastic Net, reflecting the model’s sensitivity to features that contribute to overall livability.
4. BsmtQual_Gd (10,700): The good quality of a basement is crucial in both models, with Elastic Net assigning a higher importance, likely due to its ability to focus on specific, impactful features.
5. Neighborhood_CollgCr (10,621): The College Creek neighborhood remains an important predictor, with a higher coefficient in Elastic Net, possibly reflecting the model’s ability to capture neighborhood-specific effects.
6. KitchenQual_Gd (9,626): A good kitchen is a major selling point, with Elastic Net highlighting its importance more than Ridge, possibly due to the model’s feature selection capabilities.
7. RoofMatl_WdShngl (8,956): This feature appears in the Elastic Net model but not in Ridge, indicating Elastic Net’s ability to identify specific materials that significantly influence price.
8. Condition1_Norm (8,415): The condition of being in a normal proximity remains significant, with Elastic Net slightly increasing its importance, reflecting the model’s more nuanced feature handling.
9. BsmtExposure_Gd (8,107): Good basement exposure is again highlighted, with a higher coefficient in Elastic Net, showing its enhanced focus on key features.
10. Functional_Typ (7,768): The functionality of the house (e.g., room layout and usability) appears in the Elastic Net model, indicating its role in house pricing—a feature not highlighted by Ridge.

Comparison and Implications

• Feature Overlap: Both models prioritize similar features, particularly those related to overall quality, size, and key areas like the kitchen and basement. This consistency reinforces the importance of these features in determining house prices.
• Elastic Net’s Enhanced Feature Selection: The Elastic Net model identifies additional features like RoofMatl_WdShngl and Functional_Typ, which are not as prominent in the Ridge model. This suggests that Elastic Net’s combination of L1 and L2 penalties effectively identifies features that might be overlooked in a purely L2-regularized model like Ridge.
• Coefficient Magnitude: Elastic Net generally assigns higher importance to its top features compared to Ridge, indicating that it may provide sharper distinctions between the most and least important features. This can be particularly useful when the goal is to focus on the most impactful predictors.

3. Comparison of OLS Coefficients and Feature Importance from Ridge and Elastic Net Regression Models

The Ordinary Least Squares (OLS) regression model and the regularized models, Ridge and Elastic Net, offer different insights into the relationships between features and the target variable (house prices).

Here’s a discussion of the differences between the OLS model’s coefficients and the feature importance rankings from the Ridge and Elastic Net regression models.

OLS Model’s Coefficients

The OLS model provides a direct estimation of the impact of each feature on house prices by assigning a coefficient to each variable. These coefficients indicate the expected change in the target variable for a one-unit change in the predictor, assuming all other variables remain constant.

Interpretation: The OLS coefficients offer a straightforward interpretation of feature importance. For example, a high positive coefficient for TimeSinceRemodel suggests that houses remodeled longer ago tend to be valued more, possibly due to their historical charm or durability.

Limitations: However, OLS has limitations, especially in the presence of multicollinearity (where predictor variables are highly correlated). Multicollinearity can lead to inflated or unstable coefficients, making it difficult to accurately assess the true impact of each feature.

Additionally, OLS does not include any mechanism to penalize complex models, which can lead to overfitting, especially with a large number of features.

Ridge Regression Feature Importance

Ridge Regression, through L2 regularization, shrinks the coefficients of less important features, distributing the influence more evenly across correlated variables. This regularization reduces the risk of overfitting and provides a more stable and interpretable model.

Smoothing Effect: Ridge tends to reduce the impact of features that might have large coefficients in the OLS model due to noise or multicollinearity.

Consistent Importance: Features like GrLivArea_TotRmsAbvGrd and TotalSF_HouseAge remain important in Ridge, but with moderated coefficients. This reflects Ridge’s ability to maintain the relative importance of features while preventing any single feature from dominating the model, particularly in the presence of correlated variables.

Elastic Net Regression Feature Importance

Elastic Net combines the L1 regularization of Lasso (which can drive some coefficients to zero) and the L2 regularization of Ridge, allowing it to perform both feature selection and coefficient shrinkage.

Feature Selection: Elastic Net often emphasizes fewer, more impactful features compared to Ridge. For example, TotalSF_OverallQual and TotalSF_HouseAge have even higher coefficients in Elastic Net than in Ridge, indicating that Elastic Net focuses more on the most predictive features. This makes Elastic Net particularly useful when you suspect that only a subset of the features are truly important.

Unique Features: Elastic Net can identify features that Ridge may not highlight as strongly. For instance, RoofMatl_WdShngl appears prominently in Elastic Net but not in Ridge, showing Elastic Net’s ability to highlight specific, important features through its feature selection capability.

Key Differences

Coefficient Magnitude: OLS provides coefficients that can be quite large due to the lack of regularization, which can make interpretation difficult, especially in the presence of multicollinearity. Ridge and Elastic Net, through regularization, provide more moderate and stable coefficients, with Elastic Net potentially driving some coefficients to zero.

Handling of Correlated Features: OLS struggles with correlated features, often leading to inflated coefficients. Ridge handles this by distributing importance more evenly across correlated features, while Elastic Net can selectively zero out less important correlated features, making the model more interpretable.

Feature Selection: OLS includes all features, even if some contribute little to the prediction. Ridge also includes all features but reduces the impact of less important ones. Elastic Net, by contrast, can effectively exclude irrelevant features, making it a powerful tool for models where feature selection is crucial.

Model Stability: The regularization in Ridge and Elastic Net leads to more stable models compared to OLS, particularly in the presence of a large number of features. This stability is reflected in more consistent performance metrics across cross-validation folds.

4. Discussion on Cross-Validation Techniques

Cross-validation is a critical step in the model evaluation process, providing insights into how well a model generalizes to unseen data. While cross-validation was mentioned in the previous discussion, it’s important to delve deeper into the specific technique used and its implications for model performance assessment.

Understanding Cross-Validation

Cross-validation involves splitting the dataset into multiple subsets or “folds,” training the model on some of these subsets, and then validating it on the remaining ones. This process is repeated several times, and the results are averaged to provide a more robust estimate of the model’s performance. The most common form of cross-validation is k-fold cross-validation.

k-Fold Cross-Validation

In k-fold cross-validation, the dataset is divided into k equally sized folds. The model is trained on k-1 of these folds and tested on the remaining fold. This process is repeated k times, with each fold used exactly once as the validation set. The results from each fold are then averaged to produce the final performance metrics.

In our analysis of linear models (OLS, Ridge, and Elastic Net), I utilized k-fold cross-validation, specifically with k=5.

In 5-fold cross-validation:

• The dataset is split into 5 folds.
• The model is trained on 4 folds and tested on the 5th fold.
• This process is repeated 5 times, each time with a different fold serving as the test set.
• The final performance metrics are calculated by averaging the results from all 5 iterations.

This choice is common in machine learning practice as it provides a good trade-off between computation time and model evaluation robustness. Here’s how it impacts the model evaluation:

Stability of Results: The use of 5 folds helps ensure that the performance metrics (such as MAE, MSE, RMSE, R2) are not overly influenced by a particular subset of the data. This results in more stable and reliable estimates of how the model will perform on unseen data.

Model Comparison: By applying the same cross-validation technique across all models, we ensure a fair comparison. The differences in performance metrics like MAE, MSE, and R2 can be attributed to the model’s ability to generalize, rather than to variations in data splits.

Standard Deviations: The standard deviations reported in the results (e.g., for MAE, MSE, RMSE) reflect the variability in model performance across the different folds. A large standard deviation may indicate that the model’s performance is sensitive to the specific data split, which is a crucial factor to consider when assessing model reliability.

5. Conclusion

In this exploration of linear models for regression, I’ve demonstrated the strengths and limitations of Ordinary Least Squares (OLS), Ridge, and Elastic Net regressions in predicting house prices. OLS, while straightforward and interpretable, struggles with multicollinearity and potential overfitting.

Regularized models like Ridge and Elastic Net offer significant improvements, particularly in generalization and handling correlated features.

Through cross-validation, we observed that Ridge and Elastic Net not only reduced mean squared errors but also provided more stable predictions across different data splits.

Elastic Net, with its blend of L1 and L2 regularization, was particularly effective in feature selection, identifying key predictors that might be overlooked by Ridge.

The comparison between OLS coefficients and the feature importance from regularized models highlights how regularization techniques can lead to more balanced and reliable models.

Ultimately, the choice between these models depends on the specific data characteristics and the need for feature selection or coefficient stability.

This analysis underscores the value of regularization in enhancing model robustness and accuracy, making Ridge and Elastic Net excellent choices for complex, multicollinear datasets.