Comparative Analysis of Regression and Machine Learning Methods for Predicting Student Academic Performance

Abstract

This study compares the performance of Linear Regression, Ridge Regression, Lasso Regression, and Random Forest Regression in predicting student academic performance using the Higher Education Students Performance Evaluation Dataset from the UCI Machine Learning Repository, consisting of 145 data points with GRADE as the target variable. Although various studies have compared prediction algorithms for academic performance, research integrating correlation analysis, VIF multicollinearity testing, Lasso feature selection, and Random Forest feature importance within a unified comparative framework on high-multicollinearity datasets remains limited. The selection of these four algorithms was based on the dataset's high multicollinearity characteristics (22 out of 30 features with VIF > 10), necessitating a comparison between conventional regression, regularization methods (Ridge and Lasso), and ensemble learning (Random Forest). The methodology includes data preprocessing, Pearson correlation analysis, VIF multicollinearity testing, model development, and evaluation using R², RMSE, and MAE. Correlation analysis showed that only 8 out of 30 features had a significant correlation with GRADE (highest r = 0.3355). VIF testing identified 22 features with VIF > 10, indicating very high multicollinearity. Lasso Regression was the best model with R² testing of 0.1535, RMSE of 1.9735, and MAE of 1.5905, outperforming Ridge Regression (R² = 0.1476), Random Forest (R² = 0.1346), and Linear Regression (R² = -0.0902). Although Lasso Regression provided the best performance, the R² testing value of only 0.1535 indicates that most of the variation in academic performance cannot be explained by the model, due to limited data (n=145), weak linear correlation between features and GRADE (highest r = 0.3355), and the possibility of other factors beyond the dataset affecting academic performance. Ridge used alpha 100.0 and Lasso used alpha 0.1 with elimination of 10 features. Random Forest showed overfitting with a negative average cross-validation R² (-0.0108). Feature 29 (Educational Habits) and feature 11 (Family Questions) were identified as the most consistent predictors, with feature 29 having a significant positive correlation with GRADE (r = 0.3155; p-value = 0.0001). The study concludes that Lasso Regression is more effective on datasets with high multicollinearity, and educational habits are the dominant factor in predicting academic performance.