Entropy-Based Estimation for Multivariate Logistic Regression: A Newton-Raphson Approach

Abstract

This research presents a new model for entropy-constrained logistic regression aimed at improving categorical data analysis. An estimation method was developed based on the Newton-Raphson algorithm using Shannon entropy as a regularization constraint to address maximum likelihood estimation (MLE) problems, especially in cases of small samples or inaccurate data. A Monte Carlo simulation with 5000 iterations was conducted to evaluate the efficiency of the proposed method, and the results showed that the entropy-based estimator achieves a significant reduction in the mean squared error ranging between 56% and 60% compared to the maximum likelihood estimation, in addition to having higher robustness against outliers. At a contamination rate of 10%, the mean squared error increased by 18.7% for the proposed estimator compared to 65.1% for the maximum likelihood estimation. This improvement is theoretically explained by the effect of entropy regulation in reducing the influence function, which enhances numerical stability and limits the impact of outliers on estimation accuracy. The model was also applied to real data involving 847 borrowers from three branches of the Rafidain Bank in Karbala Governorate for the purpose of predicting the type of bank, achieving a prediction accuracy of 64% with a 27% reduction in the standard error compared to the maximum likelihood method, confirming the efficiency of the improved algorithm in estimation and prediction.