The best linear sub-model selection with entropy based regularization

Abstract

This paper presents a unified framework for linear sub-model selection based on the principle of entropy maximization. By maximizing entropy under suitable constraints, we derive generalized normal distributions that define both the error model and the prior on regression weights, linking distributional assumptions directly to regularization penalties. The resulting quasi-likelihood formulation integrates likelihood maximization with regularization and allows flexible control of robustness and sparsity through two shape parameters. We develop a convex optimization approach for parameter estimation and propose a heuristic binary optimization algorithm for efficient sub-model selection guided by Bayesian Information Criteria (BIC). The framework is experimentally evaluated on simulated regression tasks with both binary and continuous weights under varying noise conditions. The experiments systematically examine the influence of the distributional parameters and on the accuracy and stability of sub-model identification. The results show that for low and moderate noise levels, the proposed method reliably recovers the true sub-model and achieves information criterion values comparable to those of established regularization techniques such as ridge and lasso regression.