Erricos John Kontoghiorghes

Branch-and-Bound Algorithms for Computing the Best-Subset Regression Models

An efficient branch-and-bound algorithm for computing the best-subset regression models is proposed. The algorithm avoids the computation of the whole regression tree that generates all possible subset models. It is formally shown that if the branch-and-bound test holds, then the current subtree together with its right-hand side subtrees are cut. This reduces significantly the computational burden of the proposed algorithm when compared to an existing leaps-and-bounds method which generates two trees. Specifically, the proposed algorithm, which is based on orthogonal transformations, outperforms by O(n3) the leaps-and-bounds strategy. The criteria used in identifying the best subsets are based on monotone functions of the residual sum of squares (RSS) such as R2, adjusted R2, mean square error of prediction, and Cp. Strategies and heuristics that improve the computational performance of the proposed algorithm are investigated. A computationally efficient heuristic version of the branch-and-bound strategy which decides to cut subtrees using a tolerance parameter is proposed. The heuristic algorithm derives models close to the best ones. However, it is shown analytically that the relative error of the RSS, and consequently the corresponding statistic, of the computed subsets is smaller than the value of the tolerance parameter which lies between zero and one. Computational results and experiments on random and real data are presented and analyzed.

Efficient algorithms for computing the best subset regression models for large-scale problems

Several strategies for computing the best subset regression models are proposed. Some of the algorithms are modified versions of existing regression-tree methods, while others are new. The first algorithm selects the best subset models within a given size range. It uses a reduced search space and is found to outperform computationally the existing branch-and-bound algorithm. The properties and computational aspects of the proposed algorithm are discussed in detail. The second new algorithm preorders the variables inside the regression tree. A radius is defined in order to measure the distance of a node from the root of the tree. The algorithm applies the preordering to all nodes which have a smaller distance than a certain radius that is given a priori. An efficient method of preordering the variables is employed. The experimental results indicate that the algorithm performs best when preordering is employed on a radius of between one quarter and one third of the number of variables. The algorithm has been applied with such a radius to tackle large-scale subset-selection problems that are considered to be computationally infeasible by conventional exhaustive-selection methods. A class of new heuristic strategies is also proposed. The most important of these is one that assigns a different tolerance value to each subset model size. This strategy with different kind of tolerances is equivalent to all exhaustive and heuristic subset-selection strategies. In addition the strategy can be used to investigate submodels having noncontiguous size ranges. Its implementation provides a flexible tool for tackling large scale models.

An alternative approach for the numerical solution of seemingly unrelated regression equations models

Abstract An alternative approach to compute the coefficients of a Seemingly Unrelated Regression Equations (SURE) model is proposed. Orthogonal transformations are employed to avoid the difficulties in directly computing the inverse of the variance-covariance matrix (or its estimate) which often lead to unnecessary loss of accuracy. The solution of the special SURE model where the problem is constrained so that the regressors in each equation contain the regressors in previous equations as a proper subset, is considered in detail.

Parallel algorithms for computing all possible subset regression models using the QR decomposition

Efficient parallel algorithms for computing all possible subset regression models are proposed. The algorithms are based on the dropping columns method that generates a regression tree. The properties of the tree are exploited in order to provide an efficient load balancing which results in no inter-processor communication. Theoretical measures of complexity suggest linear speedup. The parallel algorithms are extended to deal with the general linear and seemingly unrelated regression models. The case where new variables are added to the regression model is also considered. Experimental results on a shared memory machine are presented and analyzed.

A comparative study of algorithms for solving seemingly unrelated regressions models

The computational efficiency of various algorithms for solving seemingly unrelated regressions (SUR) models is investigated. Some of the algorithms adapt known methods; others are new. The first transforms the SUR model to an ordinary linear model and uses the QR decomposition to solve it. Three others employ the generalized QR decomposition to solve the SUR model formulated as a generalized linear least-squares problem. Strategies to exploit the structure of the matrices involved are developed. The algorithms are reconsidered for solving the SUR model after it has been transformed to one of smaller dimensions.

A graph approach to generate all possible regression submodels

A regression graph to enumerate and evaluate all possible subset regression models is introduced. The graph is a generalization of a regression tree. All the spanning trees of the graph are minimum spanning trees and provide an optimal computational procedure for generating all possible submodels. Each minimum spanning tree has a different structure and characteristics. An adaptation of a branch-and-bound algorithm which computes the best-subset models using the regression graph framework is proposed. Experimental results and comparison with an existing method based on a regression tree are presented and discussed.

Efficient strategies for deriving the subset VAR models

Abstract.Algorithms for computing the subset Vector Autoregressive (VAR) models are proposed. These algorithms can be used to choose a subset of the most statistically-significant variables of a VAR model. In such cases, the selection criteria are based on the residual sum of squares or the estimated residual covariance matrix. The VAR model with zero coefficient restrictions is formulated as a Seemingly Unrelated Regressions (SUR) model. Furthermore, the SUR model is transformed into one of smaller size, where the exogenous matrices comprise columns of a triangular matrix. Efficient algorithms which exploit the common columns of the exogenous matrices, sparse structure of the variance-covariance of the disturbances and special properties of the SUR models are investigated. The main computational tool of the selection strategies is the generalized QR decomposition and its modification.

Seemingly unrelated regression model with unequal size observations: computational aspects

The computational solution of the seemingly unrelated regression model with unequal size observations is considered. Two algorithms to solve the model when treated as a generalized linear least-squares problem are proposed. The algorithms have as a basic tool the generalized QR decomposition (GQRD) and efficiently exploit the block-sparse structure of the matrices. One of the algorithms reduces the computational burden of the estimation procedure by not computing explicitly the RQ factorization of the GQRD. The maximum likelihood estimation of the model when the covariance matrix is unknown is also considered.

An Exact Least Trimmed Squares Algorithm for a Range of Coverage Values

A new algorithm to solve exact least trimmed squares (LTS) regression is presented. The adding row algorithm (ARA) extends existing methods that compute the LTS estimator for a given coverage. It employs a tree-based strategy to compute a set of LTS regressors for a range of coverage values. Thus, prior knowledge of the optimal coverage is not required. New nodes in the regression tree are generated by updating the QR decomposition of the data matrix after adding one observation to the regression model. The ARA is enhanced by employing a branch and bound strategy. The branch and bound algorithm is an exhaustive algorithm that uses a cutting test to prune nonoptimal subtrees. It significantly improves over the ARA in computational performance. Observation preordering throughout the traversal of the regression tree is investigated. A computationally efficient and numerically stable calculation of the bounds using Givens rotations is designed around the QR decomposition, avoiding the need to explicitly update the triangular factor when an observation is added. This reduces the overall computational load of the preordering device by approximately half. A solution is proposed to allow preordering when the model is underdetermined. It employs pseudo-orthogonal rotations to downdate the QR decomposition. The strategies are illustrated by example. Experimental results confirm the computational efficiency of the proposed algorithms. Supplemental materials (R package and formal proofs) are available online.

Optimisation, econometric and financial analysis

Optimisation Models and Methods: A Supply Chain Network Perspective for Electric Power Generation, Supply, Transmission, and Consumption.- Worst-Case Modelling for Management Decisions under Incomplete Information, with Application to Electricity Spot Markets.- An Approximate Winner Determination Algorithm for Hybrid Procurement Mechanisms in Logistics.- Proximal-ACCPM: A Versatile Oracle Based Optimization Method.- A Survey of Different Integer Programming Formulations of the Travelling Salesman Problem.- Econometric Modelling and Prediction: The Threshold Accepting Optimization Algorithm in Economics and Statistics.- The Autocorrelation Functions in SETARMA Models.- Trend Estimation and De-Trending.- Non-Dyadic Wavelet Analysis.- Measuring Core Inflation by Multivariate Structural Time Series Models.- Financial Modelling: Random Portfolios for Performance Measurement.- Real Options with Random Controls, Rare Events, and Risk-to-Ruin.