Simon Thomas

Cross-validation pitfalls when selecting and assessing regression and classification models

BackgroundWe address the problem of selecting and assessing classification and regression models using cross-validation. Current state-of-the-art methods can yield models with high variance, rendering them unsuitable for a number of practical applications including QSAR. In this paper we describe and evaluate best practices which improve reliability and increase confidence in selected models. A key operational component of the proposed methods is cloud computing which enables routine use of previously infeasible approaches.MethodsWe describe in detail an algorithm for repeated grid-search V-fold cross-validation for parameter tuning in classification and regression, and we define a repeated nested cross-validation algorithm for model assessment. As regards variable selection and parameter tuning we define two algorithms (repeated grid-search cross-validation and double cross-validation), and provide arguments for using the repeated grid-search in the general case.ResultsWe show results of our algorithms on seven QSAR datasets. The variation of the prediction performance, which is the result of choosing different splits of the dataset in V-fold cross-validation, needs to be taken into account when selecting and assessing classification and regression models.ConclusionsWe demonstrate the importance of repeating cross-validation when selecting an optimal model, as well as the importance of repeating nested cross-validation when assessing a prediction error.

The role of multiple enzyme activation in metabolic flux control

Abstract Over recent years, the concept that flux through a metabolic pathway is controlled by a single rate-limiting enzyme has been challenged from a number of quarters. We have presented three lines of evidence that the control of pathway flux by activation of multiple several steps, including steps responsible for consumption of pathway products, is an important feature of physiological flux control in response to external stimmuli. Theoretical tools exist that can be used to analyze the distribution of flux control between the steps involved in signal transduction and enzyme activation.

APPLICATION OF A GENERIC PHYSIOLOGICALLY BASED PHARMACOKINETIC MODEL TO THE ESTIMATION OF XENOBIOTIC LEVELS IN RAT PLASMA

The routine assessment of xenobiotic in vivo kinetic behavior is currently dependent upon data obtained through animal experimentation, although in vitro surrogates for determining key absorption, distribution, metabolism, and elimination properties are available. Here we present a unique, generic, physiologically based pharmacokinetic (PBPK) model and demonstrate its application to the estimation of rat plasma pharmacokinetics, following intravenous dosing, from in vitro data alone. The model was parameterized through an optimization process, using a training set of in vivo data taken from the literature and validated using a separate test set of in vivo discovery compound data. On average, the vertical divergence of the predicted plasma concentrations from the observed data, on a semilog concentration-time plot, was approximately 0.5 log unit. Around 70% of all the predicted values of a standardized measure of area under the concentration-time curve (AUC) were within 3-fold of the observed values, as were over 90% of the training set t1/2 predictions and 60% of those for the test set; however, there was a tendency to overpredict t1/2 for the test set compounds. The capability of the model to rank compounds according to a given criterion was also assessed: of the 25% of the test set compounds ranked by the model as having the largest values for AUC, 61% were correctly identified. These validation results lead us to conclude that the generic PBPK model is potentially a powerful and cost-effective tool for predicting the mammalian pharmacokinetics of a wide range of organic compounds, from readily available in vitro inputs only.

Simulation Modelling of Human Intestinal Absorption using Caco-2 Permeability and Kinetic Solubility Data for Early Drug Discovery

Measurement of permeation across a monolayer of the human adenocarcinoma cell line, Caco-2, is a popular surrogate for a compounds permeation across the human intestinal epithelium. Taken alone, however, Caco-2 permeability has certain limitations in the prediction of the extent of absorption of an orally-administered compound, because it does not take into account confounding factors such as solubility and dissolution in the gastrointestinal (GI) tract fluids. A simulation model is described that uses Caco-2 permeability measured in the apical to basolateral direction plus kinetic solubility in buffered solution (both measured at pH 7.4) to predict human intestinal absorption. The model features novel treatment of time-varying fluid volume in the GI tract, as a consequence of secretions into, and absorption of fluid from, the upper part of the GI tract. The model has been trained and cross-validated with data for 120 combinations of compound and dose. It has superior predictive power to recently published simulation and quantitative structure property relationship models, and is suitable for high-throughput screening during lead identification and lead optimisation in drug discovery.

MetaCon - A Computer Program for the Algebraic Evaluation of Control Coefficients of Metabolic Networks

A computer program, MetaCon, has been developed which fully automates all stages of carrying out a control analysis to give values for the flux control, concentration control and branch-point distribution control coefficients of a metabolic pathway. The program input is a text file containing processing instructions and the pathway details. The pathway is analyzed for the presence of branches and moiety-conserved cycles. The results of this stage are used along with inferred and additional elasticity terms to build an elasticity matrix of the form of the Fell and Sauro matrix method1,2, though the process involves symbolic evaluation of Reder’s equations3. Inversion of the matrix yields the expressions for the control coefficients in terms of the elasticities, metabolite concentrations and fluxes which appear in the elasticity matrix. All calculations are carried out algebraically rather than numerically, so that if the values of one or more variables are unknown, the control coefficients will be produced as polynomials in terms of the unknown(s).