David Witmer

Sloppy models, parameter uncertainty, and the role of experimental design

Computational models are increasingly used to understand and predict complex biological phenomena. These models contain many unknown parameters, at least some of which are difficult to measure directly, and instead are estimated by fitting to time-course data. Previous work has suggested that even with precise data sets, many parameters are unknowable by trajectory measurements. We examined this question in the context of a pathway model of epidermal growth factor (EGF) and neuronal growth factor (NGF) signaling. Computationally, we examined a palette of experimental perturbations that included different doses of EGF and NGF as well as single and multiple gene knockdowns and overexpressions. While no single experiment could accurately estimate all of the parameters, experimental design methodology identified a set of five complementary experiments that could. These results suggest optimism for the prospects for calibrating even large models, that the success of parameter estimation is intimately linked to the experimental perturbations used, and that experimental design methodology is important for parameter fitting of biological models and likely for the accuracy that can be expected from them.

Effects of atherogenic diet on hepatic gene expression across mouse strains

Diets high in fat and cholesterol are associated with increased obesity and metabolic disease in mice and humans. To study the molecular basis of the metabolic response to dietary fat, 10 inbred strains of mice were fed atherogenic high-fat and control low-fat diets. Liver gene expression and whole animal phenotypes were measured and analyzed in both sexes. The effects of diet, strain, and sex on gene expression were determined irrespective of complex processes, such as feedback mechanisms, that could have mediated the genomic responses. Global gene expression analyses demonstrated that animals of the same strain and sex have similar transcriptional profiles on a low-fat diet, but strains may show considerable variability in response to high-fat diet. Functional profiling indicated that high-fat feeding induced genes in the immune response, indicating liver damage, and repressed cholesterol biosynthesis. The physiological significance of the transcriptional changes was confirmed by a correlation analysis of transcript levels with whole animal phenotypes. The results found here were used to confirm a previously identified quantitative trait locus on chromosome 17 identified in males fed a high-fat diet in two crosses, PERA x DBA/2 and PERA x I/Ln. The gene expression data and phenotype data have been made publicly available as an online tool for exploring the effects of atherogenic diet in inbred mouse strains (http://cgd-array.jax.org/DietStrainSurvey).

How to Refute a Random CSP

Let P be a k-ary predicate over a finite alphabet. Consider a random CSP(P) instance I over n variables with m constraints. When m ≫ n the instance will be unsatisfiable with high probability and we want to find a certificate of unsatisfiability. When P is the 3-ary OR predicate, this is the well-studied problem of refuting random 3-SAT formulas and an efficient algorithm is known only when m ≫ n3/2. Understanding the density required for refutation of other predicates is important in cryptography, proof complexity, and learning theory. Previously, it was known that for a k-ary predicate, having m ≫≫n[k/2] constraints suffices for refutation. We give a criterion for predicates that often yields efficient refutation algorithms at much lower densities. Specifically, if P fails to support a t-wise uniform distribution, then there is an efficient algorithm that refutes random CSP(P) instances whp when m ≫ nt/2. Indeed, our algorithm will “somewhat strongly” refute the instance I, certifying Opt(I) ≤ 1 - Ωk(1). If t = k then we get the strongest possible refutation, certifying Opt(I) ≤ E[P] + o(1). This last result is new even for random k-SAT. Prior work on SDP hierarchies has given some evidence that efficient refutation of random CSP(P) may be impossible when m ≫ nt/2; thus there is an indication that our algorithms dependence on m is optimal for every P, at least in the context of SDP hierarchies. As an application of our result, we falsify assumptions used to show hardness-of-learning in recent work of Daniely, Linial, and Shalev-Shwartz.

Sparsest cut on bounded treewidth graphs: algorithms and hardness results

We give a 2-approximation algorithm for the non-uniform Sparsest Cut problem that runs in time nO(k), where k is the treewidth of the graph. This improves on the previous 22k-approximation in time poly(n) 2O(k) due to Chlamtac et al. [18]. To complement this algorithm, we show the following hardness results: If the non-uniform Sparsest Cut has a ρ-approximation for series-parallel graphs (where ρ ≥ 1), then the MaxCut problem has an algorithm with approximation factor arbitrarily close to 1/ρ. Hence, even for such restricted graphs (which have treewidth 2), the Sparsest Cut problem is NP-hard to approximate better than 17/16 - ε for ε > 0; assuming the Unique Games Conjecture the hardness becomes 1/αGW - ε. For graphs with large (but constant) treewidth, we show a hardness result of 2 - ε assuming the Unique Games Conjecture. Our algorithm rounds a linear program based on (a subset of) the Sherali-Adams lift of the standard Sparsest Cut LP. We show that even for treewidth-2 graphs, the LP has an integrality gap close to 2 even after polynomially many rounds of Sherali-Adams. Hence our approach cannot be improved even on such restricted graphs without using a stronger relaxation.

Reply to Comment on “Sloppy models, parameter uncertainty, and the role of experimental design”

We welcome the commentary from Chachra, Transtrum, and Sethna1 regarding our paper “Sloppy models, parameter uncertainty, and the role of experimental design,”2 as their intriguing work shaped our thinking in this area.3 Sethna and colleagues introduced the notion of sloppy models, in which the uncertainty in the values of some combinations of parameters is many orders of magnitude greater than others.4 In our work we explored the extent to which large parameter uncertainties are an intrinsic characteristic of systems biology network models, or whether uncertainties are instead closely related to the collection of experiments used for model estimation. We were gratified to find the latter result –– that parameters are in principle knowable, which is important for the field of systems biology. The work also showed that small parameter uncertainties can be achieved and that the process can be greatly accelerated by using computational experimental design approaches5–9 deployed to select sets of experiments that effectively exercise the system in complementary directions.2 The comment by Chachra et al. does not disagree with any of these points, but rather emphasizes two quantitative issues.1 Firstly, even when all parameter combinations have small uncertainties, the fit model can still be sloppy in that some parameter combinations are known orders of magnitude better than others (in our paper this ratio of uncertainties was around 300).1, 2 This is certainly correct, although to truly ask whether sloppiness is inherent in the model or is due to the experiments used for fitting, one should apply optimal experimental design to the objective of minimizing sloppiness. We have done an initial trial and were able to establish all parameter directions to near 10% or less uncertainty while reducing the ratio to 55, and we expect that with more effort further reductions could be achieved. Secondly, Chachra et al. commented that the quantity of data required to achieve small parameter uncertainties could be large.1 We certainly agree. In our paper we effectively used 3,000 individual measurements spread across five experimental perturbations (600 data points per experiment), each measurement with the relatively high precision of 10%, to fit just 48 parameters. A greater number of less precise experimental measurements would be required, but the number could be decreased if less precision in the fit parameters were required. As but one example of how this tradeoff plays out in the example used in our paper,2 if the total number of measurements were reduced from 600 data points per experiment to just 68, then 13 experimental perturbations are required. If the experimental uncertainty were then doubled from 10% to 20%, then the required number of perturbations would increase further to 33, but if the desired parameter uncertainty then were to similarly double from 10% to 20%, the number of experimental perturbations would return to 13 (and is, in fact, a mathematically equivalent problem with an identical set of solutions). It should be noted that we did not optimize the selection of species or time points to measure, although it is known that not all contribute equally,7–9 and our techniques applied to species and time point selection could presumably lead to significant data reductions. This consideration, coupled with dramatic increases in capacities of new technologies for making large-scale measurements in systems biology, makes it is less likely that data limitations will be determining. Moreover, the application of optimal experimental design computations to plan experimental campaigns should then be increasingly useful to strategically plan experiments. This example emphasizes the tradeoff between the number of measurements per experimental perturbation and the number of experimental perturbations. Depending on the relative effort of producing one or the other, an appropriately customized campaign could be developed. Finally, it remains an unanswered question just how accurately parameters need to be known to achieve accurate predictions. One of the notions arising from the concept of model sloppiness is that some predictions can be made quite accurately with very inaccurate parameters,3 but this is of little use without a method for knowing when one is in this situation. Propagation of parameter uncertainty is one approach to estimating prediction accuracy. By clarifying the link between parameter uncertainty and experimental conditions, our work points to another approach.2 Because the link between parameter uncertainty and experimental conditions extends to experiments that have yet to be done (namely, predictions), combinations of experimental perturbations and measurements that would not further reduce parameter uncertainty significantly are expected to be well represented by the current model and should be relatively high confidence predictions. We are investigating this relationship in more detail.

Efficiency analysis of formally verified adaptive cruise controllers

We consider an adaptive cruise control system in which control decisions are made based on position and velocity information received from other vehicles via V2V wireless communication. If the vehicles follow each other at a close distance, they have better wireless reception but collisions may occur when a follower car does not receive notice about the decelerations of the leader car fast enough to react before it is too late. If the vehicles are farther apart, they would have a bigger safety margin, but the wireless communication drops out more often, so that the follower car no longer receives what the leader car is doing. In order to guarantee safety, such a system must return control to the driver if it does not receive an update from a nearby vehicle within some timeout period. The value of this timeout parameter encodes a tradeoff between the likelihood that an update is received and the maximum safe acceleration. Combining formal verification techniques for hybrid systems with a wireless communication model, we analyze how the expected efficiency of a provably-safe adaptive cruise control system is affected by the value of this timeout.

Lower Bounds for CSP Refutation by SDP Hierarchies

For a k-ary predicate P, a random instance of CSP(P) with n variables and m constraints is unsatisfiable with high probability when m >= O(n). The natural algorithmic task in this regime is refutation: finding a proof that a given random instance is unsatisfiable. Recent work of Allen et al. suggests that the difficulty of refuting CSP(P) using an SDP is determined by a parameter cmplx(P), the smallest t for which there does not exist a t-wise uniform distribution over satisfying assignments to P. In particular they show that random instances of CSP(P) with m >> n^{cmplx(P)/2} can be refuted efficiently using an SDP. In this work, we give evidence that n^{cmplx(P)/2} constraints are also necessary for refutation using SDPs. Specifically, we show that if P supports a (t-1)-wise uniform distribution over satisfying assignments, then the Sherali-Adams_+ and Lovasz-Schrijver_+ SDP hierarchies cannot refute a random instance of CSP(P) in polynomial time for any m <= n^{t/2-epsilon}.