A-ComVar: A Flexible Extension of Common Variance Design
AA-ComVar: A Flexible Extension of Common Variance Designs
Shrabanti ChowdhuryDepartment of Genetics and Genomic SciencesIcahn School of Medicine at Mount SinaiNew York, NY 10029Joshua LukemireDepartment of Biostatistics and BioinformaticsEmory UniversityAtlanta, GA 30322 Abhyuday MandalDepartment of StatisticsUniversity of GeorgiaAthens, GA 30606November 11, 2019
Abstract
We consider nonregular fractions of factorial experiments for a class of linear models. Thesemodels have a common general mean and main effects, however they may have different 2-factor interactions. Here we assume for simplicity that 3-factor and higher order interactionsare negligible. In the absence of a priori knowledge about which interactions are important, itis reasonable to prefer a design that results in equal variance for the estimates of all interactioneffects to aid in model discrimination. Such designs are called common variance designs and canbe quite challenging to identify without performing an exhaustive search of possible designs. Inthis work, we introduce an extension of common variance designs called approximate commonvariance, or A-ComVar designs. We develop a numerical approach to finding A-ComVar designsthat is much more efficient than an exhaustive search. We present the types of A-ComVardesigns that can be found for different number of factors, runs, and interactions. We furtherdemonstrate the competitive performance of both common variance and A-ComVar designsusing several comparisons to other popular designs in the literature.
Keywords:
Class of Models, Model Identification, Common Variance, Placket-Burman, Adap-tive Lasso, Approximate Common Variance, Genetic Algorithm1 a r X i v : . [ s t a t . C O ] N ov Introduction
Fractional factorial designs are widely used in many scientific investigations because they providea systematic and statistically valid strategy for studying how multiple factors impact a responsevariable through main effects and interactions. When several factors are to be tested, often theexperimenter does not know which factors have important interactions. Instead, the experimenterwill need to perform model selection after conducting the experiment to identify important interac-tions. Generally this process will involve fitting different models under consideration and examiningstatistical significance of the interaction terms. Some techniques have been developed concerningfinding efficient fractional factorial plans for this purpose. There is a rich literature on identifica-tion and discrimination to find the model best describing the data (Srivastava, 1976; Srivastavaand Ghosh, 1976; Srivastava and Gupta, 1979).Consider a design with m factors of interest. Following Ghosh and Chowdhury (2017), considerthe following class of s candidate models for describing the relationship between p ( ≤ m ) of the m factors and the n × y , E ( y ) = β j n + X β + X ( i )2 β i , i = 1 , . . . , s (1) V ar ( y ) = σ I , where n is the number of runs, β is the general mean, j n is a vector of ones, β is the vector of p main effects that are common in all s models. The other parameters, β i , are specific for the i th model and hence β i (cid:54) = β i (cid:48) for i (cid:54) = i (cid:48) , i = 1 , . . . , s . We call these parameters “uncommonparameters.” The design matrices X and X ( i )2 correspond to the main effects and i th set of 2-factor interactions, respectively. Following Ghosh and Flores (2013) and Ghosh and Chowdhury(2017) we consider the situation of p = m for generating A-ComVar designs using our proposedapproach, described in Section 4. However, we consider models with p ≤ m cases in our examplescomparing the performance of our proposed designs with some popular designs from the literature.Under the above setup, model selection consists of identifying the correct i from the s candidatemodels. This process is complicated by the fact that the variance estimates for the uncommonparameters are generally not the same, which can pre-bias the experiment towards identifying2ertain interactions as significant over others, i.e. making some i more likely to be selected thanothers regardless of the true underlying model. To address this issue, Ghosh and Flores (2013)introduced the notion of common variance designs for a single uncommon parameter. These designsestimate the uncommon parameter in all models with equal variance, which is desirable in theabsence of any a priori information about the true model. Ghosh and Chowdhury (2017) generalizedthis concept of common variance to k ( k ≥
1) uncommon parameters in each model in the class.Under the situation of k >
1, Ghosh and Chowdhury (2017) defined a common variance design tobe the one satisfying | X ( i ) (cid:48) X ( i ) | to be a constant, for all i , X ( i ) = (cid:16) j n , X , X ( i )2 (cid:17) .The concept of variance-balancedness is not totally new. Different types of “variance-balanceddesigns” estimating all or some of the treatment-contrasts with identical variance were developedby Calvin (1986), Cheng (1986), Gupta and Jones (1983), Hedayat and Stufken (1989), Khatri(1982), Mukerjee and Kageyama (1985), among others.While common variance designs have been identified for two and three level factorial experimentswith a single 2-factor interaction (Ghosh and Flores, 2013; Ghosh and Chowdhury, 2017), it remainsto develop a method which can find them for general number of factors and interactions. To date,these designs have been found using exhaustive searches, which becomes prohibitively expensiveas the number of factors and runs increases. This leads us to introduce approximate commonvariance (A-ComVar) designs, which relax the requirement that the variance of the uncommonparameters be exactly equal. We introduce an objective function that allows us to rank designsunder consideration, and we develop a genetic algorithm for searching for these designs. Moreover,we investigate the performance of both common variance and A-ComVar designs for model selectionusing the adaptive lasso regression technique in simulation (Kane and Mandal, 2019). We findcomparable performance of common variance and A-ComVar designs to Placket-Burman designs,which further demonstrates the usefulness of designs that prioritize having a similar variance forthe uncommon parameters in the model.The rest of the article is organized as follows. In Section 2 we present the current state ofknowledge for both two-level and three-level common variance designs. For three-level designs wealso present the exhaustive search result for m = 3. In Section 3 we introduce our numericalapproach for finding A-ComVar designs. In Section 4 we conduct extensive studies to both (i)examine our numerical approach’s ability to find A-ComVar designs as we increase the number3f factors and number of interactions in the model and (ii) compare these A-ComVar designs topotential competitor designs from the literature. Finally, Section 5 contains some discussion ofthe results and some future directions for our work. The Appendix contains an illustration of ourgenetic algorithm, as well as Tables corresponding to all results. The term “common variance” for the class of variance-balanced designs was first introduced inGhosh and Flores (2013). As a more stringent criteria, the authors also introduced the concept ofoptimum common variance (OPTCV), which is satisfied by designs having the smallest value ofcommon variance in a class of common variance designs with p ≤ m factors and n runs. Severalcharacterizations of common variance and optimal common variance designs were presented thatprovide efficient ways for checking the common variance or OPTCV property of a given design.These characterizations were obtained in terms of the projection matrix, eigenvalues of the modelmatrix, balancedness, and orthogonal properties of the designs. In Corollary 1 of Ghosh and Flores(2013), they stated one sufficient condition of common variance designs in terms of equality of thevectors of eigen values of X ( i ) (cid:48) X ( i ) , X ( i ) = (cid:16) j n , X , X ( i )2 (cid:17) , for all i . We present one design inTable 3 from Ghosh and Flores (2013) for m = 5 and n = 12, that has identical vectors of eigenvalues for all i . In Section 4.3 we compare the performance of this particular design with that ofPlackett-Burman design for model selection to demonstrate further usefulness of such designs.In their work, Ghosh and Flores (2013) presented several general series of designs with thecommon variance property. For example, they identified two fold-over designs with the commonvariance property with all m factors of the design and n = 2 m and n = 2 m + 2 runs respectively: d (2 m ) m = I m − J m − I m + J m . (2 m +2) m = j (cid:48) m − j (cid:48) m I m − J m − I m + J m . As reported in Ghosh and Flores (2013), both of these designs are balanced arrays of full strengthand orthogonal arrays of strength 1, for all m . Moreover, the design d (2 m ) m is OPTCV for m = 4and d (2 m +2) m is OPTCV for m = 3. Ghosh and Chowdhury (2017) presented common variance designs for 3 m fractional factorial exper-iments. Consider the following model for a 3 m factorial experiment, with one 2-factor interactioneffect in the model, i.e. k = 1: E ( y ) = β j n + X β + X ( i )2 β i , V ar ( y ) = σ . A design for such an experiment would have the common variance property iff
V ar ( ˆ β i ) σ is constantfor all i = 1 , . . . , (cid:0) m (cid:1) , for the situation p = m .Ghosh and Chowdhury (2017) presented two general series of 3 m fractional factorial commonvariance designs d and d with n runs. The design d has a common variance value given by V ar (cid:16) ˆ β ( i )2 (cid:17) σ = − m + m , for m ≥ n = 2 m + 2 runs, while design d has a common variancevalue given by V ar (cid:16) ˆ β ( i )2 (cid:17) σ = m m − , for m ≥ n = 3 m . Also, the design d is efficient commonvariance (ECV, as termed in Ghosh and Chowdhury (2017)) design for m = 2, and design d isECV for m = 3.Ghosh and Chowdhury (2017) also presented several sufficient conditions for general fractionalfactorial designs to have the common variance property, including the special case for 3 m designs interms of the projection matrix of the design and the columns of 2-factor interaction. For example, adesign is common variance if (i) P X ( i )2 = P X ( i )2 , for i , i ∈ { , . . . , s } , where P is the projectionmatrix defined as I n − X ( X (cid:48) X ) − X (cid:48) , and X contains the columns corresponding to the generalmean and main effects from the model matrix X ( i ) = (cid:16) j n , X , X ( i )2 (cid:17) , and X ( i )2 corresponds to the5 th i , i ∈ { , . . . , s } , (i) (cid:16) X ( i )2 ± X ( i )2 (cid:17) belongs to the column space of X and (ii) X ( i )2 = F X ( i )2 holds, where the permutation matrix F obtained from the identity matrix satisfies F (cid:48) P F = P .For 3 fractional factorial experiment, Chowdhury (2016) conducted a complete search of com-mon variance designs for n = 8 to n = 27, since n = 8 is the minimum number of runs needed toestimate all the parameters considering all 3 factors are present in the model (one general mean,6 main effects, one 2-factor interaction effect). The results of this search are presented in Table1. The complete search revealed that common variance designs only exist for n = 8 , , ,
11 for3 factorial experiments. For each of the runs multiple groups of common variance designs wereobtained, having different common variance values, among which 32 designs for n = 11; 48 designsfor n = 10; 8256 designs for n = 9; and 9600 designs for n = 8, are the efficient common variancedesigns giving the minimum value of common variance in the respective classes. Possible Satisfying No. of No. of No. of CV n Designs Rank Condition Non-CV CV designs with CV value= (cid:0) n (cid:1) ( | X (cid:48) X | >
0) designs designs this value11 13,037,895 6,926,898 6,924,772 2,096 32 0.21512,064 0.222210 8,436,285 2,792,387 2,775,747 16,640 48 0.256448 0.266716 0.283716,512 0.296316 0.40009 4,686,825 636,348 588,348 48,000 8,256 0.333332 0.381013,056 0.416726,640 0.444416 0.50008 2,220,075 49,628 23,340 26,288 9,600 0.666716,688 0.8889
Table 1: Complete search results for finding common variance designs for 3 factorial experiments.6 Identifying Common Variance Designs
Ghosh and Flores (2013) and Ghosh and Chowdhury (2017) presented some general series of designssatisfying the common variance property for two- and three-level factorial experiments obtainedvia exhaustive searches of the design space. Such searches become extremely computationallychallenging as the number of factors increases. For example, for a 3 factorial experiment with one2-factor interaction ( k = 1) the possible set of candidate designs with 8 runs is (cid:0) (cid:1) = 2220075, with9 runs is (cid:0) (cid:1) = 4686825, with 10 runs is (cid:0) (cid:1) = 8436285, and so on. For a 3 factorial experiment,the cardinality of this set increases to (cid:0) (cid:1) = 1 . × , even for the designs with the smallestpossible number of runs. This rapid growth in the size of the search space makes exhaustive searchesfor common variance designs impossible for anything but small design problems.In light of the difficulty in finding common variance designs, we introduce a class of approximatecommon variance (A-ComVar) designs. Instead of having exactly equal variance for the uncommonparameters for the s models under consideration, A-ComVar designs try to maximize the ratioof the minimum variance to the maximum variance. In doing so, they contain common variancedesigns as a sub-case where the minimum variance is exactly equal to the maximum variance. Inrelaxing the requirement that the variances be exactly equal, we are able to develop an objectivefunction and algorithm for identifying these A-ComVar designs without performing an exhaustivesearch. In this section we propose to use a genetic algorithm to identify A-ComVar Designs. We start bydefining an objective function that seeks to quantify our goal. Denote the variance of the interactioneffect for the i th model as σ β i and let σ β = s (cid:80) i σ β i . The objective function for designs thatdiscriminate between models with a single interaction term ( k = 1) is: f ( d ; φ ) = 1 /σ β φ × (cid:80) si =1 ( σ β i − σ β ) , (2)where σ β i is replaced by the determinant of the lower-right k × k sub-matrix of the inverse of the7isher information matrix for k >
1, which bears some similarity to the idea behind D -optimaldesign of experiments. The value of the objective function increases as the variance of the estimatesdecreases through the numerator, encouraging designs with small variances for the interactionterms. However, this value is also strongly penalized towards zero as the individual model variancesmove away from the average model variance. The strength of this penalty is controlled by thetuning parameter φ , which we recommend setting to a very large value. In our experiments wefound φ = 10 × to be adequate. The φ parameter is just to force differences in variance acrossmodels under consideration to “cost” more than the potential variance improvement from a designunder some subset of those models, and thus setting it to any suitable large value should suffice.Taken together the numerator allows us to differentiate between designs with common varianceto select the better one, and the denominator encourages common variance designs by penalizingdiffering variance under alternative models under consideration.This maximization approach will prefer A-ComVar designs with exactly common variance. Ofcourse, in many experimental situations a common variance design may not exist. For example inthe exhaustive search, Chowdhury (2016) found that common variance designs did not exist for 3 experiments for 13 runs. This leads us to the principal advantage of our approach: when a commonvariance design does not exist we can still find designs with variance that is as close as possible tobeing equal. To assess the quality of an A-ComVar design, we define the A-ComVar ratio r ACV = min i { var ( ˆ β i ) } max i { var ( ˆ β i ) } . (3)Clearly when a design has common variance, r ACV = 1. When a design does not have commonvariance, r ACV gives us an idea of how far we are from common variance. For example, if r ACV = 0 . Algorithm 1
Pseudo-code for the genetic algorithm to find A-ComVar designs. function A-ComVarDESIGN (design problem, mutation prob., num replace, max iter., φ ) for Each chromosome do initialize chromosome to random design Calculate fitness end for while termination criteria not met do Identify worst num replace chromosomes Use a crossover to generate num replace new chromosomes Mutate the num replace new chromosomes
Replace the worst chromosomes with the num replace new chromosomes
Calculate fitness for new chromosomes end while end function Numerical Examples − Designs with One -Factor Interaction We conducted a series of experiments to investigate the ability of our approach to find A-ComVar de-signs and to gain a better understanding of when common variance designs can be found. We startedby examining designs with a single 2-factor interaction. We consider 2 m and 3 m experiments, with m = 4 , . . . , m = 3 , . . . , m experiments, we considered run sizes of n m = m + 2 , . . . , m + 11,and for the 3 m experiments, we considered run sizes of n m = 2 m + 2 , . . . , m + 11. For eachcombination of settings, we ran our genetic algorithm 100 times and stored the r ACV results. Thetuning parameters used were a mutation probability of 0 .
05 and a maximum of 10,000 iterations.Figure 1 displays the results for the 2 m cases and Figure 2 displays the results for the 3 m cases.We first note that our results are consistent with the findings of Ghosh and Chowdhury (2017), whoused exhaustive searches to identify common variance designs. For example, Ghosh and Chowdhury(2017) found that common variance designs exist for 3 designs with 8 runs, which agrees with theboxplots in the first panel of Figure 2. This supports our use of the genetic algorithm approach withthe objective function described above. Furthermore, in cases where the common variance designseither do not exist or could not be found, our approach was able to find designs that attempt toget as close as possible to common variance. For example, it is known from exhaustive searchesthat no common variance design exists for a 3 experiment with 12 runs. However, the proposedapproach was able to find designs where the smallest variance was greater than 0 . − Designs with Two -Factor Interactions For designs with multiple 2-factor interactions (i.e. k > var ( ˆ β i ) with the determinant of the block of the inverse of the Fisher informationmatrix corresponding to the interactions terms. That is, we take the determinant of the bottom-right k × k sub-matrix of var ( ˆ β ).To demonstrate the approach, we conducted another experiment with two 2-factor interactions(i.e. k = 2). We consider 2 m experiments, with m = 4 , . . . , p = m . We considered10 .00.10.20.30.40.50.60.70.80.91.0 6 7 8 9 10 11 12 13 14 15 Number of runs r A C V ll llllllllllllllllll llllllllll llllllll lllllllllllllllllllllllll llllllllllll lllllll ll Number of runs r A C V ll l lllllllllllllllllllllllll llllllllllllllllllllllllll llllllll lll Number of runs r A C V llllll llllllllllllllllllllllllllllllllllll ll ll Number of runs r A C V lllllllllllllllllllllll lllllll lll llll Number of runs r A C V llllllllllllllllll llllll lllllll llllllllllll l lllllllllllll lllllllll ll Number of runs r A C V Figure 1: Ratios r ACV for the 2 m case across 1000 replicates for each experimental setting.11 llllllllllll lllllllll llllllll llllllllllllllllllllllllllllllllllllllll llllllllllllllllllllll Number of runs r A C V llll llllllllllllllllllllllll lllllllllllllllllllllll llllllllll Number of runs r A C V llllllllllllllllll lllllllllllllllllllll lllllllllllllllll llllllllllllllll llll llllllll llllll ll lllll Number of runs r A C V llllll ll lllll lllll l l l Number of runs r A C V Figure 2: Ratios r ACV for the 3 m case across 1000 replicates for each experimental setting.12 .00.10.20.30.40.50.60.70.80.91.0 10 11 12 13 14 15 16 Number of runs r A C V lllllllllllllllllllllll lllllllllllllllllllll lll Number of runs r A C V llllllllllllllllll llllllllllll lllllllll Number of runs r A C V lllllllllllllllllllllllllll llllllllllllllllllllllllllll l l Number of runs r A C V Figure 3: Ratios r ACV for the 2 m case across 1000 replicates for each experimental setting with k = 2. 13un sizes of n m = m + 6 , . . . , m + 12. For each combination of settings, we ran our geneticalgorithm 100 times and stored the r ACV results. The tuning parameters used were a mutationprobability of 0 .
05 and a maximum of 10,000 iterations.Figure 3 shows the results. As before, we can see that in many cases the genetic algorithm isable to find common variance designs. In cases where common variance designs cannot be found,the approach is often able to identify a design resulting in relatively close to common variance. − Common Variance Design and A-ComVar Designs for ModelSelection
We next perform a series of studies to demonstrate the advantages of pursuing A-ComVar designs.We do this by considering data generated from a variety of true models and testing whether amodel selection procedure is able to identify the true model using observations collected using thedesigns under consideration. We used the adaptive lasso (Kane and Mandal, 2019) to fit the model.We chose the adaptive lasso method of Kane and Mandal (2019) because they showed that thistechnique is suitable for identifying the correct model for designs with complex aliasing and thatit outperforms other popular variable selection methods including the Dantzig Selector (Candeset al., 2007), LARS (Yuan et al., 2007), and the Nonnegative Garotte estimator (Breiman, 1995;Yuan et al., 2009).Our procedure is as follows. For a model with p active main effects, let F , . . . , F p denotethe active factors, which are selected at random from the set of all factors of the designs at eachreplication. The corresponding effects, β , . . . , β p , as well as any interaction effects, are set to beeither “big” or “small,” where “big” effects are drawn from a U (1 . , .
5) distribution and “small”effects from a U (0 . , . σ , is chosen, completing thespecification of the true underlying model. The total number of different models, effect sizes, anderror standard deviations considered can be found in any of Tables 8 −
14. The first column in eachtable corresponds to the true model under consideration, and the second column gives informationabout the strengths of the active effects (b − “big” and s − “small”). For example, row 25 of Table14 corresponds to a model with three active main effects ( F , F , and F ) as well as one activeinteraction ( F F ). Here, the second column tells us that F and F have “big” effects and F and F F have “small” effects. 14escription Design 1 Design 22 experiment with 12 runs Common Variance (Table 3) Plackett-Burman (Table 3)A-ComVar (Table 4, D ) Plackett-Burman (Table 3)A-ComVar (Table 4, D ) Ghosh & Tian (Table 5, D )A-ComVar (Table 4, D ) Bayes Optimal (Table 5, D )A-ComVar (Table 4, D ) Li & Nachtsheim (Table 5, D )3 experiment with 20 runs A-ComVar (Table 4, D ) CCD (Table 6, D )3 experiment with 18 runs A-ComVar (Table 4, D ) OME (Table 6, D )Table 2: Description of the designs used for Example 3. The tables listed in parenthesis are wherethe corresponding design is presented.Next, for each design under consideration, a data set is generated from the true underlyingmodel using the randomly selected factors of the design. A model is fit to this data set using theadaptive lasso, and we measure whether or not the true underlying model was identified. Thisprocess is then repeated 100 times for the same set of true active coefficients, and we store thepercentage of the times the correct model was identified.For each model, design, and error standard deviation under consideration, this process of ran-domly selecting active factors in the model, generating observations from the design, and measuringhow often the correct model is identified is repeated 50 times, resulting in 50 replicates per combina-tion of settings. Here each replicate is a measurement of the percentage of times the data obtainedusing the design was able to correctly identify the true underlying model. Table 2 displays a list ofthe model comparisons we made. In Tables 8 −
14 we report the average percentage of times (over50 replications) the correct model was identified by the respective designs.The results with model × variance × design breakdown can be found in Tables 8 −
14 in theAppendix. Figures 4 − d (12)5 design with 5 factors and 12 runs presented in Table 2 with PB design presented in Table 3. Hererepresents the common variance design d (12)5 and represents Plackett-Burman design.Common Variance vs Plackett-Burman . .
25 0 . .
75 1 1 .
25 1 . s P e r f o r m an c e . .
25 0 . .
75 1 1 .
25 1 . s P e r f o r m an c e . .
25 0 . .
75 1 1 .
25 1 . s P e r f o r m an c e seems to outperform the A-ComVar design in several cases. This is likely because the Ghosh andTian (2006) design is optimal w.r.t all six standard optimality criteria, and thus it is hard to beatits performance. However, designs of this quality cannot always be obtained for arbitrary numbersof factors or runs, thus one advantage of our numerical approach is that it can be used for caseswhere such designs cannot be obtained via exhaustive search or by using theoretical results. A B C D E -1 -1 -1 -1 11 1 -1 -1 -11 -1 1 -1 -11 -1 -1 1 -1-1 1 1 -1 -1-1 1 -1 1 -1-1 -1 1 1 -11 1 1 1 -11 1 1 -1 11 1 -1 1 11 -1 1 1 1-1 1 1 1 1
A B C D E F G H I J K d (12)5 with common variance for 5 factors and 12 runs (left) and Plackett-BurmanDesign with 11 factors and 12 runs (right). In this work we introduced A-ComVar Designs, an extension of common variance designs. Ourproposed approach addresses the difficulties associated with finding common variance designs viaexhaustive search. Through several examples, we demonstrated that the proposed algorithmic16igure 5: Comparison of model selection performance of two-level A-ComVar design with 5 factorsand 12 runs with other designs. Here represents our A-ComVar design with 2 levels,represents Placket-Burman design, represents a two-level design with 5 factors and 12 runsfrom Ghosh and Tian (2006), represents a two-level Bayes optimal design with 5 factors and 12runs from Bingham and Chipman (2007), represents a two-level design with 5 factors and 12runs from Li and Nachtsheim (2000).A-ComVar vs Placket-Burman . .
25 0 . .
75 1 1 .
25 1 . s P e r f o r m an c e . .
25 0 . .
75 1 1 .
25 1 . s P e r f o r m an c e . .
25 0 . .
75 1 1 .
25 1 . s P e r f o r m an c e A-ComVar vs Ghosh and Tian (2007) . .
25 0 . .
75 1 1 .
25 1 . s P e r f o r m an c e . .
25 0 . .
75 1 1 .
25 1 . s P e r f o r m an c e . .
25 0 . .
75 1 1 .
25 1 . s P e r f o r m an c e A-ComVar vs Bayes Optimal (Bingham and Chipman, 2007) . .
25 0 . .
75 1 1 .
25 1 . s P e r f o r m an c e . .
25 0 . .
75 1 1 .
25 1 . s P e r f o r m an c e . .
25 0 . .
75 1 1 .
25 1 . s P e r f o r m an c e A-ComVar vs Li and Nachtsheim (2000) . .
25 0 . .
75 1 1 .
25 1 . s P e r f o r m an c e . .
25 0 . .
75 1 1 .
25 1 . s P e r f o r m an c e . .
25 0 . .
75 1 1 .
25 1 . s P e r f o r m an c e A B C D E -1 -1 -1 1 1-1 -1 1 -1 1-1 -1 1 1 -1-1 -1 1 1 1-1 1 -1 -1 -1-1 1 1 1 11 -1 -1 -1 -11 -1 1 1 11 1 -1 -1 -11 1 -1 -1 11 1 -1 1 -11 1 1 -1 -1 D A B C D -1 -1 -1 -1-1 -1 -1 0-1 -1 0 0-1 -1 1 1-1 0 0 0-1 0 1 0-1 1 -1 -1-1 1 -1 0-1 1 0 1-1 1 1 -10 0 0 00 0 0 10 0 1 10 1 1 01 -1 -1 11 -1 1 -11 -1 1 01 0 0 11 1 -1 -11 1 1 0 D A B C D E -1 -1 -1 -1 -1-1 -1 1 1 0-1 1 -1 -1 -1-1 1 -1 1 -1-1 1 -1 1 0-1 1 0 1 1-1 1 1 1 -10 -1 0 0 -10 0 0 0 10 0 0 1 -11 -1 -1 -1 01 -1 -1 1 -11 -1 1 -1 -11 -1 1 -1 01 -1 1 0 01 -1 1 1 01 1 -1 -1 -11 1 1 1 0Table 4: (1) D (left): two-level A-ComVar design for k = 1 with m = 5 and n = 12, (2) D (middle): three-level A-ComVar design for k = 1 with m = 4 and n = 20 and (3) D (right):three-level A-ComVar design for k = 1, m = 7, n = 18, used in Exzmple 3. D A B C D E D A B C D E D A B C D E D (left): two-level Bayes optimal design with m = 5 and n = 12 Bingham and Chipman(2007), (2) D (middle): two-level design with m = 5 and n = 12 from Li and Nachesheim (2000)and (3) D (right): two-level design with m = 5, n = 12 from Ghosh and Tian (2006), used inExample 3. 18 A B C -1 -1 -11 -1 -1-1 1 -11 1 -1-1 -1 11 -1 1-1 1 11 1 1-1.682 0 01.682 0 00 -1.682 00 1.682 00 0 -1.6820 0 1.6820 0 00 0 00 0 00 0 00 0 00 0 0 D A B C D E F G -1 -1 -1 -1 -1 -1 -10 0 0 0 0 0 -11 1 1 1 1 1 -1-1 -1 0 1 0 1 -10 0 1 -1 1 -1 -11 1 -1 0 -1 0 -1-1 0 -1 1 1 0 00 1 0 -1 -1 1 01 -1 1 0 0 -1 0-1 1 1 -1 0 0 00 -1 -1 0 1 1 01 0 0 1 -1 -1 0-1 0 1 0 -1 1 10 1 -1 1 0 -1 11 -1 0 -1 1 0 1-1 1 0 0 1 -1 10 -1 1 1 -1 0 11 0 -1 -1 0 1 1Table 6: (1) D (left): Central Composite Design with m = 3 and n = 20, (2) D (middle):three-level orthogonal main effect plan with m = 7 and n = 18 used in Example 3.19igure 6: Comparison of model selection performance of three-level A-ComVar designs with centralcomposite and orthogonal main effect designs. Here represents our three-level A-ComVar designwith 4 factors and 20 runs, represents a central composite design with 3 factors and 20 runs,represents our three-level A-ComVar design with 7 factors and 18 runs, represents a orthogonalmain effect plan with 7 factors and 18 runs.3 level A-ComVar vs CCD . .
25 0 . .
75 1 1 .
25 1 . s P e r f o r m an c e . .
25 0 . .
75 1 1 .
25 1 . s P e r f o r m an c e . .
25 0 . .
75 1 1 .
25 1 . s P e r f o r m an c e . .
25 0 . .
75 1 1 .
25 1 . s P e r f o r m an c e . .
25 0 . .
75 1 1 .
25 1 . s P e r f o r m an c e . .
25 0 . .
75 1 1 .
25 1 . s P e r f o r m an c e approach allows us to quickly find common variance designs that overlap with those known in theliterature. Furthermore, in cases where common variance designs do not exist or cannot be found,our approach allows identification of designs with close to common variance. Comparisons to aPlackett-Burman design and several other standard optimal designs from literature demonstratedthat such designs perform quite well in practice, and that in many cases these A-ComVar designsperform as well as common variance designs.There are several avenues here for future work. First, we considered only the cases with two-leveland three-level factors. Future work could consider finding A-ComVar designs with mixed − levelfactors. Second, we utilized a genetic algorithm to find these designs. There are numerous otheroptimization approaches that could be used to maximize the objective function in (2). In somecases, these other approaches may succeed in finding designs with a better ratio of minimumto maximum variance of the uncommon parameters. Third, there is another approach to findingcommon variance designs through hierarchical designs (Chowdhury, 2016). These designs are foundby identifying a common variance design for a smaller number of runs and then adding runs while20rying to preserve the common variance property. It is possible that a similar idea could bedeveloped for A-ComVar designs. Finally, future work could study the types of A-ComVar designsthat can be found when the number of interactions in the model increases beyond two. References
Bingham, D. R. and Chipman, H. A. (2007). Incorporating prior information in optimal design formodel selection.
Technometrics , 49(2):155–163.Breiman, L. (1995). Better subset regression using the nonnegative garrote.
Technometrics ,37(4):373–384.Calvin, J. A. (1986). A new class of variance balanced designs.
Journal of statistical planning andinference , 14(2-3):251–254.Candes, E., Tao, T., et al. (2007). The Dantzig selector: Statistical estimation when p is muchlarger than n.
The annals of Statistics , 35(6):2313–2351.Cheng, C.-S. (1986). A method for constructing balanced incomplete-block designs with nestedrows and columns.
Biometrika , 73(3):695–700.Chowdhury, S. (2016).
Common Variance Fractional Factorial Designs for Model Comparisons .PhD thesis, University of California Riverside.Ghosh, S. and Chowdhury, S. (2017). CV, ECV, and robust CV designs for replications under aclass of linear models in factorial experiments.
Journal of Statistical Planning and Inference ,188:1–7.Ghosh, S. and Flores, A. (2013). Common variance fractional factorial designs and their optimalityto identify a class of models.
Journal of Statistical Planning and Inference , 143(10):1807–1815.Ghosh, S. and Tian, Y. (2006). Optimum two level fractional factorial plans for model identificationand discrimination.
Journal of multivariate analysis , 97(6):1437–1450.Gupta, S. and Jones, B. (1983). Equireplicate balanced block designs with unequal block sizes.
Biometrika , 70(2):433–440. 21edayat, A. and Stufken, J. (1989). A relation between pairwise balanced and variance balancedblock designs.
Journal of the American Statistical Association , 84(407):753–755.Kane, A. and Mandal, A. (2019). A new analysis strategy for designs with complex aliasing.
TheAmerican Statistician .Khatri, C. (1982). A note on variance balanced designs.
Journal of Statistical Planning andinference , 6(2):173–177.Li, W. and Nachtsheim, C. J. (2000). Model-robust factorial designs.
Technometrics , 42(4):345–352.Lin, C. D., Anderson-Cook, C. M., Hamada, M. S., Moore, L. M., and Sitter, R. R. (2015).Using genetic algorithms to design experiments: a review.
Quality and Reliability EngineeringInternational , 31(2):155–167.Mandal, A., Wong, W. K., and Yu, Y. (2015). Algorithmic searches for optimal designs.
Handbookof Design and Analysis of Experiments , pages 755–783.Mukerjee, R. and Kageyama, S. (1985). On resolvable and affine resolvable variance-balanceddesigns.
Biometrika , 72(1):165–172.Srivastava, J. (1976). Some further theory of search linear models. In
Contribution to AppliedStatistics , pages 249–256. Springer.Srivastava, J. and Ghosh, S. (1976). A series of balanced factorial designs of resolution v which allowsearch and estimation of one extra unknown effect.
Sankhy¯a: The Indian Journal of Statistics,Series B , pages 280–289.Srivastava, J. and Gupta, B. (1979). Main effect plan for 2m factorials which allow search andestimation of one unknown effect.
Journal of Statistical Planning and Inference , 3(3):259–265.Yuan, M., Joseph, V. R., and Lin, Y. (2007). An efficient variable selection approach for analyzingdesigned experiments.
Technometrics , 49(4):430–439.Yuan, M., Joseph, V. R., Zou, H., et al. (2009). Structured variable selection and estimation.
TheAnnals of Applied Statistics , 3(4):1738–1757. 22 ppendix − Genetic Algorithm
In this work we used a genetic algorithm to find designs that maximize the A-ComVar objectivefunction. This appendix provides specific details on the algorithm. Keeping with the standardgenetic algorithm terminology, we use the word chromosome to describe a single candidate design.Each chromosome is comprised of the factor settings for each factor at each design point. Each ofthese individual factor settings is known as a gene . The population is the set of all chromosomes,i.e. all designs that we are currently considering.We illustrate a simple version of the genetic algorithm below. In this example we search for a 6run A-ComVar design for an experiment with three two-level factors and one interaction. We labelthe factors as A , B , and C . For simplicity, we assume that the population size is 3, although inreal applications it will generally be larger.Since this experiment has three two-level factors, there are 8 possible design points to pick the6 points for our design from. The 8 points are shown in Table 7. There are (cid:0) (cid:1) = 3 possible modelswith all main effects and one interaction. For notational simplicity we label these models by thecorresponding interaction: ( AB ), ( AC ), and ( BC ). Our goal is to obtain a design under which thevariance of the interaction term is identical, or close to identical, under all three of these models. A B C -1 -1 -1-1 -1 1-1 1 -1-1 1 11 -1 -11 -1 11 1 -11 1 1Table 7: Set of possible design points for the Appendix example.
0. Initialization
First, each of the three chromosomes is initialized to a random start. To obtain the randomstart for a specific chromosome, we simply sample six of the rows in Table 7 without replacement.Our initialization procedure results in the following three chromosomes:231 -1 -1-1 1 11 -1 -11 -1 11 1 -11 1 1 -1 -1 -1-1 1 -11 -1 -11 -1 11 1 -11 1 1 -1 -1 -1-1 -1 1-1 1 11 -1 11 1 -11 1 1PopulationChromosome 1 Chromosome 2 Chromosome 3After initializing, we need to calculate the fitness for each of these chromosomes using theobjective function in expression (2). In order to evaluate the objective function, we need to calculate σ i for i = 1 , ,
3, which correspond to models ( AB ), ( AC ), and ( BC ), respectively. Then, we takethe average of these three values to be σ and can evaluate the objective function. These steps areillustrated below for the first chromosome. 241 -1 -1-1 1 11 -1 -11 -1 11 1 -11 1 11 -1 -1 -1 11 -1 1 1 -11 1 -1 -1 -11 1 -1 1 -11 1 1 -1 11 1 1 1 1 1 -1 -1 -1 11 -1 1 1 -11 1 -1 -1 -11 1 -1 1 11 1 1 -1 -11 1 1 1 1 1 -1 -1 -1 11 -1 1 1 11 1 -1 -1 11 1 -1 1 -11 1 1 -1 -11 1 1 1 10.19 -0.06 0.00 0.00 0.00-0.06 0.19 0.00 0.00 0.000.00 0.00 0.25 -0.13 -0.130.00 0.00 -0.13 0.25 0.130.00 0.00 -0.13 0.13 0.25 0.19 -0.06 0.00 0.00 0.00-0.06 0.19 0.00 0.00 0.000.00 0.00 0.25 -0.13 0.130.00 0.00 -0.13 0.25 -0.130.00 0.00 0.13 -0.13 0.25 0.25 -0.13 0.00 0.00 -0.13-0.13 0.25 0.00 0.00 0.130.00 0.00 0.19 -0.06 0.000.00 0.00 -0.06 0.19 0.00-0.13 0.13 0.0 0.0 0.25 σ = 0 . σ = 0 . σ = 0 . σ β = 0 .
25, Fitness = 4.0Model ( AB ) Model ( AC ) Model ( BC ) X (1) X (2) X (3) | X (1) T X (1) | − | X (2) T X (2) | − | X (3) T X (3) | − Illustration of fitness calculation.The above procedure is repeated for each of the three chromosomes. In this case, all threedesigns end up having the same fitness value. We now summarize each chromosome below:-1 -1 -1-1 1 11 -1 -11 -1 11 1 -11 1 1 -1 -1 -1-1 1 -11 -1 -11 -1 11 1 -11 1 1 -1 -1 -1-1 -1 1-1 1 11 -1 11 1 -11 1 1Fitness: 4.0 Fitness: 4.0 Fitness: 4.0PopulationChromosome 1 Chromosome 2 Chromosome 325ow that we have completed the initialization process, we can begin the main loop over thealgorithm.
1. Identify worst chromosome(s)
The first step is to identify the worst chromosomes. These are the chromosomes that will bereplaced by new offspring. Since we only have three chromosomes in the population, we will onlyidentify and replace the single worst chromosome. In the case of a tie (as we have here), thechromosome to be replaced is randomly chosen. In this case we have chosen chromosome 3 to bereplaced.
2. Generate replacement using crossover
We next generate a replacement for the worst chromosome (3) using crossover from 2 randomlyselecting remaining chromosomes. Since our example only has three chromosomes, we simply usethe remaining chromosomes (1 and 2). In the crossover, a random cut point is selected, and thetwo chromosomes are combined using the values from the first chromosome for the factors to theleft of the cut point, and the values from the second chromosome for the factors to the right of thecut point. This process is illustrated below:-1 -1 -1-1 1 11 -1 -11 -1 11 1 -11 1 1 -1 -1 -1-1 1 -11 -1 -11 -1 11 1 -11 1 1 -1 -1 -1-1 -1 1-1 1 11 -1 11 1 -11 1 1CrossoverChromosome 1 Chromosome 2 Offspringcut point cut pointNote that it is possible to consider other ways of producing offspring via crossover. For example,the cut point could be different for each support point, or they could be “horizontal” instead of“vertical,” choosing certain rows from the first chromosome and the remaining rows from the second.
3. Mutation
In addition to crossover, more novelty can be introduced to the solution by randomly changing,or mutating, some of factor settings. For our purpose, the probability of each factor setting (gene)26 odel Size σ = 0 . σ = 0 . σ = 0 . σ = 0 . σ = 1 σ = 1 . σ = 1 . Table 8: Average Percentage of Correctly Identified Models for the common variance design D and the Plackett-Burman design.mutating is identical.
4. Replacement and Fitness Evaluation
Following Steps 3 and 4, we are now ready to replace the old chromosome with the offspring. Inthis step, the worst chromosome(s) is replaced by the offspring created in Steps 2 −
3. The fitnessof this new chromosome is evaluated and stored.
Appendix − Tables for Example 3
Tables 8 −
14 below present detailed results for each of the comparisons in Example 3.27 odel Size σ = 0 . σ = 0 . σ = 0 . σ = 0 . σ = 1 σ = 1 . σ = 1 . Table 9: Average Percentage of Correctly Identified Models for A-ComVar design D and thePlackett-Burman design. Model Size σ = 0 . σ = 0 . σ = 0 . σ = 0 . σ = 1 σ = 1 . σ = 1 . Table 10: Average Percentage of Correctly Identified Models for A-ComVar design D and BayesOptimal design D from Bingham and Chipman (2007).28 odel Size σ = 0 . σ = 0 . σ = 0 . σ = 0 . σ = 1 σ = 1 . σ = 1 . Table 11: Average Percentage of Correctly Identified Models for A-ComVar design D and design D from Li and Nachtsheim (2000). Model Size σ = 0 . σ = 0 . σ = 0 . σ = 0 . σ = 1 σ = 1 . σ = 1 . Table 12: Average Percentage of Correctly Identified Models for A-ComVar design D and design D from Ghosh and Tian (2006). 29 odel Size σ = 0 . σ = 0 . σ = 0 . σ = 0 . σ = 1 σ = 1 . σ = 1 . Table 13: Average Percentage of Correctly Identified Models for 3 − level A-ComVar design D andCCD D . Model Size σ = 0 . σ = 0 . σ = 0 . σ = 0 . σ = 1 σ = 1 . σ = 1 . Table 14: Average Percentage of Correctly Identified Models for 3 − level A-ComVar design D andOME D8