Experimental Design - Math Site

Topics CoveredIntroductionFull 2 Level Factorial DesignsFractional 2 Level Factorial Designs (Generators, Resolutionand Plackett Burman Designs)Variances, Significance and Error BoundsResponse Surface Designs (Full 3 Level, Box Behnken, andCentral Composite Designs)Mixtures and Scheffé DesignsSequential Designs, Evolutionary Operation, and theSimplex MethodCEI Inc.

What is Experimental Design?• Designing experiments to yield the mostinformation from the fewest runs• Isn’t this obvious? Shouldn’t most peoplewith a science or engineering backgroundknow this stuff intuitively?CEI Inc.

History• Agricultural field trial work of Fisher* inthe 1920s• Chemical industry work of George Box,starting in 1951 with the publication of hiswork term notes from ICI***R.A. Fisher, ”Studies in Crop Variation. II. The Manurial Response of Different PotatoVarieties”, J. Agricultural Sci., 13, 311-320 (1923)** G.E.P. Box and K.B. Wilson, ”On the experimental attainment of optimal conditions”, J.Roy. Statist. Soc., B13,1-45 (1951)CEI Inc.

Ronald A. FisherCEI Inc.

George E.P. BoxCEI Inc.

Fisher’s window is a 7x7Latin square for yieldstudies on potatoes.Memorial windows forVenn and Fisher, CaiusCollege, CambridgeCEI Inc.

N-space and Real Space• It is often useful to move back and forthbetween sampling for a spatial function ofx and y (or, of x, y and z) and testing afunction of several other independentvariables (temperature, pressure,concentration of reactants, etc.)CEI Inc.

Two Basic Types of Design• One that identifies influential variables orthe “effects” different variables have• One that determines approximate functionalform of a relationship or the “responsesurface”CEI Inc.

Two Level Factorial Designs• If we are to determine which variablesinfluence a result, usually use a two levelfactorial design• For example, determine if temperature andpressure influence the specific volume ofammonia -- will look at P and T at “high”and “low” levelsCEI Inc.

high x xPressurelow x xlowTemp.CEI Inc.high

Specific Volume of Ammonia• Four runs are needed to get all combinationsof “high” and “low”• Usual to use coded independent variablesX c = 2 (x – x av )/(x high –x low )CEI Inc.

Coding• For example, if the temperature ranges from40 o C to 140 o C– If T = 40 o C, T c = -1– If T = 140 o C, T c = +1CEI Inc.

xP+1x-1 +1Tx-1x xFull Two Level Factorial Design in Two FactorsCEI Inc.

Experimental RunsOrder T, o C P, kPa T coded P coded V, m 3 /kg3 40 500 -1 -1 0.292 40 1000 -1 +1 0.144 140 500 +1 -1 0.401 140 1000 +1 +1 0.19CEI Inc.

Effect of Temperature?V0.2950.215-1 +1CEI Inc.T c

Effect of Pressure?0.3450.165-1 +1CEI Inc.P

Effect of T, P Interaction?0.270.24-1 +1CEI Inc.TP

Effect of T 2 ? P 2 ?In 2 level factorial designs, thesquared terms are always equalto 1.CEI Inc.

Empirical Models• It is often preferable to fit to some sort ofempirical model• It can be shown that the most generalpolynomial model that can be estimatedfrom a full 2 level factorial design in twofactors isy = β 0 + β 1 x 1 + β 2 x 2 + β 12 x 1 x 2CEI Inc.

Our data can be written out asβ 0 - β 1 - β 2 + β 12 = 0.29β 0 - β 1 + β 2 - β 12 = 0.14β 0 + β 1 - β 2 - β 12 = 0.40β 0 + β 1 + β 2 + β 12 = 0.19i.e. four equations in four unknowns, so thereshould be a solution.CEI Inc.

The solution is easier if we use matrices(really).y = X βX t y = X t X βNote that this isnot really leastsquares estimationas there is noredundancy.β = (X t X) -1 X t yCEI Inc.

In our example,1 -1 -1 11 -1 1 -11 1 -1 -11 1 1 1β 0β 10.290.14=β 20.40β 12 0.19CEI Inc.

X t X =1 1 1 1-1 -1 1 1-1 1 -1 11 -1 -1 11 -1 -1 11 -1 1 -11 1 -1 -11 1 1 1=4 0 0 00 4 0 00 0 4 00 0 0 4CEI Inc.

¼ 0 0 0(X t X) -1 =0 ¼ 0 00 0 ¼ 00 0 0 ¼1 1 1 10.291.02X t y =-1 -1 1 1-1 1 -1 10.140.40=0.16-0.361 -1 -1 10.19-0.06CEI Inc.

0.255β = (X t X) -1 X t y =0.040-0.090-0.015CEI Inc.

or,y = 0.255 + 0.04 T coded –0.09 P coded – 0.015 T coded P codedCEI Inc.

What if we havethree factors?CEI Inc.

x 1 x 2 x 3-1 -1 -1-1 -1 1-1 1 -1-1 1 11 -1 -11 -1 11 1 -11 1 1CEI Inc.

Three Independent Variables• Our example could be the partial oxidationof o-xylene–x 1 could be coded temperature–x 2 could be coded partial pressure of oxygen inthe feed to the reactor–x 3 could be the coded presence or absence ofbismuth molybdate catalystCEI Inc.

In two-level factorial designs, can incorporate Boolean variablesi.e. x 3 = -1 if no catalyst used= +1 if catalyst used.With three variables, the most general polynomial model thatcan be generated from a full 2 level factorial design isy = β o + β 1 x 1 + β 2 x 2 + β 3 x 3 + β 12 x 1 x 2 + β 13 x 1 x 3 + β 23 x 2 x 3 +β 123 x 1 x 2 x 3Note: there are still no squared terms.CEI Inc.

Exercise (left to student)Show that, for a full 2 level design in three variables, the(X t X) -1 matrix is:1/8 0 0 0 0 0 0 00 1/8 0 0 0 0 0 0..0 0 . . . 0 1/8CEI Inc.

Fractional Factorial Designs• The word “full” has been used a lot here.Are there 2 level designs that are not “full”?• Consider the 3 variable design above. In afull design, 2 3 or 8 runs are needed. Whatif we can only do 4 runs -- is there anintelligent way to choose the runs?CEI Inc.

x 1 x 2 x 3-1 -1 -1-1 -1 1-1 1 -1-1 1 11 -1 -11 -1 11 1 -11 1 1CEI Inc.

(poor choice)We could just take the first 4 runs (where x 1 is always –1),but then x 1 doesn’t vary. Can we pick the 4 runs so that all ofthe x i terms vary?(good choice)x 1 x 2 x 3-1 -1 1-1 1 -1Note: each x i varies andno columns are identical1 -1 -11 1 1CEI Inc.

However, you can confirm thatx 1 = x 2 x 3x 2 = x 1 x 3x 3 = x 1 x 2These are termed“confounding”relationships.and x 1 x 2 x 3 = 1Thus, the 8 parameter model cannot be determined by thisdata (we should have known that you can’t estimate 8parameters with 4 runs).CEI Inc.

Often, make the assumption that interaction effects are negligible(relative to linear effects). Then, fit a simpler model to the data:y = β o + β 1 x 1 + β 2 x 2 + β 3 x 3but recognize that, for example, β 2 x 2 really represents β 2 x 2 +β 13 x 1 x 3 , and so on.How did I come up with this subset of 8 runs that ensured thatthe linear effects wouldn’t be confounded with each other?CEI Inc.

Generators• I used the relationship 1 = x 1 x 2 x 3 (i.e. thethree way interaction term is set to theconstant, or β 0 and β 123 x 1 x 2 x 3 will beindistinguishable).• Multiply the equation above by x 1 :x 1 = x 12 x 2 x 3But x 12 = 1 in a 2 level factorial design, sox 1 = x 2 x 3 .CEI Inc.

Similarly,x 2 = x 1 x 3x 3 = x 1 x 2Exercise (left to students)Show that x 1 x 2 x 3 = -1 picks out the other four runs in thefull 2 3 design.CEI Inc.

Resolution• In our 4 run subset of the 2 3 design, (termeda 2 3-1 fractional factorial design),– zero order terms (the constant) wereconfounded with third order terms– first order terms were confounded with secondorder terms• Note that the sum of the orders of theconfounded terms is threeCEI Inc.

Our 4 run subset of the 2 3 design is thentermed to have “resolution III” and isdenoted as a2 3-1 III designThe higher the resolution, the better. If we tookour “poor choice” subset where x 1 = -1, thatwould be resolution I.CEI Inc.

Plackett Burman Designs• In general terms, fractional 2 level factorialdesigns are used to screen a large number ofpossibly influential variables to see if theytruly influence the response. We usuallyassume that the effects of second order andhigher terms (interactions) are much lessthan those of the constant and linear terms.CEI Inc.

Plackett Burman Designs• The most efficient of these “screening”designs is due to Plackett and Burman*• 12 runs can test up to 11 independentvariables (other PB designs have 20, 24, 28,and so on for the number of runs)* R.L Plackett and J.P. Burman, “The Design of Optimal MultifactorialExperiments”, Biometrika, 33, 305-325, (1946)CEI Inc.

Plackett Burman Designs• The Plackett Burman designs are based oncertain properties of Hadamard matrices• All PB designs are of resolution III• PB designs are sometimes termed“saturated”CEI Inc.

1 + + + + + + + + + + +2 - + - + + + - - - + -3 - - + - + + + - - - +4 + - - + - + + + - - -5 + + - - + - + + + - -6 - + + - - + - + + + -7 - - + + - - + - + + +8 + - - + + - - + - + +9 + + - - + + - - + - +10 + + + - - + + - - + -11 - + + + - - + + - - +12 + - + + + - - + + - -Plackett Burmandesign to test 11variables in 12runsCEI Inc.

Significance, Variances and ErrorBounds• The fitted models we have discussed utilizeall of the experimental data to generatecoefficients -- there are no “degrees offreedom” left to estimate variances• If there is an independent estimate of the“pure error variance” (call it s 2 ) and if it canbe assumed that this does not vary acrossthe domain of interest, thenCEI Inc.

V(β) = (X t X) -1 s 2and V(y) = X (X t X) -1 X t s 2CEI Inc.

NoteAlthough the V( ) notation suggests that weare talking about variances, it should beobvious that we are only talking aboutestimates of variances. If we wish to findconfidence intervals for the true variances,we will need to use the χ 2 test; confidenceintervals for the true values of the βparameters or the predicted y values willrequire a t-test.CEI Inc.

Note to the NoteIf the confidence interval for one of the βterms includes zero, the data are telling usthat that particular combination of variablesis not influencing the result (or if it doeshave an influence, the influence is less thatthe effects of the uncontrollable variables(i.e. “the noise”)). The parameter and thecorresponding combination of variablesshould be omitted from the model.CEI Inc.

Aside on NomenclatureIf the prediction variance (V(y)) is only afunction of the distance from the centre ofthe design (and not otherwise a function ofposition), we say that the design isrotatable. It can be shown that this leads torobust estimation of the response surface.CEI Inc.

Response Surface Designs• Once we have determined which variablesinfluence a response, we would like toquantify “how much” or determine amaximum or minimum over the domainunder study• 1. If we have a really good idea of thefunctional relationship between dependentand independent variables, use a “parameterCEI Inc.

estimation design”. Briefly, test in areas of thedomain where a small change in the parameterswill yield a large change in the response. Wewill not deal further with this here.2. If we do not know the functional form ofthe relationship, we may proceed by assumingan empirical relationship. The simplestfunctional form that will allow for localmaxima and minima is the general quadratic:CEI Inc.

1 variable: y = β 0 + β 1 x 1 + β 11 x 122 variables: y = β 0 + β 1 x 1 + β 2 x 2 + β 12 x 1 x 2 + β 11 x 12+ β 22 x 223 variables: y = β 0 + β 1 x 1 + β 2 x 2 + β 3 x 3 + β 12 x 1 x 2 +β 13 x 1 x 3 + β 23 x 2 x 3 + β 11 x 12 + β 22 x 22 + β 33 x 32and so on.CEI Inc.

We cannot use 2 level factorial designs toestimate quadratic terms (recall that in 2 leveldesigns, x i2 = 1). Therefore, we must use 3levels (at least) for each independent variable.CEI Inc.

Three Level Factorial Designs1 Independent Variable:y0 x-1 1Test at x coded = -1, 0 and +1.CEI Inc.

Three Level Factorial Designs2 Independent Variablesx 2x 1 x xx x x-101x 19 Runsx-1xxCEI Inc.

Three Level Factorial Designsx 1 -1 -1 -1 0 0 0 1 1 1x 2 -1 0 1 -1 0 1 -1 0 1Bonus: we only have to estimate 6 terms (β 0 , β 1 , β 2 , β 12 ,β 11 , β 22 ) using these 9 data points, so that an estimate ofthe variance can be made at the same time.CEI Inc.

Example: More on v for NH 3Estimate a general quadratic model for thespecific volume of ammonia with temperature andpressure as the independent variables, valid in theregion from 0 to 100 o C and 100 to 300 kPa.Experimental data are available from a 3 2 factorialdesign; these data are given in the following table.CEI Inc.

Specific Volumes of Ammonia in m 3 /kg0 o C 50 o C 100 o C100 kPa 1.3145 1.5664 1.8145200 kPa 0.6471 0.7774 0.9035300 kPa 0.4243 0.5143 0.5997CEI Inc.

Other Response Surface Designs• As the number of independent variablesincreases, the number of runs required for afull 3 n design greatly increases. Forinstance, if n = 3, 3 n = 27; if n = 4, 3 n = 81.If experiments cost time and/or money, thismay be unacceptable.• However, we do not really need this largenumber of runs. ConsiderCEI Inc.

If we have 2 independent variables, we need toestimate 6 coefficients.If we have 3 independent variables, we need toestimate 10 coefficients.If we have 4 independent variables, we need toestimate 15 coefficientsIn general, if we have n independent variables, weneed to estimate (n+2)(n+1)/2 coefficients.Obviously, 3 n runs may be overkill. Are there designswhich still have all variables tested at at least threelevels, but which have fewer runs?CEI Inc.

Box Behnken Designs• The most widely used three level factorialdesign is due to Box and Behnken*• Designs are available for three or moreindependent variables.• Let us examine the three variable case.* G.E.P. Box and D.W. Behnken, ”Some new three level designs for the study of quantitativevariables”, Technometrics, 2, 455, (1960)CEI Inc.

3 Factor Box Behnken DesignNote: there are 12 individual runs on the centre points of theedges of a cube and 3 points at the centre of the cube.CEI Inc.

3 Factor Box Behnken DesignThe actual runs are given byx 1 -1 1 -1 1 -1 1 -1 1 0 0 0 0 0 0 0x 2 -1 -1 1 1 0 0 0 0 -1 1 -1 1 0 0 0x 3 0 0 0 0 -1 -1 1 1 -1 -1 1 1 0 0 0The actual runs would, of course, be conducted in arandom order.CEI Inc.

Box Behnken Example• Consider a 10 m x 10 m plot of land from which aleaking gasoline tank had been removed. Thecontaminated soil had been returned to theexcavation and clean fill added on top tocompensate for the volume of the tank.• The tank had been resting on a layer of compactedclay 4.8 m below grade. The clean fill isestimated to occupy the upper 2.5 m of thesubsurface.CEI Inc.

Use a Box Behnken design to estimate the parameters in aquadratic expression for hydrocarbon concentration.If z is the distance below grade and x and y are rectangularco-ordinates relative to the centre of the 10 m x 10 m square,x = -5 implies x coded = -1 and x = 5 implies x coded = 1y = -5 implies y coded = -1 and y = 5 implies y coded = 1z = 5 implies z coded =1, z = 2.5 implies z coded = -1 andz = 3.75 implies z coded = 0Samples were taken and the following were the results:CEI Inc.

x -1 1 -1 1 -1 1 -1 1y -1 -1 1 1 0 0 0 0z 0 0 0 0 -1 -1 1 1c HC 800 50 550 10 40 0 70 10x 0 0 0 0 0 0 0y -1 1 -1 1 0 0 0z -1 -1 1 1 0 0 0c HC 10 0 60 5 1100 1050 1100Note that hydrocarbon concentrations are given in mg/kg (ppm)CEI Inc.

Question for DiscussionWhy isn’t there a Box Behnkendesign for two independentvariables?CEI Inc.

Other Response Surface Designs• Central composite designs (CCDs) are alsoused, although these tend to be less efficientthan Box Behnken• CCDs require five levels and usually consistof a 2 level design plus a “star-shaped”design. These were dealt with in the 1951Box Wilson paper (qv).CEI Inc.

Construction of a CCD in two factorsCEI Inc.

Mixtures and Scheffé Designs• One area where one can encounter problems is inthe optimal design of mixtures, because thephysical requirement that the sum of all the massfractions must be one reduces the rank of the Xmatrix• The way around this and the way to approachexperimental design is due to Scheffé**H. Scheffé, ”Experiments with mixtures”, J. Roy. Statist. Soc, B20, 344-360 (1958), andH. Scheffé “The simplex-centroid design for experiments with mixtures”, J. Roy. Statist. Soc.,B25, 235- 263 (1963)CEI Inc.

Henry SchefféCEI Inc.

In order to not have this problem with rank, Scheffé foundthat you could eliminate the constant.Consider a linear function of the two components of amixture, x 1 and x 2y = β 0 + β 1 x 1 + β 2 x 2Since 1 = x 1 + x 2 , we can writey = β 0 (x 1 + x 2 ) + β 1 x 1 + β 2 x 2= (β 0 + β 1 )x 1 +(β 0 + β 2 )x 2?CEI Inc.

Scheffé also came up with simplex-centroidexperimental designs. For three components,The design points correspond to all permutations of the pure components(e.g., 1 0 0; 0 1 0; 0 0 1), the permutations of the binary blends (½ ½ 0; ½ 0½; 0 ½ ½) and the one blend involving equal parts of all three components.CEI Inc.

The points in the simplex centroid design for three components are:x 1 1 0 0 ½ ½ 0 1/3x 2 0 1 0 ½ 0 ½ 1/3x 3 0 0 1 0 ½ ½ 1/3Note that this design only contains 7 runs. A full quadratic model forthree components requires 9, since we will eliminate the constant. To getsufficient data, it is usual to addx 1 2/3 1/6 1/6x 2 1/6 2/3 1/6x 3 1/6 1/6 2/3CEI Inc.

Let’s look at an example that takes us back to experimentaldesign’s agricultural roots. Synthetic mixtures of sand, silt andclay are used to grow potatoes under controlled conditions. Theresponse is measured as kg potatoes yield per square metre ofbed. Find the general quadratic model that best fits the data:Sand Fraction 1 0 0 ½ ½ 0 1/3 2/3 1/6 1/6Silt Fraction 0 1 0 ½ 0 ½ 1/3 1/6 2/3 1/6Clay Fraction 0 0 1 0 ½ ½ 1/3 1/6 1/6 2/3Yield (kg/m 2 ) 18 12 6 16 11 10 15 19 15 9CEI Inc.

Sequential DesignsA sequential design is one in which futureexperimental conditions depend on the resultsof previously conducted experiments. The onesequential design that we have seen is thecentral composite design. Another sequentialdesign that is also used is the procedure knownas Evolutionary Operation (or EVOP)CEI Inc.

Evolutionary Operation (EVOP)The term Evolutionary Operation (or EVOP) is dueto George Box*, although the method as presentlyknown (using the “simplex” algorithm) is due toSpendley et al**; the numerical algorithm wasfurther refined by Nelder and Mead***.*G.E.P. Box, “Evolutionary operation: A method for increasing industrialproductivity”, Applied Statistics 6, 81-101 (1957).**W. Spendley, G.R. Hext and F.R. Himsworth, “Sequential application ofsimplex designs in optimization and EVOP”, Technometrics 4, 441-461 (1962).***J.A. Nelder and R. Mead, “A simplex method for function minimization”,Computer Journal 7, 308-313 (1965).CEI Inc.

Evolutionary OperationThe basic idea is to determine (by some othertechnique) the factors that influence theresponse. EVOP can then be used to determinewhat values of these factors to use to maximize(or minimize) the response.If n factors have been found that influence theresponse, n + 1 runs that are linearlyindependent are made (e.g. if there are 2important factors, 3 runs are made at conditionsthat do not lie on a straight line).CEI Inc.

Evolutionary Operation, cont’dThe n + 1 points in n-space form a simplex (3points in 2-space form a triangle, 4 points in 3-space form a tetrahedron,…). The simplexmethod consists of discarding the worst result andtesting at one new condition; this is donerepeatedly until a maximum (or minimum) isreached.CEI Inc.

Evolutionary Operation, cont’dOne of the main advantages of EVOP is that afterthe process begins, the runs can often be madewithout major disturbances of the process understudy. As the process continues, the changes in theinput variables required by the process becomesmaller and smaller.To illustrate the method, let us proceed with anobvious example with two input variables.CEI Inc.

Geometry of 2 – D Simplex MethodnbrwcCEI Inc.bw worst pointb best pointnb next bestc centroid of pointsleft after wremovedr reflection of wthrough c

EVOP ExampleZ is the response to two input variables x and y;Z is given byZ = 100 / (1 + x 2 + y 2 ) .Clearly, we can see that Z is maximized for x =y = 0 and takes on no other local maxima orminima for any finite, real x and y values.Assuming that we do not know about thismaximum, the simplex method tells us to findZ at 3 points:CEI Inc.

x 2 3 2y 0 3 2Z 20 5.26 11.1Clearly (3,3) is w, (2,0) is band (2,2) is nb. Removingw, the centroid of theremaining points is (2,1).To reflect w through c, either1. Add the vector sum of c – w to c, or2. Simply compute r = 2c – wIn our case, r = 2(2,1) - (3,3) = (1,-1)CEI Inc.

Computing Z(r) = 33.3, our three points in thesimplex are nowx 2 2 1The worst point is at(2,2). The centroidy 0 2 -1 after the worst point isZ 20 11.1 33.3removed is (3/2, -1/2).The reflection of theworst point throughthe centroid is 2c – wor (1,-3).Z(r) = 9.09 (worse than last worst)! Now what?CEI Inc.

Geometry of 2 – D Simplex MethodnbrwcCEI Inc.bw worst pointb best pointnb next bestc centroid of pointsleft after wremovedr reflection of wthrough c

If the result at the reflection is worse than the resultat w, replace r by the midpoint from w to c. In ourcase, this contraction would be ½{(2,2) + (3/2, -1/2)} or (1.75, 0.75). Z(1.75, 0.75) = 21.62. Nowour three points entering the next iteration are:x 2 1 1.75y 0 -1 0.75Z 20 33.3 21.62CEI Inc.

For the next iteration, w = (2, 0), c =(1.375, -0.125) and r = (0.75, -0.25).Then, Z(r) = 61.5 and we are starting toconverge on (0, 0) in this third iteration.If we continue on with this process, wegradually approach the optimal conditions.Plotting the worst and best values of Z ateach iteration gives the following graph.CEI Inc.

1201008060402001 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16Here the best value of Z at each iteration givesthe upper curve and the worst value gives thelower curve. The “S-shaped” or logistic curvefor the worst values is typical of many EVOPs.CEI Inc.

Another observation from the graph is that,after 11 or 12 iterations, even the “worst”conditions are close to optimal. If this were aplant process, the EVOP process couldcontinue as part of a continuous improvementplan making small incremental gains withlittle or no downside risk to the operation.CEI Inc.

Warning!CEI Inc.

Warning #1The experimental runs used to convergeon the maximum (or minimum) responseare not good data to to use for generatinga response surface model. If you need amodel, generate data using aconventional response surface design.CEI Inc.

Warning #2If there are uncontrollable inputs thatare comparable in size to the effects ofthe controllable inputs, the EVOPprocess may never converge. Thesimplex may either oscillate withvariations in the uncontrollable inputsor diverge completely.CEI Inc.

Warning #2, and Possible CureA possible cure is to deal with average responses,where the averaging is over a longer time periodthan the characteristic times associated with theuncontrollable inputs.If the uncontrollable inputs have vastly differentcharacteristic times, this averaging may not bepossible and a method other than EVOP must beused for optimization.CEI Inc.

EpilogueWith all the matrix algebra and n-space geometry, don’t forget that thisall started out as a way to grow thebest potatoes with the minimumamount of manure.CEI Inc.