SCIENTIFIC AMERICAN

SCIENTIFICAMERICANJANUARY 1994$3.95Sctm hmij (or stnimte quark matter,\ aiunpse at tunr se?: evolved.* -»,«? , - - t hettnj lostIf^ ^ ^ ^ ^COPYRIGHT1991 BY SCIENTIFIC AMERICAN, INC. ALL RIGHTS RESERVEDAn even break of the rack creates an intractableproblem: calculating the paths the balls will take.

Breaking IntractabilityProblems that would otherwise he impossibleto solve can now be computed, as long as onesettles for what happens on the averageby Joseph F. Traub and Henryk WozniakowskiAlthough mathematicians and scientistsmust rank among theL most rational people in theworld, they will often admit to fallingprey to a curse. Called the curse of dimension,it is one many people experi-A potentially intractable problemence in some form. For example, a family'sdecision about whether to refinancetheir mortgage with a 15- or 30-yearloan can be extremely difficult to make,because the choice depends on an interplayof monthly expenses, income,future tax and interest ratesand other uncertainties. In science,the problems are moreesoteric and arguably muchharder to cope with. In thecomputer-aided design ofpharmaceuticals, for instance,one might need to know 7 how 7tightly a drug candidate willbind to a biological receptor.Assuming a typical number of8,000 atoms in the drug, thebiological receptor and thesolvent, then because of the^ three spatial variables neededA \ to describe the position of. V \ each atom, the calculation in-^ volves 24,000 variables. Simplyput, the more variables, ordimensions, there are to consider,the harder it is to accomplisha task. For manyproblems, the difficulty grow 7 sexponentially with the numberof variables.The curse of dimension canelevate tasks to a level of difficultyat which they become intractable.Even though scientistshave computers at theirdisposal, problems can haveso many variables that nofuture increase in computerspeed will make it possible tosolve them in a reasonableamount of time.Can intractable problems bemade tractable—that is, solvablein a relatively modestamount of computer time?Sometimes the answer is, happily,yes. But we must be willingto do without a guaranteeof achieving a small error inour calculations. By settlingfor a small error most of thetime (rather than always), some kindsof multivariate problems become tractable.One of us (Wozniakowski) formallyproved that such an approachworks for at least two classes of mathematicalproblems that arise quite frequentlyin scientific and engineeringtasks. The first is integration, a fundamentalcomponent of the calculus. Thesecond is surface reconstruction, inwhich pieces of information are usedto reconstruct an object, a techniquethat is the basis for medical imaging.Fields other than science can benefitfrom ways of breaking intractability.For example, financial institutions oftenhave to assign a value to a pool of mortgages,which is affected by mortgageeswho refinance their loans. If w 7 e assumea pool of 30-year mortgages and permitrefinancing monthly, then this taskcontains 30 years times 12 months, or360 variables. Adding to the difficulty 7is that the value of the pool dependson interest rates over the next 30 years,which are of course unknown.We shall describe the causes of intractabilityand discuss the techniquesthat sometimes allow 7us to break it.This issue belongs to the new 7field ofinformation-based complexity, whichexamines the computational complexityof problems that cannot be solvedexactly. We shall also speculate brieflyon how 7 information-based complexity 7might enable us to prove that certainscientific questions can never be answeredbecause the necessary computingresources do not exist in the universe.If so, this condition would set limitson what is scientifically knowable.Information-based complexity focuseson the computational difficultyof so-called continuous problems.Calculating the movement of theplanets is an example. The motion isgoverned by a system of ordinary differentialequations—that is, equationsthat describe the positions of the planetsas a function of time. Because timecan take any real value, the mathemati-90B SCIENTIFIC AMERICAN January 1994

cal model is said to be continuous. Continuousproblems are distinct from discreteproblems, such as difference equationsin which time takes only integervalues. Difference equations appear insuch analyses as the predicted numberof predators in a study of predatorpreypopulations or the anticipated pollutionlevels in a lake.In the everyday process of doing scienceand engineering, however, continuousmathematical formulations predominate.They include a host of problems,such as ordinary and partial differentialequations, integral equations,linear and nonlinear optimization, integrationand surface reconstruction.These formulations often involve a largenumber of variables. For example, computationsin chemistry, pharmaceuticaldesign and metallurgy often entail calculationsof the spatial positions andmomenta of thousands of particles.Often the intrinsic difficulty of guaranteeingan accurate numerical solutiongrows exponentially with the numberof variables, eventually making theproblem computationally intractable.The growth is so explosive that in manycases an adequate numerical solutioncannot be guaranteed for situationscomprising even a modest number ofvariables.To state the issue of intractabilitymore precisely and to discuss possiblecures, we will consider the example ofcomputing the area under a curve. Theprocess resembles the task of computingthe vertical area occupied by a collectionof books on a shelf. More explicitly,we will calculate the area betweentwo bookends. Without loss of generality,we can assume the bookends restat 0 and 1. Mathematically, this summingprocess is called the computationof the definite integral. (More accurately,the area is occupied by an infinitenumber of books, each infinitesimallythin.) The mathematical input to thisproblem is called the integrand, a functionthat describes the profile of thebooks on the shelf.Calculus students learn to computethe definite integral by following a setof prescribed rules. As a result, the studentsarrive at the exact answer. Butmost integration problems that arise inpractice are far more complicated, andthe symbolic process learned in schoolcannot be carried out. Instead the integralmust be approximated numerically—thatis, by a computer. More exactly 7 ,one computes the integrand values atfinitely many points. These integrandvalues result from so-called informationoperations. Then one combines thesevalues to produce the answer.Knowing only these values does notJOSEPH F. TRAUB and HENRYK WOZNIAKOWSKI have been collaborating since 1973.Currently the Edwin Howard Armstrong Professor of Computer Science at ColumbiaUniversity, Traub headed the computer science department at Carnegie Mellon Universityand was founding chair of the Computer Science and Telecommunications Board ofthe National Academy of Sciences. In 1959 he began his pioneering research in what istoday called information-based complexity and has received many honors, includingelection to the National Academy of Engineering. He is grateful to researchers at theSanta Fe Institute for numerous stimulating conversations concerning the limits of scientificknowledge. Wozniakowski holds two tenured appointments, one at the Universityof Warsaw and the other at Columbia University. He directed the department ofmathematics, computer science and mechanics at the University of Warsaw and was thechairman of Solidarity there. In 1988 he received the Mazur Prize from the Polish MathematicalSociety. The authors thank the National Science Foundation and the Air ForceOffice of Scientific Research for their support.completely identify the true integrand.Because one can evaluate the integrandonly at a finite number of points, the informationabout the integrand is partial.Therefore, the integral can, at best,only be approximated. One typicallyspecifies the accuracy of theapproximation by stating thatthe error of the answer fallswithin some error threshold.Mathematicians represent thiserror with the Greek letter epsilon,£.Even this goal cannot beachieved without further restriction.Knowing the integrandat, say 7 , 0.2 and 0.5 indicatesnothing about the curvebetween those two points. Thecurve can assume any shapebetween them and thereforeenclose any area. In our bookshelfanalogy, it is as if an artbook has been shoved betweena run of paperbacks. Toguarantee an error of at moste, some global knowledge ofthe integrand is needed. Onemay need to assume, for example,that the slope of thefunction is always less than45 degrees—or that only paperbacksare allowed on thatshelf.In summary, an investigatortrying to solve an integralmust usually do it numericallyon a computer. The input tothe computer is the integrandvalues at some points. Thecomputer produces an outputthat is a number approximatingthe integral.The basic concept ofcomputational complexitycan now 7be introduced.We want to find the intrinsicdifficulty of solving theintegration problem. Assumethat determining integrandvalues and using combinatoryoperations, such as addition, multiplicationand comparison, each have agiven cost. The cost could simply be theamount of time a computer needs toperform the operation. Then the computationalcomplexity of this integra-One solution to an intractable problemSCIENTIFIC AMERICAN January 1994 90C

SAMPLING POINTS indicate where to evaluate functions in the randomized and average-casesettings. The points are plotted in two dimensions for visual clarity. Thepoints chosen can be spaced over regular intervals such as grid points (a), or inrandom positions (b). Two other types, so-called Hammersley points (c) and hyperbolic-crosspoints (d), represent optimal places in the average-case setting.ation problem can be defined as the minimalcost of guaranteeing that the computedanswer is within an error threshold,8, of the true value. The optimalinformation operations and the optimalcombinatory algorithm are thosethat minimize the cost.Theorems have shown that the computationalcomplexity of this integrationproblem is on the order of the reciprocalof the error threshold (1/8). Inother words, it is possible to choose aset of information operations and acombinatory algorithm such that thesolution can be approximated at a costof about 1/8. It is impossible to dobetter. With one variable, or dimension,the problem is rather easy. The computationalcomplexity is inversely proportionalto the desired accuracy.But if there are more dimensions tothis integration problem, then the computationalcomplexity scales exponentiallywith the number of variables. Ifd represents the number of variables,then the complexity is on the order of(1/8) J —that is, the reciprocal of theerror threshold raised to a power equalto the number of variables. If onewants eight-place accuracy (down to0.00000001) in computing an integralthat has three variables, then the complexityis roughly 10 24 . In other words,it would take a trillion trillion integrandvalues to achieve that level of accuracy.Even if one generously assumesthe existence of a sequential computerthat performs 10 billion function evaluationsper second, the job would take100 trillion seconds, or more than threemillion years. A computer with a millionprocessors would still take 100 millionseconds, or about three years.To discuss multivariate problemsmore generally, we must introduce oneadditional parameter, called r. This parameterrepresents the smoothness ofthe mathematical inputs. By smoothness,we mean that the inputs consistof functions that do not have any suddenor dramatic changes. (Mathematicianssay that all partial derivatives ofthe function up to order r are bounded.)The parameter takes on nonnegativeinteger values; increasing values indicatemore smoothness. Hence, r=0represents the least amount of smoothness(technically, the integrands areonly continuous—they are rather jaggedbut still connected as a single curve).Numerous problems have a computationalcomplexity that is on the orderof (1/8) d / r . For those of a more technicalpersuasion, multivariate integration,surface reconstruction, partial differentialequations, integral equationsand nonlinear optimization all have thiscomputational complexity.If the error threshold and the smoothnessparameter are fixed, then the computationalcomplexity depends exponentiallyon the number of dimensions.Hence, the problems become intractablefor high dimensions. An impedimenteven more serious than intractabilitymay occur: a problem may be unsolvable.A problem is unsolvable if onecannot compute even an approximationat finite cost. This is the case whenthe mathematical inputs are continuousbut jagged. The smoothness pa-90D SCIENTIFIC AMERICAN January 1994

ameter is zero, and the computationalcomplexity becomes infinite. Hence, formany problems with a large number ofvariables, guaranteeing that an approximationhas a desired error becomes anunsolvable or intractable task.Mathematically, the computationalcomplexity results we have describedapply to the so-called worst-case deterministicsetting. The "worst case" phrasingcomes from the fact that the approximationprovides a guarantee thatthe error always falls within e. In otherwords, for multivariate integration, anapproximation within the error thresholdis guaranteed for every integrandthat has a given smoothness. The word"deterministic'' arises from the factthat the integrand is evaluated at deterministic(in contrast to random) points.In this worst-case deterministic setting,many multivariate problems areunsolvable or intractable. Because theseresults are intrinsic to the problem,one cannot get around them by inventingother methods.One possible way to break unsolvabilityand intractability isthrough randomization. To illustratehow randomization works, wewill again use multivariate integration.Instead of picking points deterministicallyor even optimally, we allow (in aninformal sense) a coin toss to make thedecisions for us. A loose analogy mightbe sampling polls. Rather than ask everyregistered voter, a pollster conductsa small, random sampling to determinethe likely winner.Theorems indicate that with a randomselection of points, the computationalcomplexity is at most on the orderof the reciprocal of the square ofthe error threshold (l/£ 2 ). Thus, theproblem is always tractable, even if thesmoothness parameter is equal to zero.The workhorse of the randomizedapproach has been the Monte Carlomethod. Nicholas C. Metropolis andStanislaw M. Ulam suggested the ideain the 1940s. In the classical MonteCarlo method the integrand is evaluatedat uniformly distributed randompoints. The arithmetic mean of thesefunction values then serves as the approximationof the integral.Amazingly enough, for multivariateintegration problems, randomizationof this kind makes the computationalcomplexity independent of dimension.Problems that are unsolvable or intractableif computed from the best possibledeterministic points become tractableif approached randomly. (If r ispositive, however, then the classicalMonte Carlo method is not the optimalone; see box on the opposite page.)Average-Case ComplexityOne does not get so much for nothing.The price that must be paid forbreaking the unsolvabihty or intractabilityis that the ironclad guarantee thatthe error is at most 8 is lost. Insteadone is left only with a weaker guaranteethat the error is probably no morethan 8—much as a preelection poll isusually correct but might, on occasion,predict a wrong winner. In other words,a worst-case guarantee is impossible;one must be content with a weakerassurance.Randomization makes multivariateintegration and many other importantproblems computationally feasible. Itis not, however, a cure-all. Randomizationfails completely for some kinds ofproblems. For instance, in 1987 Greg W.Wasilkowski of the University of Kentuckyshowed that randomization doesnot break intractability for surface re-In the text, we mention that the average-case complexity of multivariate integrationis on the order of the reciprocal of the error threshold (1 /8) andthat for surface reconstruction, it is the square of that reciprocal (1/e 2 ). Forsimplicity, we ignored some multiplicative factors that depend on dand 8.Here we provide more rigorous statements.The average computational complexity, comp a v g (8, d; INT), of multivariateintegration is bounded byg ] (d) (_ i \id-w 02(d) f 1 y- 1)/2log-J < comp a v g (8, d; INT) < —j~ ^log£The average computational complexity, comp a v g (e, d; SUR), of surface reconstructionis bounded byg 3(d) f i yw-n g 4(d) f}yw-n~i 5 — V 1 0 9 e" J -c o mP a v g ( £ ' ^' S U R > ^ { l o g JGood estimates of g^d), g 2 (d), g 3 (d) and g 4 (d) are currently not known.SCIENTIFIC AMERICAN January 1994 91

construction. Is there an approach thatdoes and that works over a broad classof mathematics problems?There is indeed. It is the average-casesetting, in which we seek to break unsolvabilityand intractability by replacinga worst-case guarantee with a weakerone: that the expected error is atmost 8. The average-case setting imposesrestrictions on the kind of mathematicalinputs. These restrictions arechosen to represent what would happenmost of the time. Technically, theconstraints are described by probabilitydistributions; the distributions describethe likelihood that certain inputsoccur. The most commonly used distributionsare Gaussian measures and, inparticular, Wiener measures.Although it was known since the1960s that multivariate integration istractable on the average, the proof wasnonconstructive. That is, it did not specifythe optimal points to evaluate theintegrand, the optimal combinatory algorithmand the average computationalcomplexity. Attempts to apply ideasfrom other areas of computation to determinethese unknowns did not work.For example, evaluating the integrandat regularly spaced points, such as thoseon a grid, are often used in computation.But theorems have shown them tobe poor choices for the average-casesetting. A proof was given in 1975 byN. Donald YMsaker of the Universityof California at Los Angeles. It was latergeneralized in 1990 by Wasilkowski andAnargyros Papageorgiou, then studyingfor his Ph.D. at Columbia University.The solution came in 1991, whenWozniakowski found the construction.As sometimes happens in science, a resultfrom number theory, a branch ofmathematics far removed from average-casecomplexity theory, was crucial.Part of the key came from work onnumber theory by Klaus F. Roth of ImperialCollege, London, a 1958 FieldsMedalist. Another part was provided byrecent work by Wasilkowski.Let us describe the result more precisely.First, put the smoothness parameterat zero—that is, tackle a problemthat is unsolvable in the worst-case deterministicsetting. Next, assume thatintegrands are distributed according toa Wiener measure. If we ignore certainDiscrete Computational ComplexityThis article discusses intractability and breaking of intractability for multivariateintegration and surface reconstruction. These are two examplesof continuous problems. But what is known about the computational complexityof discrete, rather than continuous, problems? The famous travelingsalesman problem is an example of a discrete problem, in which the goal isto visit various cities in the shortest distance possible.A discrete problem is intractableif its computationalcomplexity increases exponentiallywith the number of its inputs.The intractability of manydiscrete problems in the worstcasedeterministic setting hasbeen conjectured but not yetproved. What is known is thathundreds of discrete problemsall have essentially the samecomputational complexity. Thatmeans they are all tractable orall intractable, and the commonbelief among experts is thatthey are all intractable. For technicalreasons, these problemsare said to be NP-complete. Oneof the great open questions indiscrete computational complexitytheory is whether the NPcompleteproblems are indeedintractable [see "Turing Machines,"by John E. Hopcroft; SCI­ENTIFIC AMERICAN, May 1984].multiplicative factors for simplicity'ssake, the average computational complexityhas been proved to be inverselyproportional to the error threshold (onthe order of 1/e) [see box on page 91].For small errors, the result is a majorimprovement over the classical MonteCarlo method, in which the cost is inverselyproportional to the square ofthe error threshold (1/e 2 ).The average case offers a differentkind of assurance from that providedby the randomized (Monte Carlo) setting.The error in the average-case settingdepends on the distribution of theintegrands, whereas the error in therandomized setting depends on a distributionof the sample points. In ourbooks-on-a-shelf analogy, the distributionin the average-case setting mightrule out the inclusion of many oversizebooks, whereas the distribution in therandomized setting determines whichbooks are to be sampled.In the average-case setting the optimalevaluation points must be deterministicallychosen. The best points areHammersley points or hyperbolic-crosspoints [see illustration on pages 90Dand 91]. These deterministic pointsoffer a better sampling than randomlyselected or regularly spaced (or grid)points. They make what would be impossibleto solve tractable on average.Is surface reconstruction also tractableon the average? This query is particularlyimportant because, as alreadymentioned, randomization does nothelp. Under the same assumptions weused for integration, we find that theaverage computational complexity is onthe order of 1/e 2 . Hence, surface reconstructionbecomes tractable on average.As was the case for integration,hyperbolic-cross points are optimal.We are now testing whether the averagecase is a practical alternative. APh.D. student at Columbia, Spassirnir H.Paskov, is developing software to comparethe deterministic techniques withMonte Carlo methods for integration.Preliminary results obtained by testingcertain finance problems suggest thesuperiority of the deterministic methodsin practice.In our simplified description, we ignoreda multiplicative factor that affectsthe computational complexity. This factordepends on the number of variablesin the problem. When the number ofvariables is large, that factor can becomehuge. Good theoretical estimatesof the factor are not known, and obtainingthem is believed to be very hard.Wozniakowski uncovered a solution:get rid of that factor. Specifically, we saya problem is strongly tractable if thenumber of function evaluations needed92 SCIENTIFIC AMERICAN January 1994