a dynamic programming approach to optimization for pavement ...

2nd North American Pavement Management Conference (1987)MARKOV PREDICTION MODEL1t is not possible to describe the functioning of the dynamicprogramming algorithm without first describing the predictionmodel used, as the two are inextricably linked. The following isa necessarily abbreviated and generalized desciption of themodel; a more detailed and comprehensive description has alreadybeen published elsewhere (3,5). Much of the terminology used inthis description also is used in dynamic programming.The PC1 range of 0 to 100 is divided into 10 states,each state being 10 PC1 points wide. A pavement is modelled tobegin its life in near-perfect condition, and to deteriorate asit is subjected to a sequence of duty-cycles. A duty-cycle isdefined as one year's imposition of the effects of weather andtraffic. A state vector indicates the probability of a pavementsection being in each of the 10 states in any given year. Figure1 shows the schematic representation of state, state vector andduty cycle.The sections are grouped into families, and all of thesections of a family are categorized into one of the ten statesat any age. It is assumed that all of the pavement sections arein state 1 (PC1 of 90 to 100) at an age of 0 years.To model the way in which the pavement deterioriateswith time, it is necessary to identify the Markov probabilitymatrix. The assumption made is that the pavement condition willnot drop by more than one state in a single year. Thus, thepavement will either stay in its current state or transit to thenext lowest state in one year. The probability transition matrixhas a diagonal structure as shown in Figure 2. The entry of 1 inthe last row of the transition matrix indicates a trappingstate. The pavement condition cannot transit from this stateunless repair action is performed.The state vector for any duty cycle t,S(t), is obtainedby multiplying the initial state vector S(0) by the transitionmatrix P raised to the power of t. ThusIf the transition matrix probabilities can be estimated, thefuture condition of the road at any duty cycle (time) t can bepredicted.The probabilities are estimated using a non-linear programmingapproach which has its objective function theminimization of the absolute distance between the actual PC1versus age data points, and the expected (predicted) PC1 for thecorresponding age generated by the Markov chain using the nineprobability parameters. It has been found that this approach canaccurately model the pavement deterioration over time. A sampleoutput of this program is shown in Figure 3.TRB Committee AFD10 on Pavement Management Systems is providing the information contained herein for use by individual practitioners in state and localtransportation agencies, researchers in academic institutions, and other members of the transportation research community. The information in this paper wastaken directly from the submission of the author(s).

2nd North American Pavement Management Conference (1987)I STATE IISTATE VECTORTRB Committee AFD10 on Pavement Management Systems is providing the information contained herein for use by individual practitioners in state and localtransportation agencies, researchers in academic institutions, and other members of the transportation research community. The information in this paper wastaken directly from the submission of the author(s).

2nd North American Pavement Management Conference (1987)INTRODUCTION TO DYNAMIC PROGRAMMINGDynamic programming is an "approach" to optimization. It is notbased upon a single, well-defined procedure such as the simplexalgorithm of linear programming. Rather, it seeks to take asingle, complicated problem and break it down into a number ofsimpler, more easily solvable problems. The solution to theseproblems is generally much faster and requires much lesscomputational effort.The principle of optimality, the foundation of dynamicprogramming, states that "every optimal policy consists only ofoptimal subpolicies" (6). Instead of examining all possiblecombinations of decisions, dynamic programming examines a small,carefully chosen subset of the ombinations, rejecting thosecombinations that cannot possibly t a d to an optimal solution.If performed correctly, the subset examined is guaranteed tocontain the optimal solution.\Dynamic programming has the advantage over almost allclassical optimization techniques that it will yield the absoluteor global maxima or minima rather than local optima. Dynamicprogramming is also extremely robust in that it can handleintegrality, negativity and discreetness of variables and otherconstraints very easily. It also, by its nature, produces theoptimal solution to all of its constituent subproblems. Thesesolutions may be of interest to the pavement manager, especiallyin time-varying problems such as pavement performance.The major drawback with dynamic programming is that theproposed problem to be solved may not be formulatable in a mannercapable of using the power of dynamic programming effectively.However, if it is possible to set up the problem in an effectiveform, dynamic programming provides an outstanding optimizationtool.Structure of dynamic programming: The basic components ofdynamic programming are states, stages, decision variables,returns and transformation or transition functions (6). Aphysical system is considered to progress through a series ofconsecutive stages. In pavement performance, each year is viewedas a stage.At each stage, the system must be capable of being fullydescribed by the state variables or state vector. In the presentcase, as described earlier, each state is a 10 PC1 bracket forevery pavement family and the condition of the pavement at anyyear (stage) can be defined as being in one of the ten states.At each stage, for every possible state, there must be aset of allowable decisions. The decisions being made in thedynamic programming model are what repair alternative toimplement in each state at every stage.Finally, there is the transformation or transitionfunction- If a process is in a given state, and a feasibledecision is made, there must be a function that determines towhich state the process moves. In general, dynamic programmingtransformation functions can be deterministic or stochastic. InTRB Committee AFD10 on Pavement Management Systems is providing the information contained herein for use by individual practitioners in state and localtransportation agencies, researchers in academic institutions, and other members of the transportation research community. The information in this paper wastaken directly from the submission of the author(s).

this particular case, the transition fupction is defined by theMarkov probability matrix derived in the curve-fitting processdescribed earlier and hence is a stochastic process.In summary, the problem set-up for this dynamicprogramming formulation is:MINIMIZE: Expected Cost over a specified life-cycle lengthsubject to keeping all sections above a defined performancestandard.The dynamic programming parameters are:STATES: Each bracket of 10 PC1 points in a family.STAGES: Each year in the analysis period.DECISION VARIABLES: Which maintenance treatment to apply.TRANSFORMATION FUNCTION: The Markov Transition ProbabilityMatrix defines the transformation.RETURN: Expected cost if a particular decision is made in eachstate at each stage.INPUTS REQUIRED FOR THE DYNAMIC PROGRAMMING ALGORITHMThe inputs required for the dynamic programming algorithm are:2nd North American Pavement Management Conference (1987)'fpij ; i = 1,..,10 statesj = 1, ..,m families2. Cost of applying treatment k to family j in state iCi.k ; k = l,..,n maintenance alternatives4. Number of Years in the life-cycle analysis; N5. Interest Rate ; r6. Idflation Rate ; f7. Rate of Increase in Funding ; qdeteriorate to before performing some major maintenance.designated by S j*1. Markov Transition Probabilities for state i of matrix jRoutine maintenance is always designated as k -1.The cost is entered on a dollar per square yard basis.3. Feasibility Indicator for alternative k when in state i offamily j.Rijk = 1 if maintenance alternative is feasible= 0 if maintenance alternative is infeasible8. The associated benefit over one year of being in state iBi = 95, 85,.., 5 for i = 1,2,.., 10The benefit is taken to be the area under the PC1 curve overa period of one year.9. The minimum allowable state for each family, i.e. the loweststate that the network manager will allow a particular family toThis isTRB Committee AFD10 on Pavement Management Systems is providing the information contained herein for use by individual practitioners in state and localtransportation agencies, researchers in academic institutions, and other members of the transportation research community. The information in this paper wastaken directly from the submission of the author(s).

2nd North American Pavement Management Conference (1987)FAMILY JRoutine Maintenence(K = 1)FAMILY J'Non Routine OptionY STAGE N N-1 ii 118n-2I I!:STAGE N N-1 • • • n n-1 n-2FIGURE 4TRB Committee AFD10 on Pavement Management Systems is providing the information contained herein for use by individual practitioners in state and localtransportation agencies, researchers in academic institutions, and other members of the transportation research community. The information in this paper wastaken directly from the submission of the author(s).

2nd North American Pavement Management Conference (1987)after the repair alternative is carried out. The family that thepavement moves to as a result of having this alternative9 j 9performed is defined in the input transformation matrix. pOrexample, if a thin overlay is performed on an AC pavementsection, that section will move to the thin overlay family forperformance prediction after the overlay is placed.a*The optimal cost is then given by:with the associated optimal maintenance alternative to beperformed for this (i,j) familylstate cqmbination in year N - ~being the choice' of k that minimizes the above expression.This backward recursion is performed for everysuccessive year of the analysis period until the analysis foryear 0, or stage N, is reached.Dynamic Programming OutputThe output from the dynamic programming program consists of:(i) a file containing the optimal maintenance alternative inevery year for every familylstate combination.(ii) the discounted present worth costs expected to be accruedover the life-cycle specified if the optimal decisions areimplemented.(iii) the expected effectiveness accrued as a result of followingthe optimal decisions is calculated for every familylstatecombination.(iv) the effectiveness/cost ratio for every familylstatecombination is calculated.Thus, all that is necessary is to define which familylstatecombination any particular section belongs to, and the optimalmaintenance alternative and associated cost and effectiveness arereadily obtained. As the intention of the research is to produceupdated programs for both the PAVER and Micro PAVER systems, itwas imperative that all of the programs be executable on a microcomputer.The dynamic programming program executes extremelyquickly at this level.A very short example follows to illustrate the operationof the program. Performance curves were developed for the GreatLakes Naval Center based on PC1 surveys performed there.Families were defined on the basis of branch use and surfacetype. For the branch use of "roadway", there were four familiesdefined: asphalt concrete, surface treated, thin overlay andstructural overlay.There were four maintenance alternatives considered;routine maintenance, surface treatment, thin overlay and structuraloverlay. Dollar cost as a function of PC1 was defined forboth initial repair cost and subsequent routine maintenance costin the Purdue University research [4]. Markov probabilitycalculations for each family were performed, and the probabilitytransition matrices obtained.TRB Committee AFD10 on Pavement Management Systems is providing the information contained herein for use by individual practitioners in state and localtransportation agencies, researchers in academic institutions, and other members of the transportation research community. The information in this paper wastaken directly from the submission of the author(s).

2nd North American Pavement Management Conference (1987)The dynamic programming program was then run on a Compaq286 microcomputer with an 80287 math co-processor installed. Aneffective interest rate of 5% was input, and a life-cycleanalysis of 30 years was specified. The program took 45 secondsto produce the entire output for every year of the 30 years.Figure 5 illustrates a small part of the output. Thisis the output in year 1, the present time, for family 4. This isthe family composed of sections with structural overlays. As canbe seen, routine maintenance was selected as the optimal treatmentto apply in the upper states, surface treatment in themiddle states and structural overlay in the lower states.obviously, this result is completely dependent upon the cost andperformance relationships defined, but the overall trend of theoptimal decisions is certainly reasonable.FUTURE DEVELOPMENTSThere are a number of potential areas for further use of thisapproach in pavement management. It should be possible toexplore the impact of deferred maintenance by limiting themaintenance options available to just routine maintenance for agiven number of years, and noting the detrimental effect thatthis policy has on the cost and E/C ratios for the life-cycleperiod.The o~tput from dynamic programming fits easily into anetwork prioritization framework. The flowchart describing theoverall intended process is shown in Figure 6. The details ofthe prioritization package will be described in the future. Itis also intended to validate the dynamic programming outputs fora number of different databases by comparing its output to thatobtained from a more traditional, deterministic approach to lifecycleeffectiveness and cost calculation.REFERENCES1. Shahin a'nd Kohn, "Overview of the 'PAVER' Pavement ManagementSystem", Technical Manuscript M-310, USA-CERL, January 1982.2. Shahin, Nunez, Broten, Carpenter and Sameh, "New Techniquesfor Modeling Pavement Deterioration", Presented at TransportationResearch Board Annual Meeting, January 1987.3. Butt, Shahin, Feighan, Carpenter, "Pavement PerformancePrediction Model Using the Markov Process", Presented atTransportation Research Board Annual Meeting, January 1987.4. Reichelt, Sharaf, Sinha and Shahin, "The Relationship ofPavement Maintenance Costs to the Pavement Condition Index",USA-CERL Interim Report M-87/02, February 1987.5. Keane and Wu, "An Integrated Decision-Making Methodology forOptimal Maintenance Strategies", Master's Thesis, University ofIllinois, 1985.6. Cooper and Cooper, "Introduction to Dynamic Programming",Pergamon Press, 1981.2.205TRB Committee AFD10 on Pavement Management Systems is providing the information contained herein for use by individual practitioners in state and localtransportation agencies, researchers in academic institutions, and other members of the transportation research community. The information in this paper wastaken directly from the submission of the author(s).

2nd North American Pavement Management Conference (1987)rWINTENANCECILTERNATIVEPRESENT WORTHCOSTEFFECTIVENESSE/C RATIOR.n.R.H.R.H.8.1.S.T.S.T.5.0.S.O.S.O.S.O.17.3521.2022.5223.2623.8624.7125-8128.8433.8438.842402.02276.62305.02353.92341 -92331.92342.02332.02322 0 .2312.0138.5107.4102.3101.198.294.490.780.968.659.5FIGURE 5IPAVER DATABASE RhW DATA IiINPUT: Common Characteristics to clas~ify sectionsinto families (rurfacm type, traffic, primary sourceof distress).OUTPUT: Families of pavemmnt sections with PC1versus Age data.IHARKOV PREDICTION PROCESSINPUT: Families with PC1 vetrue Age data.OUTPUT: narkov Transition Probabilities for eachfamily and maintenance alternative.IIDYNAHIC PROGRAHHING PROGRAM IINPUT: narkov Transition Probabilities, costs bystatm and family for each alternative, planninghorizon, interest rates, performance standards byfamily, benefit by state.OUTPUTI Optimal maintmmcr drcision (on thr basirof minimized Cost or maximized B/C ratio) for eachstate of each family1 associated cost and benefit.IIPRIORITIZATION PROGRAHI IINPUT: B/C ratio for mach sectionr weights byfamily (and possibly by state), Actual Budget,my nmcmssary (user-definmd) rections that mustbe repairmd, evrn if wb-optimal.OUTPUT: Ranked list of sections that can bmrepaired within budget limitations.IPROJECT LEVEL ANALYSISFIGURE 6IITRB Committee AFD10 on Pavement Management Systems is providing the information contained herein for use by individual practitioners in state and localtransportation agencies, researchers in academic institutions, and other members of the transportation research community. The information in this paper wastaken directly from the submission of the author(s).