Adaptive Critic Designs - Neural Networks, IEEE ... - IEEE Xplore

More documents

Recommendations

Info

1002 <strong>IEEE</strong> TRANSACTIONS ON NEURAL NETWORKS, VOL. 8, NO. 5, SEPTEMBER 1997goal of action’s training as follows:(21)Similar with what we stated in the Section II-C on GDHP,training of the action network is not carried out through theinternal pathways of the critic network leading from itsoutput to the input since it would be equivalent to returningto ADHDP. To train the action network, we use only thecritic’s outputs so as to meet (21). The goal (21) isthe same for all AD forms of ACD’s.III. GENERAL TRAINING PROCEDURE AND RELATED ISSUESThis training procedure is a generalization of that suggestedin [26], [30], [33], [38], and [43], and it is applicable toany ACD. It consists of two training cycles: critic’s andaction’s. We always start with critic’s adaptation alternating itwith action’s until an acceptable performance is reached. Weassume no concurrent adaptation of the model network, whichis previously trained offline, and any reasonable initializationfor and .In the critic’s training cycle, we carry out incremental optimizationof (2) and/or (5) by exploiting a suitable optimizationtechnique (e.g., LMS). We repeat the following operationstimes:for HDP, DHP, GDHP for ADHDP, ADDHP, ADGDHP1.0. Initialize t =0and R(0) Initialize t =0;R(0), and A(0)1.1. V (t) =f C [R(t);W C ] V(t)=f C [R(t);A(t);W C ]1.2. A(t) =f A [R(t);W A ] R(t+1)=f M [R(t);A(t);W M ]1.3. R(t +1)=A(t+1)=f A [R(t+1);W A ]f M [R(t);A(t);W M ]1.4. V (t +1)=f C [R(t+1);W C ] V(t+1)=f C [R(t+1);A(t+1);W C ]1.5. Compute E 1 (t);E 2 (t) from (2) and/or (5), and @V (t)=@W C ,tobeused in an optimization algorithm, then invoke the algorithm toperform one update of the critic’s weights W C . For the updateexample, see (4) and (11).1.6. t = t +1; continue from 1.1.Here stands for or , , ,andare the action, the critic and the modelnetworks, with their weights , respectively.In the action’s training cycle, we also carry out incrementallearning through an appropriate optimization routine, as in thecritic’s training cycle above. The list of operations for theaction’s training cycle is almost the same as that for the critic’scycle above (lines 1.0–1.6). However, we need to use (9) or(21), rather than (2) and/or (5); and instead ofbefore invoking the optimization algorithm forupdating the action’s weights [see (10) for the updateexample].The action’s training cycle should be repeated timeswhile keeping the critic’s weights fixed. We point outthat and are lengths of the corresponding trainingcycles. They are problem-dependent parameters of looselyspecified values. Ifwe can easily combineFig. 8. Test results of the autolander problem given for one of the mostchallenging cases where wind gusts were made 50% stronger than in standardconditions. After the ACD’s were trained on the number of landings shown,they were tested in 600 more trials, without any adaptation. Although theaverage training is much longer for GDHP and DHP than for HDP andADHDP, we could not observe an improvement of performance for eitherHDP or ADHDP if we continued their training further. Tight success meanslanding within a shortened touchdown region of the runway (it is the mostimportant characteristic). Loose success means landing within the limits ofthe standard runway. Similar results were obtained in various other flightconditions.both the cycles to avoid duplicating the computations in lines1.1–1.4. After the action’s training cycle is completed, onemay check action’s performance, then stop or continue thetraining procedure entering the critic’s training cycle again, ifthe performance is not acceptable yet. 3It is very important that the whole system consisting ofACD and plant would remain stable while both the networks ofACD undergo adaptation. Regarding this aspect of the trainingprocedure, we recommend to start the first training cycle ofthe critic with the action network trained beforehand to act asa stabilizing controller of the plant. Such a pretraining couldbe done on a linearized model of the plant (see, e.g., [45]).Bradtke et al. [26] proved that, in the case of the wellknownlinear quadratic regulation, a linear critic networkwith quadratic inputs trained by the recursive least squaresalgorithm in an ADHDP design converges to the optimal cost.If the regulator always outputs actions which are optimal withrespect to the target vector for the critic’s adaptation, i.e.,(22)where ,then the sequence is stabilizing, and it converges to theoptimal control sequence.Control sequences obtained through classical dynamic programmingare known to guarantee stable control, assuminga perfect match between the actual plant and its model usedin dynamic programming. Balakrishnan et al. [43] suggestedto stretch this fact over to a DHP-based ACD for linearand nonlinear control of systems with known models. Intheir design, one performs a training procedure similar to the3 Like many other training procedures, ours also implicitly assumes asufficiently varied set of training examples (e.g., different training trajectories)repeated often enough in order to satisfy persistent excitation—a property wellknown in a modern identification and adaptive control literature (see, e.g.,[37]).
PROKHOROV AND WUNSCH: ADAPTIVE CRITIC DESIGNS 1003above. Each training cycle is continued till convergence of thenetwork’s weights (i.e., , in the procedureabove). It is also suggested to use a new randomly chosenon every return to the beginning of the critic’s training cycle(line 1.6 is modified as follows: ; continue from 1.0).It is argued that whenever the action’s weights converge onehas a stable control, and such a training procedure eventuallyfinds the optimal control sequence.While theory behind classical dynamic programming demandschoosing the optimal vector of (22) for eachtraining cycle of the action network, we suggest incrementallearning of the action network in the training procedure above.A vector produced at the end of the action’s trainingcycle does not necessarily match the vector . However,our experience [28], [30], [44], [46], along with successfulresults in [33], [38], and [43], indicates that choosingprecisely is not critical.No training procedure currently exists that explicitly addressesissues of an inaccurate or uncertain model .It appears that model network errors of as much as 20%are tolerable, and ACD’s trained with such inaccurate modelnetworks are nevertheless sufficiently robust [30]. Althoughit seems consistent with assessments of robustness of conventionalneurocontrol (model-reference control with neuralnetworks) [31], [32], further research on robustness of controlwith ACD is needed, and we are currently pursuing this work.To allow using the training procedure above in presenceof the model network’s inaccuracies, we suggest to run themodel network concurrently with the actual plant or anothermodel, which imitates the plant more accurately than the modelnetwork but, unlike this network, it is not differentiable. Theplant’s outputs are then fed into the model network every sooften (usually, every time step) to provide necessary alignmentsand prevent errors of multiple-step-ahead predictionsfrom accumulating. Such a concurrently running arrangementis known under different names including teacher forcing[35] and series-parallel model [36]. After this arrangementis incorporated in an ACD, the critic will usually inputthe plant’s outputs, rather than the predicted ones from themodel network. Thus, the model network is mainly utilizedto calculate the auxiliary derivativesand.IV. SIMPLE ACD’S VERSUS ADVANCED ACD’SThe use of derivatives of an optimization criterion, ratherthan the optimization criterion itself, is known as being themost important information to have in order to find an acceptablesolution. In the simple ACD’s, HDP, and ADHDP,this information is obtained indirectly: by backpropagationthrough the critic network. It has a potential problem of beingtoo coarse since the critic network in HDP is not trained toapproximate derivatives of directly. An approach to improveaccuracy of this approximation has been proposed in [27]. It issuggested to explore a set of trajectories bordering a volumearound the nominal trajectory of the plant during the critic’straining, rather than the nominal trajectory alone. In spite ofthis enhancement, we still expect better performance from theadvanced ACD’s.Furthermore, Baird [39] showed that the shorter the discretizationinterval becomes, the slower the training of AD-HDP proceeds. In continuous time, it is completely incapableof learning.DHP and ADDHP have an important advantage over thesimple ACD’s since their critic networks build a representationfor derivatives of by being explicitly trained on themthrough and . For instance, in thearea of model-based control we usually have a sufficientlyaccurate model network and well-definedand. To adapt the action network we ultimately needthe derivatives or , rather than the functionitself. But an approximation of these derivatives is alreadya direct output of the DHP and ADDHP critics. Althoughmultilayer neural networks are well known to be universalapproximators of not only a function itself (direct output of thenetwork) but also its derivatives with respect to the network’sinputs (indirect output obtained through backpropagation) [41],we note that the quality of such a direct approximation isalways better than that of any indirect approximation for givensizes of the network and the training data. Work on a formalproof of this advantage of DHP and ADDHP is currently inprogress, but the reader is referred to the Section V for ourexperimental justification.<strong>Critic</strong> networks in GDHP and ADGDHP directly approximatenot only the function but also its derivatives. Knowingboth and its derivatives is useful in problems where availabilityof global information associated with the functionitself is as important as knowledge of the slope of , i.e., thederivatives of [40]. Besides, any shift of attention paid tovalues of or its derivatives during training can be readilyaccommodated by selecting unequal learning rates andin (11) (see Section II-C). In Section II-C we described threeGDHP designs. While the design of Fig. 5 seems to be themost straightforward and beneficial from the viewpoint ofsmall computational expenses, the designs of Figs. 3 and 4use the critic network more efficiently.Advanced ACD’s include DHP, ADDHP, GDHP, andADGDHP, the latter two being capable of emulating allthe previous ACD’s. All these designs assume availabilityof the model network. Along with direct approximation ofthe derivatives of , it contributes to a superior performanceof advanced ACD’s over simple ones (see the next Sectionfor examples of performance comparison). Although the finalselection among advanced ACD’s should certainly be basedon comparative results, we believe that in many applicationsthe use of DHP or ADDHP is quite enough. We also notethat the AD forms of the designs may have an advantage intraining recurrent action networks.V. EXPERIMENTAL STUDIESThis section provides an overview of our experimental workon applying various ACD’s to control of dynamic systems. Fordetailed information on interesting experiments carried out by
Page 2 and 3: 998 IEEE TRANSACTIONS ON NEURAL NET
Page 4 and 5: 1000 IEEE TRANSACTIONS ON NEURAL NE

Adaptive Critic Designs - Neural Networks, IEEE ... - IEEE Xplore

Create successful ePaper yourself

Delete template?

Save as template?