Optimal Tracking Control for a Class of Nonlinear Discrete-Time ...

ZHANG et al.: OPTIMAL TRACKING CONTROL FOR A CLASS OF NONLINEAR DISCRETE-TIME SYSTEMS WITH TIME DELAYS BASED ON HDP 1861VI. CONCLUSIONIn this paper, we proposed an effective HDP algorithmto solve optimal tracking problem for a class of nonlineardiscrete-time systems with time delays. First we defined aperformance index for time delay systems. Then a novel iterationHDP algorithm has been developed to solve the optimaltracking control problem. Two neural networks have been usedto facilitate the implementation of the iteration algorithm.Simulation examples have demonstrated the effectiveness ofthe proposed optimal tracking control algorithm.REFERENCES[1] M. Z. Manu and J. Mohammad, Time-Delay Systems Analysis, Optimizationand Applications. New York: North-Holland, 1987.[2] D. H. Chyung, “On the controllability of linear systems with delay incontrol,” IEEE Trans. Autom. Control, vol. 15, no. 2, pp. 255–257, Apr.1972.[3] D. H. Chyung, “Controllability of linear systems with multiple delaysin control,” IEEE Trans. Autom. Control, vol. 15, no. 6, pp. 694–695,Dec. 1970.[4] H. Y. Shao and Q. L. Han, “New delay-dependent stability criteriafor neural networks with two additive time-varying delay components,”IEEE Trans. Neural Netw., vol. 22, no. 5, pp. 812–818, May 2011.[5] J. P. Richard, “Time-delay systems: An overview of some recentadvances and open problems,” Automatica, vol. 39, no. 10, pp. 1667–1694, Oct. 2003.[6] S. C. Tong, Y. M. Li, and H. G. Zhang, “Adaptive neural networkdecentralized backstepping output-feedback control for nonlinear largescalesystems with time delays,” IEEE Trans. Neural Netw., vol. 22,no. 7, pp. 1073–1086, Jul. 2011.[7] V. N. Phat and H. Trinh, “Exponential stabilization of neural networkswith various activation functions and mixed time-varying delays,” IEEETrans. Neural Netw., vol. 21, no. 7, pp. 1180–1184, Jul. 2010.[8] Y. J. Liu, C. L. P. Chen, G.-X. Wen, and S. C. Tong, “Adaptive neuraloutput feedback tracking control for a class of uncertain discrete-timenonlinear systems,” IEEE Trans. Neural Netw., vol. 22, no. 7, pp. 1162–1167, Jul. 2011.[9] M. Wang, S. S. Ge, and K. S. Hong, “Approximation-based adaptivetracking control of pure-feedback nonlinear systems with multipleunknown time-varying delays,” IEEE Trans. Neural Netw., vol. 21,no. 11, pp. 1804–1816, Nov. 2010.[10] W. S. Chen and L. C. Jiao, “Adaptive tracking for periodically timevaryingand nonlinearly parameterized systems using multilayer neuralnetworks,” IEEE Trans. Neural Netw., vol. 21, no. 2, pp. 345–351, Feb.2010.[11] W. Y. Lan and J. Huang, “Neural-network-based approximate outputregulation of discrete-time nonlinear systems,” IEEE Trans. NeuralNetw., vol. 18, no. 4, pp. 1196–1208, Jul. 2007.[12] C. M. Zhang, G. Y. Tang, and S. Y. Han, “Approximate design ofoptimal tracking controller for systems with delayed state and control,”in Proc. IEEE Int. Conf. Control Autom., Christchurch, New Zealand,Dec. 2009, pp. 1168–1172.[13] Y. D. Zhao, G. Y. Tang, and C. Li, “Optimal output tracking controlfor nonlinear time-delay systems,” in Proc. 6th World Congr. Intell.Control Autom., Dalian, China, Jun. 2006, pp. 757–761.[14] Y. M. Park, M. S. Choi, and K. Y. Lee, “An optimal tracking neurocontrollerfor nonlinear dynamic systems,” IEEE Trans. Neural Netw.,vol. 7, no. 5, pp. 1009–1110, Sep. 1996.[15] J. Fu, H. B. He, and X. M. Zhou, “Adaptive learning and control forMIMO system based on adaptive dynamic programming,” IEEE Trans.Neural Netw., vol. 22, no. 7, pp. 1133–1148, Jul. 2011.[16] R. E. Bellman, Dynamic Programming. Princeton, NJ: Princeton Univ.Press, 1957.[17] P. J. Werbos, “A menu of designs for reinforcement learning over time,”in Neural Networks for Control, W. T. Miller, R. S. Sutton, and P. J.Werbos, Eds. Cambridge, MA: MIT Press, 1991, pp. 67–95.[18] A. Al-Tamimi, F. L. Lewis, and M. Abu-Khalaf, “Model-free Q-learningdesigns for linear discrete-time zero-sum games with application toH-infinity control,” Automatica, vol. 43, no. 3, pp. 473–481, Mar. 2007.[19] D. Vrabie, O. Pastravanu, M. Abu-Khalaf, and F. L. Lewis, “Adaptiveoptimal control for continuous-time linear systems based on policyiteration,” Automatica, vol. 45, no. 2, pp. 477–484, Feb. 2009.[20] K. G. Vamvoudakis and F. L. Lewis, “Online actor critic algorithm tosolve the continuous-time infinite horizon optimal control problem,”Automatica, vol. 46, no. 5, pp. 878–888, May 2010.[21] W. B. Powell, Approximate Dynamic Programming: Solving the Cursesof Dimensionality. New York: Wiley, 2009.[22] C. Zheng and S. Jagannathan, “Generalized Hamilton–Jacobi–Bellmanformulation-based neural network control of affine nonlinear discretetimesystems,” IEEE Trans. Neural Netw., vol. 19, no. 1, pp. 90–106,Jan. 2008.[23] P. He and S. Jagannathan, “Reinforcement learning neural-networkbasedcontroller for nonlinear discrete-time systems with inputconstraints,” IEEE Trans. Syst., Man, Cybern., Part B: Cybern.,vol. 37, no. 2, pp. 425–436, Apr. 2007.[24] J. J. Murray, C. J. Cox, G. G. Lendaris, and R. Saeks, “Adaptivedynamic programming,” IEEE Trans. Syst., Man, Cybern., Part C:Appl. Rev., vol. 32, no. 2, pp. 140–153, May 2002.[25] B. H. Li and J. Si, “Approximate robust policy iteration using multilayerperceptron neural networks for discounted infinite-horizon Markovdecision processes with uncertain correlated transition matrices,” IEEETrans. Neural Netw., vol. 21, no. 8, pp. 1270–1280, Aug. 2010.[26] J. Si and Y. T. Wang, “Online learning control by association andreinforcement,” IEEE Trans. Neural Netw., vol. 12, no. 2, pp. 264–276,Mar. 2001.[27] R. Enns and J. Si, “Helicopter trimming and tracking control usingdirect neural dynamic programming,” IEEE Trans. Neural Netw.,vol. 14, no. 4, pp. 929–939, Jul. 2003.[28] F. Y. Wang, N. Jin, D. R. Liu, and Q. L. Wei, “Adaptive dynamicprogramming for finite-horizon optimal control of discrete-timenonlinear systems with ε-error bound,” IEEE Trans. Neural Netw.,vol. 22, no. 1, pp. 24–36, Jan. 2011.[29] H. G. Zhang, Q. L. Wei, and D. R. Liu, “An iterative adaptivedynamic programming method for solving a class of nonlinear zero-sumdifferential games,” Automatica, vol. 47, no. 1, pp. 207–214, Jan. 2011.[30] P. J. Werbos, “Approximate dynamic programming for real-time controland neural modeling,” in Handbook of Intelligent Control: Neural,Fuzzy, and Adaptive Approaches, D.A.WhiteandD.A.Sofge,Eds.New York: Van Nostrand, 1992, ch. 13.[31] J. Seiffertt, S. Sanyal, and D. C. Wunsch, “Hamilton–Jacobi–Bellmanequations and approximate dynamic programming on time scales,”IEEE Trans. Syst., Man, Cybern., Part B: Cybern., vol. 38, no. 4, pp.918–923, Aug. 2008.[32] Y. Zhao, S. D. Patek, and P. A. Beling, “Decentralized Bayesian searchusing approximate dynamic programming methods,” IEEE Trans. Syst.,Man, Cybern., Part B: Cybern., vol. 38, no. 4, pp. 970–975, Aug. 2008.[33] F. L. Lewis and D. Vrabie, “Reinforcement learning and adaptivedynamic programming for feedback control,” IEEE Circuits Syst. Mag.,vol. 9, no. 3, pp. 32–50, Sep. 2009.[34] F. Y. Wang, H. G. Zhang, and D. R. Liu, “Adaptive dynamicprogramming: An introduction,” IEEE Comput. Intell. Mag., vol. 4,no. 2, pp. 39–47, May 2009.[35] S. J. Bradtke, B. E. Ydestie, and A. G. Barto, “Adaptive linear quadraticcontrol using policy iteration,” in Proc. Amer. Control Conf., vol.3.Baltimore, MD, Jun. 1994, pp. 3475–3479.[36] A. Al-Tamimi, F. L. Lewis, and M. Abu-Khalaf, “Discrete-timenonlinear HJB solution using approximate dynamic programming:Convergence proof,” IEEE Trans. Syst., Man, Cybern., Part B: Cybern.,vol. 38, no. 4, pp. 943–949, Aug. 2008.[37] H. G. Zhang, Q. L. Wei, and Y. H. Luo, “A novel infinite-time optimaltracking control scheme for a class of discrete-time nonlinear systemsvia the greedy HDP iteration algorithm,” IEEE Trans. Syst., Man,Cybern., Part B: Cybern., vol. 38, no. 4, pp. 937–942, Aug. 2008.[38] H. G. Zhang, Y. H. Luo, and D. R. Liu, “Neural-network-based nearoptimalcontrol for a class of discrete-time affine nonlinear systemswith control constraints,” IEEE Trans. Neural Netw., vol. 20, no. 9, pp.1490–1503, Sep. 2009.[39] A. Isidori, Nonlinear Control Systems II. New York: Springer-Verlag,2005.[40] S. B. Gershwin and D. H. Jacobson, “A controllability theory fornonlinear systems,” IEEE Trans. Autom. Control, vol. 16, no. 1, pp.37–46, Feb. 1971.[41] A. Bicchi, A. Marigo, and B. Piccoli, “On the reachability of quantizedcontrol systems,” IEEE Trans. Autom. Control, vol. 47, no. 4, pp.546–563, Apr. 2002.[42] A. Ben-Israel and T. N. E. Greville, Generalized Inverse: Theory andApplications, 2nd ed. New York: Springer-Verlag, 2002.