Model Selection Based on the Modulus of Continuity

More documents

Recommendations

Info

References1. Akaike, H.: Information theory and an extansion of the maximum likelihood principle. In Petrov, B. and Csaki,F., editors, 2nd Interanational Symposium on Information Theory, VOL. 22 (1973), 267–281, Budapest.2. Schwartz, G.: Estimating the dimension of a model. Ann. Stat., 6 (1978),461–464.3. Wahba, G., Golub, G., and Heath, M.: Generalized cross-validation as a method for choosing a good ridge parameter.Technometrics, 21 (1979), 215–223.4. Rissanen, J.: Stochastic complexity and modeling. Annals of Statistics, 14 (1986), 1080–1100.5. Barron, A., Rissanen, J., and Yu, B.: The minimum description length principle in coding and modeling. IEEETransactions on Infomation Theory, 44 (1998), 2743–2760.6. Dean P. Foster and Edward I. George: The risk inflation criterion for multiple regression. Annals of Statistics, 22(1994), 1947–1975.7. Vladimir N. Vapnik: Statistical Learning Theory. J. Wiley, 19988. Olivier Chapelle, Vladimir Vapnik, Yoshua Bengio: <strong>Model</strong> selection for small sample regression. Machine Learning,48 (2002), 315-333, Kluwer Academic Publishers.9. Vladimir Cherkassky, Xuhui Shao, Filip M. Mulier, Vladimir N. Vapnik: <strong>Model</strong> complexity control for regressionusing VC generalization bounds. IEEE transactions on Neural Networks, VOL. 10, NO. 5 (1999),1075–1089.10. G.G. Lorentz: Approximation of functions. Chelsea Publishing Company, New York, 1986.28
AppendixA-1. Proof of Theorem 1The main idea of proving this theorem is to decompose the risk functional R(f n ) L1intofour components: the modulus of continuity for target function w(f, h), the modulus of continuityfor approximation function w(f n , h), additive noise and the empirical risk functional|y i − f n (x i )|. First, we will prove the theorem for noiseless target function, that is, y = f(x).For each x ∈ X, there is x i ∈ X i , i = 1, · · · , N, with |x − x i | < h. This implies that|f(x) − f n (x)| |f(x) − f(x i )| + |f(x i ) − f n (x i )| + |f n (x) − f n (x i )| ω(f, h) + |f(x i ) − f n (x i )| + |f n (x) − f n (x i )|. (37)Let f n (x) = ∑ ni=1 w iφ i (x). For each x, y ∈ X with |x − y| h, we haveThat is,|f n (x) − f n (y)| = |n∑i=1w i (φ i (x) − φ i (y))| max1in |φ i(x) − φ i (y)| max1in {ω(φ i, h)}n∑|w i |i=1n∑|w i |. (38)i=1n∑|f(x) − f n (x)| ω(f, h) + |f(x i ) + f n (x i )| + max {ω(f, h)} |w i | (39)1infrom (37) and (38). By taking the integral of (39) in X i satisfying P (X i ) = 1/N, for i =i=11, · · · , N, we get∫X i|f(x) − f n (x)|dP (x) 1 N(ω(f, h) + |f(x i ) − f n (x i )| + max{ω(φ i , h)})n∑|w i | . (40)i=1By taking the integral of (39) in X, we get∫N∑∫R(f n ) L1 = |f(x) − f n (x)|dP (x) = |f(x) − f n (x)|dP (x)X iXi=1 R emp (f n ) L1 + ω(f, h) + max{ω(φ i , h)}29n∑|w i |. (41)i=1
Page 2: the trade-off between the under-fit
Page 10 and 11: Therefore, the upper bound of risk
Page 12 and 13: similar complexity while the fourth
Page 14 and 15: To see the risk prediction of the m
Page 16 and 17: 32.521.510.50AIC BIC SEB MC(a)32.52
Page 24 and 25: 0.80.60.40.20 0.025 0.05 0.1(a)0.80
Page 26 and 27: 10.90.8train errortest errorBICSEBA
Page 30: Now, let us consider the target fun

Model Selection Based on the Modulus of Continuity

Create successful ePaper yourself

Delete template?

Save as template?