例題ではじめる部分空間法 - パターン認識へのいざない - - 東京農工大学

69 ()MATLAB/Octave ·3 download http://www.tuat.ac.jp/ ∼ s-hotta/RSJ2012(2)

05163849 69 () [1]対 応 付 け2知 パターン (クラスが 未 知 ) 未7クラス( 概 念 )利 用練 パターン (クラスが 既 知 ) 訓(3)

別 辞 書 識別 部 識 69 [1]識 別 演 算 部5力 出前 処 理 部特 徴 抽 出 部照 合未 知 パターン(4)

69 (Bayes decision rule)maxjP (ω j |x) = P (ω k |x) ⇒ x ∈ ω k P (ω j |x) = P (ω j)p(x|ω j )∑j P (ω j)p(x|ω j )P (ω j ): ω j ()P (ω j |x): x ω j ()p(x|ω j ): ω j x ()p(x|ω j ) ()(6)

69 [2, 3, 4] : () · ()(7)

69 : () : () 顔5学 習訓 練 標 本 を 部 分 空 間 で 圧 縮 クラスごとにエントロピー 最 小 化猫バイクcj 1+ c j 2+ cjr=認 識c + ck1 k 2 + ckr=認 識 結 果(8)・・・~顔未 知 標 本・・・顔 を 表 現 できない バイクで

69 : minu i−d∑p(u i ) log p(u i )i=1: + ∑ ri=1 w i(u ⊤ i x) 2 x xuu 2 1TU jx− x(9)

69 (1925- ) OCR“ASPET/71” & () (1910-1993)() ()“” (10)

69 1/3 [5]0 “” subspacesub () ( ⇐⇒super) () ( 210 )(11)

21 69 2/301⃝(12)3

69 3/3線 型 部 分 空 間 でなければ同 じ 直 線 上 に 存 在 する 点 同 士 の121足 し 算 ・ 引 き 算 ・スカラー 倍 が同 じ 直 線 上 にのらない02⃝ 3⃝2⃝ (affine subspace, linear manifold, linearvariety)(13)

69 d :⎛ ⎞x 1⎜x = . ⎟⎝⎠ , x ⊤ = (x 1 , ..., x d )x dx ⊤ x = ∑ di=1 x2 i =∥x∥2x : ∥x∥ = √ x ⊤ x:cos θ =x⊤ y∥x∥∥y∥:∥ x y∥ 2 = 2(1 − cos θ)∥x∥ − ∥y∥半 径 1n + 1 n (n ≥ 3 )(14)

69 r d u 1 , u 2 , ..., u r d × r () U = (u 1 |u 2 | · · · |u r ) (U ⊤ U = I)xc 2u 2u 10x~c 1R r ˜x c = U ⊤ x = (u ⊤ 1 x|u ⊤ 2 x| · · · |u ⊤ r x) ⊤R d ˜x ˜x = Uc = UU ⊤ x (x U )UU ⊤ (15)

69 ∂∂x x i ∂∂x i( ) ⊤∂ ∂∂x = ∂· · ·∂x 1 ∂x d (1) f = x ⊤ x = ∑ di=1 x2 i x ( ) ⊤∂∂x f = ∇f = ∂f∂x 1· · · ∂f∂x d= (2x1 · · · 2x d ) ⊤ = 2x (2)d × d A f = x ⊤ Ax x ∂∂x f = 2Ax (3) f = y ⊤ Ax x ∂∂x f = A⊤ y(16)

69 argmaxsubject to argmaxsubject to max f(x)f(x) max f(x)∥x∥=1∥x∥ = 1 x f(x) argmaxxf(x)f(x) x arg argmin f(x) xsubject to · · ·· · · s.t. max f(x), s.t. · · ·xx f(x) (17)

69 [6]SVM g(x) = 0 f(x) λ L = f(x) − λg(x) 1 ∂∂x L = ∇f − λ∇g = 0, ∂L∂λ = 0 d + 1 x 1 ,...,x d , λ d + 1 1 1 (18)

69 () A u ⊤ u = 1 u ⊤ Au max u ⊤ Auus.t. u ⊤ u − 1 = 0 λ L = u ⊤ Au − λ(u ⊤ u − 1)∂L/∂u = 0, ∂L/∂λ = 0∂L/∂u = 2Au − 2λu = 0 Au = λu 2 A ( u ⊤ u ⊤ u = 1 )2 2 (19)

69 (subspace method) [2, 4] [7] [8] ()(20)

69 n d x i (i = 1, ..., n) C j U j x x maxj=1,...,C {∥U⊤ j x∥} = ∥U ⊤ k x∥ ⇒ x ∈ class k(21)

69 class jclass jx U T jd jxxU j0x0x分 空 間 法 最 小 距 離 法部 x ⇒ d 2 j = ∥x∥2 − ∥U ⊤ j x∥2 ∥x∥ 2 ∥U ⊤ j x∥2 (22)

69 cos θ 1/3U d × r 1 x U Uc c ⊤ c = 1 :max x ⊤ Uc = cos θcs.t. c ⊤ c − 1 = 0(1) ∥x∥ = 1∥Uc∥ = √ (Uc) ⊤ (Uc) = √ c ⊤ Ic = 1j xuu 2 1Uc(23)

69 cos θ 2/3 (1) L = x ⊤ Uc − λ 2 (c⊤ c − 1)∂L/∂c = U ⊤ x − λc = 0 c = 1 λ U⊤ x (2)c ⊤ c = 1 λ 2 (U⊤ x) ⊤ (U ⊤ x) = 1 λ 2 ∥U⊤ x∥ 2 = 1 λ = ∥U ⊤ x∥ λ (2) c =U⊤ x∥U ⊤ x∥ x U 1 (24)

69 cos θ 3/3 c = (U ⊤ x)/∥U ⊤ x∥ x ⊤ Uc cos θ = x ⊤ Uc = x⊤ UU ⊤ x∥U ⊤ x∥= ∥U⊤ x∥ 2∥U ⊤ x∥ = ∥U⊤ x∥ = λ cos θ (U ⊤ x)/∥U ⊤ x∥ ∈ R r (UU ⊤ x)/∥U ⊤ x∥ ∈ R d cos θ ∥U ⊤ x∥ xuu 2 1TUU xUc =TU x(25)

69 U 1/2(26)0

69 U 2/2n x 1 , ..., x n d i = ∥x i ∥ 2 − (u ⊤ 1 x i ) 2 u 1 ∥x i ∥ 2 u 1 (u ⊤ 1 x i ) 2 u 1 :(∑n n)∑max (u ⊤ 1 x i ) 2 = u ⊤ 1 x i x ⊤ i u 1u 1i=1i=1s.t. u ⊤ 1 u 1 − 1 = 0 D = ∑ ni=1 x ix ⊤ i λ 1 r u r D r λ r r U j = (u 1 | · · · |u r ) (27)

69 d d × n X = (x 1 | · · · |x n ) d d × d D = XX ⊤ n ≪ d n × n N = X ⊤ X U X ⊤ X n ( n )(28)

69 N = X ⊤ X ∈ R n×n r 0 λ 1 , ..., λ r (D = XX ⊤ ) v 1 , ..., v r ∈ R n i d u i [9]u i = ±Xv i√λi ( EVD.m )(29)

69 : : [10, 11] P (ω j |x) = P (ω j)p(x|ω j )∑j P (ω j)p(x|ω j )(30)

69 ω j N (x|µ j , Σ j ) =1(2π) d 2 |Σ j | 1 2{exp − 1 }2 (x − µ j) ⊤ Σ −1j(x − µ j )µ j : d × 1 Σ j : d × d ⇓x i −x i (31)

69 () (32)

69 ˆµ j = 0, ˆΣj = R j R j = 1 n j∑ nji=1 x ix ⊤ i : N (x|ˆµ j , ˆΣ j ) =(1(2π) d/2 |R j | exp 1/2− 1 2 x⊤ R −1j)x(33)

69 N (x|ˆµ j , ˆΣ j ) =(1(2π) d/2 |R j | 1/2 exp− 1 2 x⊤ R −1j)x⇓g j (x) = −∥Λ − 1 2jU ⊤ j x∥2 − ∑ di=1 ln λ ji R j = U j Λ j U ⊤ j(34)

69 g j (x) = −∥Λ − 1 2jU ⊤ j x∥2 − ∑ di=1 ln λ ji x i −x i ()⇓(35)

69 w 1 ≥ w 2 ≥ · · · ≥ w d > 0 g j (x) = −∥W − 1 2 U ⊤ j x∥2 = − ∑ di=1 1 w i(u ⊤ ji x)2xxx2−( uT x)ji20u jix u T ji ⇒ (36)

69 1/w 1 ≤ 1/w 2 ≤ · · · ≤ 1/w d > 0 ⇒ w 1 ≥ w 2 ≥ · · · ≥ w d > 0 () w i : S j (x) = ∑ ri=1 w i(u ⊤ ji x)2 = ∥W 1 2 U ⊤ j x∥2 = w ⊤ c j: S j (x) x () (37)

69 CLAFIC (): w i = 1 (i = 1, ..., r) (): w i = λ ji/λ j1: w i = r − i + 1 (i = 1, ..., r)CLAFICweight valueproposedmultiplesimilaritydimensionality r ⇒ (38)

69 r∑S j (x) = (r − i + 1)(u ⊤ jix) 2i=1= r(u ⊤ j1x) 2 + (r − 1)(u ⊤ j2x) 2 + · · · + (u ⊤ jrx) 2r∑ ∑r−1= (u ⊤ jix) 2 + (u ⊤ jix) 2 + · · · + (u ⊤ j1x) 2i=1i=1(39)

69 1 http://www.gaussianprocess.org/gpml/data/ USPS 16 × 16 256 · 4649 makedata.m WSC.m(40)

69 makedata.m makedata.m USPS [−1, +1] 0 pair-wise makedata.m [0, +1] 0 9 usps.mat (41)

69 WSC.mWSC.m WSC.m Figure 1 10 Figure 2imgnum U j (r = 13) 1 r imgnum (42)

69 : i j ij class 0 1 2 3 4 5 6 7 8 9 total0 680 0 2 1 5 7 72 1 17 1 7861 0 642 0 0 1 0 2 0 2 0 6472 10 1 377 15 11 6 6 9 19 0 4543 5 0 2 369 2 19 0 4 15 2 4184 4 15 7 0 356 1 9 1 8 42 4435 10 0 3 18 10 286 8 0 15 5 3556 32 7 9 0 3 3 355 0 5 0 4147 2 3 1 0 7 0 0 348 2 39 4028 4 9 5 15 5 7 0 1 273 12 3319 0 10 0 1 45 0 0 25 2 316 399total 747 687 406 419 445 329 453 389 358 417 4649(43)

69 C nclass d d · n ndata i x i trai(:,ii) i trai_label(ii) i x test(:,ii) i test_label(ii) j U j U j C(j).U x y x ⊤ y x’*y x ∥x∥ = √ x ⊤ x norm(x)(44)

69 Figure 1 (45)

sample test0 class 1 class 2 class 3 class 4 class 5 class 6 class 7 class 8 class 9class 69 Figure 2 U j (U j U ⊤ j x)(46)

69 ¯λ = ∑ ri=1 λ i/ ∑ rank(D)j=1λ j r ()95testvalidationaccuracy [%]90850 20 40 60 80 100dimensionality of each subspace r(47)

69 r 95linear weightacuracy [%]90CLAFIC850 20 40 60 80 100multiple similarity(48)dimensionality r

69 CPU 1.86GHz 2GB32bit WindowsMATLAB (R14) (r = 13) () SVM SVM ()CLAFICCLAFICone-against-allone-against-all, RBF Kernel(49)

69 (%) (s) (KB) 96.5 0.003 5140.2 97.1 0.002 5140 92.0 0.002 532 85.8 3 × 10 −5 20 97.4 0.01 9298 (r = 13) 96.9 6 × 10 −4 260 (¯λ = 0.95) 94.3 5 × 10 −4 231 SVM 93.9 136.8 3656 SVM 98.0 222 5352(50)

69 犬体 のパターンを 観 測 しても 何 であるかはわからない 単同 じクラスから 由 来 する 複 数 のパターン (CSM): (MSM): (51)

69 Compound Subspace Method (CSM) [11]P (ω|X) =P (ω)p(X|ω)∑ω P (ω)p(X|ω) ∼n ∑T∑r∑k=1 f=1 i=1f w i ( f u ⊤ i f x k ) 2Multiple Kernel Learning (52)

69 (compound Bayesian decision problem) [12]P (ω|X) =P (ω)p(X|ω)∑ω P (ω)p(X|ω) n : X = (x 1 |x 2 | · · · |x n )n (context): ω = (ω(1), ..., ω(n)) ⊤ x i ω(i) カレイc⃝カレイひらめ(53)ひらめ

69 (compound Bayesian decision problem)P (ω|X) =P (ω)p(X|ω)∑ω P (ω)p(X|ω)ω c n p(X|ω) ⇓ CSM x i T (54)

69 CSMn x 1 , ..., x n i.i.d. P (ω j |X ) =n∏P (ω j ) p(x i |ω j )i=1c∑ n∏P (ω j ) p(x i |ω j )j=1 i=1n∑ r∑n∑n∑S j(X ) = w l (u ⊤ jlx i) 2 = ∥W 1 2 U⊤j x i∥ 2 = S j(x i)i=1 l=1i=1i=1(55)

69 CSMx T P (ω j |F) =T∏P (ω j ) p f ( f x|ω j )f=1c∑P (ω j )j=1 f=1T∏p f ( f x|ω j ) · (f )F j (F) def=T∑r f∑f=1 l=1T∑T∑f w l ( f u ⊤ jl f x) 2 = ∥ f W 1 2 f U ⊤ j f x∥ 2 = f S j ( f x)f=1f=1(56)

69 CSMn X T P (ω j | ¯X ) =T∏ n∏P (ω j ) p f ( f x i |ω j )c∑P (ω j )f=1 i=1T∏j=1 f=1 i=1n∏p f ( f x i |ω j ) · C j ( ¯X ) def=T∑f=1 i=1n∑f S j ( f x i )(57)

69 (mutual subspace method, MSM) [13] (cos θ ) (58)

69 V ∈ R d×r d: r d < d U ∈ R d×r i: r i < d VbUc maxb,c(Vb)⊤ (Uc) = cos θs.t. b ⊤ b − 1 = 0, c ⊤ c − 1 = 0(3)(59)

69 1/2 (3) L = (Vb) ⊤ (Uc) − λ 2 (b⊤ b − 1) − µ 2 (c⊤ c − 1)∂L/∂b = 0, ∂L/∂c = 0 V ⊤ Uc = λb (4)U ⊤ Vb = µc (5) (4) c (5) b V ⊤ UU ⊤ Vb = λµb, U ⊤ VV ⊤ Uc = λµc (6)(60)

69 2/2 (4) b ⊤ (5) c ⊤ λ = µ b ⊤ V ⊤ Uc = b ⊤ λb = λ (7)c ⊤ U ⊤ Vb = c ⊤ µc = µ (8) (6) λ(= µ) V ⊤ UU ⊤ V (= U ⊤ VV ⊤ U ) cos θ V ⊤ UU ⊤ V U ⊤ VV ⊤ U (61)

69 2 ETH-80 dataset [14] 8 1 10 1 41 (128 × 128 ) (30 )(62)

dog1 dog1-000-000.pngdog1-022-000.pngdog1-022-090.pngdog1-022-180.pngdog1-022-270.pngdog1-035-045.pngdog1-035-135.pngdog1-035-225.pngdog1-035-315.pngdog1-045-000.pngdog1-045-090.pngdog1-045-180.png 69 2 dog1-045-270.pngdog1-066-027.pngdog1-066-063.pngdog1-066-117.pngdog1-066-153.pngdog1-066-207.pngdog1-066-243.pngdog1-066-297.pngdog1-066-333.pngdog1-068-000.pngdog1-068-090.pngdog1-068-180.pngdog1-068-270.pngdog1-090-000.pngdog1-090-022.pngdog1-090-045.pngdog1-090-068.pngdog1-090-090.pngdog1-090-112.pngdog1-090-135.pngdog1-090-158.pngdog1-090-180.pngdog1-090-202.pngdog1-090-225.png(63)dog1-090-248.pngdog1-090-270.pngdog1-090-292.pngdog1-090-315.pngdog1-090-338.png

69 2 dog1-066-117.png66°117° + + (64)

69 make_data.m Y = 0.2989R + 0.587G + 0.114B 15 × 15 (65)

69 21 () 10 () 22 () (1 11 )(66)

69 (doubutsu.mat) 33157331736IMG(:)255173157 66120x=6625541186IMGreshape(x, [3 3])416120186IMG(:): reshape: (67)

69 CSM.m MSM.m rirdnclassntraintesttrai(ii).Xtest(ii).XEVDtrai(ii).Utest(ii).Utrai(ii).labeltest(ii).labelCONF ii (225 × 21 ) ii (225 × 21 ) ii ii ii ii (68)

69 CPU 2.8GHz 3.5GB32bit WindowsMATLAB (R14) WSC: MSM: CSM: (%) (s)WSC (r d = 20) 42.4 0.0007MSM (r i = 5,r d = 6) 71.8 0.002CSM (r d = 20) 80.9 0.002(69)

69 r r ()tangent distancek-subspace clustering(fuzzy) k-varieties clustering(70)

69 ( d = 2) local subspace classifier ()http://www.cvlab.cs.tsukuba.ac.jp/ ∼ subspace/(71)

69 I[1] , , , , “,” ,Aug. 1998.[2] S. Watanabe, P.F. Lambert, C.A. Kulikowski, J.L. Buxton, and R. Walker,“Evaluation and selection of variables in pattern recognition,” Comp. & Info.Sciences, vol. 2, (Julius Tou, ed.). New York: Academic Press, pp. 91–122, 1967.[3] T. Iijima, H. Genchi, and K. Mori, “A theory of character recognition by patternmatching method,” Proc. of 1st Int’l J. Conf. on Pattern Recognition,pp. 50–56, 1973.[4] E. Oja, Subspace methods of pattern recognition, Research Studies Press, 1983. , 1986.[5] , 1, 1996.[6] , - -, ,2005.[7] S. Watanabe, Knowing and guessing : A quantitative study of inference andinformation, John Wiley & Sons, New York, 1969., , : , , , 1987.(72)

69 II[8] - -1999.[9] , --, , 2003.[10] , “,” , PRMU2010,vol. 110, no. 296, pp. 13–18, Nov. 2010 .[11] , “,” , PRMU2010,vol. 110, no. 330, pp. 45–50, Dec. 2010.[12] J.F. Hannan and H. Robbins, “Asumptotic solutions of the compound decisionproblem for two completely specified distributions,” Annals of MathematicalStatistcs, vol. 26, no. 1, pp. 37-51, 1955.[13] “ · ”(D), vol. J68-D, no. 3, pp. 345–352, 1985.[14] B. Leibe and B. Schiele, “Analyzing appearance and contour based methods forobject categorization,” Proc. of CVPR, pp. 409–415, 2003.(73)

69 1 1: f ( x)= const.∇fg(x) = 0∇g g(x) = 0 f(x) ∇f = λ∇g(74)

69 2 2: Au = λu (u ≠ 0) u A λ 10x 2-2 02468-4x 1-10 -5 0 5 10 -10( ) 1/5 1/4A A =x Ax 1/4 1/5 × ()(75)-8-6