Video Stabilization for Robot Eye Using IMU-Aided Feature ... - KAIST

:: Algorithm of IMU-aided feature tracker ::prev curr prevload img , img , X , and Rcalculate H = KR KCtracked feature∞= 0IMUfeature−1from IMUIMU(a)for i=1to Nxpredip = x0prevfeature= H xprediprev∞ i(b)(c)Fig. 1. (a) Extracting feature in a previous frame. (b) Trackingfeature in a current frame with an original KLT feature tracker. (c)Tracking feature in a current frame with an IMU-aided KLTfeature tracker. Red, blue, and green rectangles indicate a positionof extracted feature from a previous frame (or an initial position ofthe feature in a current frame), a position of predicted the feature ina current frame, and a final target position of the tracked feature ina current frame, respectively.research. Original KLT tracker fails to track featureshown in figure 1(b) because the disparity betweeninitial (red rectangle) and target (green rectangle)feature is out of the tracking search range. However, ifthe initial position of the feature in a current frame ispredicted by 3D rotation information from IMU, theinitial position can be more close to the target featureposition enough to be within the tracking search rangeshown in figure 1(c).Correspondence pairs, x1 ↔ x2, between two imageshave following relation as equation (1).x 12= K ⎡⎣ RK − x 1+ t Z ⎤⎦(1)where x1and x 2are correspondence pairs betweentwo images ( x1and x 2are features in a previous anda current frames, respectively), Z is the depthinformation corresponding x1, K is the cameraintrinsic parameter, R and t are camera 3D rotationmeasured by IMU and 3D camera translation,respectively.If there is no camera translation motion or a depth forthe feature is infinite, the second term in equation (1)can be canceled like equation (2). Also, because cameramotion is generally much less than the depth for thefeature, equation (2) can be approximated as equation(3). In practice, large optical flows are mainly due tocamera rotation rather than translation.t = 0 ⎫ −1⎬ ⇒ t Z = 0 ⇒ x2 = KRK x (2)1Z →∞⎭for j=0to Nj+1max iteration−1T∑x∈Aj[ ( w ( p )) T( x)]p= H J I −pcurri= p + pif convergedbreakendendif well −trackedx = pCendendj+1tracked feature= C + 1xtracked featuret Z ⇒ t Z ≈ 0 ⇒ x ≈ KRK x (3)−12 1Finally, infinity homography calculated by cameraintrinsic parameter and camera rotation measured byIMU is used to predict an initial position of trackedfeature in a current frame as equation (4)−1H = ∞KRK (4)Figure 2 shows an algorithm for IMU-aided featurepredtracker. An initial position, x , of the feature in thecurrent frame is predicted by multiplying an infinityhomography, H∞, calculated by 3D rotation from IMUprevand the position, x , of the feature from the previousframe.3. VIDEO STABILIZATION FOR ROBOTEYESMost video stabilization techniques based on featuretracker such as KLT tracker may be failed to stabilize anunstable video because these trackers cannot sometimestrack feature well when large image motion occurs,above mentioned in section 2. In order to overcome thislarge image motion, IMU-aided KLT feature tracker isadopted to our video stabilization for robot eyes. Thist⎫⎪⎪⎪⎪⎪⎬⎪⎪⎪⎪⎪⎪⎭original KLT trackerFig. 2. Algorithm of IMU-aided KLT feature tracker. An initialposition of feature is predicted by an infinity homography( H∞)calculated with IMU.