available here - Documentation

available here - Documentation

available here - Documentation


Create successful ePaper yourself

Turn your PDF publications into a flip-book with our unique Google optimized e-Paper software.

OpenCVGary BradskiSenior Scientist, Willow GarageConsulting Prof. Stanford UniversityICRA 2010 ROS Tutorial1

Outline• Overview• Quick Tour• Pinhole Camera• Homography• Camera Calibration• Gradient Features• Demo– Object Recogntion Using Gradient Features– Node: Finding a Chessboard and its PoseICRA 2010 ROS Tutorial2

opencv.willowgarage.comOpenCV Overview:General Image Processing Functions> 500 algorithmsRobot supportImage PyramidsSegmentationTransformsGeometricdescriptorsFeaturesCameracalibration,Stereo, 3DUtilities andData StructuresMachineLearning:• Detection,• RecognitionTrackingMatrix MathICRA 2010 ROS Tutorial3Fitting

(‏MLL‏)‏ Machine Learning LibraryCLASSIFICATION / REGRESSIONFast Approximate NN (FLANN)Extremely Random TreesRandom ForestsStatistical Boosting, 4 flavorsCARTNaïve Bayes(‏propagation MLP (BackSVMFace Detector(‏matching (Histogram(‏Correlation‏)‏CLUSTERINGK-MeansEM(‏distance (MahalanobisCCCCBAACACBBAACBAABBCBCCAAACBABBCCBABBCC B A BBCBBCTUNING/VALIDATIONCross validationBootstrappingVariable importanceSampling methodshttp://opencv.willowgarage.com44

OpenCV History• Original goal:• Accelerate the field by lowering the bar to computer vision• Find compelling uses for the increasing MIPS out in the market• Timeline:Willow1050• Staffing:• Climbed in 1999 to average 7 first couple of years• Starting 2003 support declined between zero and one withexception of transferring the machine learning frommanufacturing work I led (equivalent of 3 people).• Support to zero the couple of years before Willow.Gary Bradski5• 5 people last year, 7 people this year5

OpenCV License• OpenCV has a BSD license– It is free and open for• Commercial and Research use• In whole or in part– Supported by:• Willow Garage (5 Developers)• Nvidia (2 Developers, CUDA)• Intel (Test standards)• Google (8 Summer of Code Interns)• Community (User groupICRA 2010 ROS Tutorial

OpenCV Tends Towards Real Timehttp://opencv.willowgarage.com

• Works on:Software Engineering– Linux, Windows, Mac OS• Languages:– C++, Python, C• Online documentation:– Online reference manuals: C++, C and Python.• We’vebeenexpandingUnittestcode• testGoogle’sorcxxonstandarizesoonWill system.ICRA 2010 ROS Tutorial

OpenCV Directory StructureICRA 2010 ROS Tutorial

New C++ API: Usage Exampledouble calcGradients(const IplImage *src, int aperture_size = 7){CvSize sz = cvGetSize(src);IplImage* img16_x = cvCreateImage( sz, IPL_DEPTH_16S, 1);IplImage* img16_y = cvCreateImage( sz, IPL_DEPTH_16S, 1);cvSobel( src, img16_x, 1, 0, aperture_size);cvSobel( src, img16_y, 0, 1, aperture_size);IplImage* imgF_x = cvCreateImage( sz, IPL_DEPTH_32F, 1);IplImage* imgF_y = cvCreateImage( sz, IPL_DEPTH_32F, 1);cvScale(img16_x, imgF_x);cvScale(img16_y, imgF_y);IplImage* magnitude = cvCreateImage( sz, IPL_DEPTH_32F, 1);cvCartToPolar(imgF_x, imgF_y, magnitude);double res = cvSum(magnitude).val[0];cvReleaseImage( &magnitude );cvReleaseImage(&imgF_x);cvReleaseImage(&imgF_y);cvReleaseImage(&img16_x);cvReleaseImage(&img16_y);Focus DetectorC: C++:double contrast_measure(const Mat& img){Mat dx, dy;Sobel(img, dx, 1, 0, 3, CV_32F);Sobel(img, dy, 0, 1, 3, CV_32F);magnitude(dx, dy, dx);return sum(dx)[0];}}return res;ICRA 2010 ROS Tutorial10

Python Face Detector Node: 1The Setup#!/usr/bin/python"""This program is demonstration python ROS Node for face and object detection using haar-like features.The program finds faces in a camera image or video stream and displays a red box around them. Pythonimplementation by: Roman Stanchak, James Bowman"""import roslibroslib.load_manifest('opencv_tests')import sysimport osfrom optparse import OptionParserimport rospyimport sensor_msgs.msgfrom cv_bridge import CvBridgeimport cv# Parameters for haar detection# From the API:# The default parameters (scale_factor=2, min_neighbors=3, flags=0) are tuned# for accurate yet slow object detection. For a faster operation on real video# images the settings are:# scale_factor=1.2, min_neighbors=2, flags=CV_HAAR_DO_CANNY_PRUNING,# min_size=

Python Face Detector Node: 2if __name__ == '__main__':pkgdir = roslib.packages.get_pkg_dir("opencv2")haarfile = os.path.join(pkgdir, "opencv/share/opencv/haarcascades/haarcascade_frontalface_alt.xml")parser = OptionParser(usage = "usage: %prog [options] [filename|camera_index]")parser.add_option("-c", "--cascade", action="store", dest="cascade", type="str", help="Haar cascade file, default %default", default = haarfile)(options, args) = parser.parse_args()cascade = cv.Load(options.cascade)br = CvBridge()def detect_and_draw(imgmsg):img = br.imgmsg_to_cv(imgmsg, "bgr8")# allocate temporary imagesgray = cv.CreateImage((img.width,img.height), 8, 1)small_img = cv.CreateImage((cv.Round(img.width / image_scale),cv.Round (img.height / image_scale)), 8, 1)# convert color input image to grayscalecv.CvtColor(img, gray, cv.CV_BGR2GRAY)# scale input image for faster processingcv.Resize(gray, small_img, cv.CV_INTER_LINEAR)The Coreif(cascade):faces = cv.HaarDetectObjects(small_img, cascade, cv.CreateMemStorage(0),haar_scale, min_neighbors, haar_flags, min_size)if faces:for ((x, y, w, h), n) in faces:# the input to cv.HaarDetectObjects was resized, so scale the# bounding box of each face and convert it to two CvPointspt1 = (int(x * image_scale), int(y * image_scale))pt2 = (int((x + w) * image_scale), int((y + h) * image_scale))cv.Rectangle(img, pt1, pt2, cv.RGB(255, 0, 0), 3, 8, 0)cv.ShowImage("result", img)cv.WaitKey(6)cv.EqualizeHist(small_img, small_img)rospy.init_node('rosfacedetect')image_topic = rospy.resolve_name("image")rospy.Subscriber(image_topic, sensor_msgs.msg.Image, detect_and_draw)rospy.spin()ICRA 2010 ROS Tutorial 12

• Main siteOpenCV Important Links– http://opencv.willowgarage.com• User site– http://opencv.willowgarage.com/wiki/FullOpenCVWiki• User group (42000 members)– http://tech.groups.yahoo.com/group/OpenCV/• Download/Install– http://opencv.willowgarage.com/wiki/InstallGuide• OpenCV Book:– http://www.amazon.com/Learning-OpenCV-Computer-Vision-Library/dp/0596516134• Developer meeting notes– http://pr.willowgarage.com/wiki/OpenCVMeetingNotesICRA 2010 ROS Tutorial

2010 Plans• Re-org‏into‏coherent‏processing‏“stacks”– Texture and patch based object recognition– Visual Odometry and VSLAM– Stereo– Image stitching– 3D model capture– User contribICRA 2010 ROS Tutorial

OpenCV Book:Learning OpenCVComputer Vision theory and practice• http://www.amazon.com/Learning-OpenCV-Computer-Vision-Library/dp/0596516134Japanese and Chinesetranslations of the bookare now available.

Summary• Comprehensive computer vision and ML library• BSD license, free for commercial or research• C++, C, Python, Linux, Windows, Mac• Re-orging into processing stacksICRA 2010 ROS Tutorial16

Outline• Overview• Quick Tour• Pinhole Camera• Homography• Camera Calibration• Gradient Features• Demo– Object Recogntion Using Gradient Features– Node: Finding a Chessboard and its PoseICRA 2010 ROS Tutorial17

Canny Edge Detector18

Distance Transform• Distance field from edges of objectsFlood FillingICRA 2010 ROS Tutorial19

Hough TransformGary Bradski, Adrian Kahler 200820

Space Variant vision: Log-Polar TransformScreen shots by Gary Bradski, 200521

Scale Spacevoid cvPyrDown(IplImage* src,IplImage* dst,IplFilter filter = IPL_GAUSSIAN_5x5);Chart by Gary Bradski, 2005Gary Bradski, 2009void cvPyrUp(IplImage* src,IplImage* dst,IplFilter filter = IPL_GAUSSIAN_5x5);Gary Bradski (c) 2008 22

ThresholdsScreen shots by Gary Bradski, 200523

Histogram EqualizationScreen shots by Gary Bradski, 200524


Morphological Operations Examples• Morphology - applying Min-Max. Filters and its combinationsImage IErosion IBDilatation IBOpening IoB= (IB)BClosing I•B= (IB)BGrad(I)= (IB)-(IB)TopHat(I)= I - (IB) BlackHat(I)= (IB) - I

Image textures• Inpainting:• Removes damage to images, in this case, it removes the text.ICRA 2010 ROS Tutorial

Segmentation• Pyramid, mean-shift, graph-cut• Here: WatershedScreen shots by Gary Bradski, 2005Gary Bradski, 2009Gary Bradski (c) 2008 28

Recent Algorithms: GrabCut• Graph Cut based segmentationImages by Gary Bradski, © 2010ICRA 2010 ROS Tutorial29

(‏Davies Motion Templates (work with James• Object silhouette• Motion history images• Motion history gradients• Motion segmentation algorithmsilhouette MHI MHGCharts by Gary Bradski, 200530

Segmentation, Motion TrackingandGesture RecognitionMotionSegmentationMotionSegmentationPoseRecognitionGestureRecognitionScreen shots by Gary Bradski, 2005

New Optical Flow Algorithms// opencv/samples/c/lkdemo.cintmain(…){…?=capture*‏CvCapturecvCaptureFromCAM(camera_id) :cvCaptureFromFile(path);if( !capture ) return -1;for(;;) {IplImage* frame=cvQueryFrame(capture);if(!frame) break;imageprocessandcopy…//‏cvCalcOpticalFlowPyrLK( …)‏;(result,”‏LkDemo‏“)‏cvShowImagec=cvWaitKey(30); // run at ~20-30fps speedif(c >= 0) {// process key}}cvReleaseCapture(&capture);}lkdemo.c, 190 lines(needs camera to run)I ( x dx , y dy , t dt ) IGX / tX Ib,/ x ( dx ( x,y), G / dt ) I I I2xxI ( x,y,t );,IyI,xII/ y ( dyy2y,b/ dt );It IIxy

Tracking with CAMSHIFT• Control game with headScreen shots by Gary Bradski, 2005

ProjectionsScreen shots by Gary Bradski, 2005

Stereo‏…‏Depth‏from‏Triangulation• Involved topic, here we will just skim the basicgeometry.• Imagine two perfectly aligned image planes:Depth “Z” and disparity “d” are inversly related:ICRA 2010 ROS Tutorial35

Stereo• In aligned stereo, depth is from similar triangles:Tl ( x x )Z frTZZxlfT xr• Problem: Cameras are almost impossible to align• Solution: Mathematically align them:ICRA 2010 ROS Tutorial36All: Gary Bradski and Adrian Kaehler: Learning OpenCV

Stereo Rectification• Algorithm steps are shown at right:• Goal:– Each row of the image contains the same worldpoints– “Epipolar‏constraint”Result: Epipolar alignment of features:All: Gary Bradski and Adrian Kaehler: Learning OpenCVICRA 2010 ROS Tutorial37

Summary• Wide ranging functionality in– Matrix funtions– Image processing– Feature extraction– Segmentation– Motion– Stereo– Object recognitionICRA 2010 ROS Tutorial38

Outline• Overview• Quick Tour• Pinhole Camera• Homography• Camera Calibration• Gradient Features• Demo– Object Recogntion Using Gradient Features– Node: Finding a Chessboard and its PoseICRA 2010 ROS Tutorial39

Most basic camera model:Pinhole Camera-- Brunelleschi, XVth CenturyICRA 2010 ROS TutorialMarc Pollefeys comp256, Lect 2

Perspective ProjectionWorld to Camera CoordinatesRobert Collins, CSE486Marc PollefeysPerspective Matrix Equation(camera coords Pt in world to pt on image) x' y ' z ' f 00.‏visiontoapproach”‏triangle’ssimilar‏“Ap0f0M001Pint C X0Y 0 Z0 1x y ffXZXZ

Consequences: Parallel lines meet• There exist vanishing pointsMarc PollefeysICRA 2010 ROS TutorialGary Bradski and Adrian Kaehler: Learning OpenCV42

Implications For Perception*Same‏size‏things‏get‏smaller,‏we‏hardly‏notice…Parallel‏lines‏meet‏at‏a‏point…* A Cartoon Epistemology: http://cns-alumni.bu.edu/~slehar/cartoonepist/cartoonepist.htmlICRA 2010 ROS Tutorial

Biological Implications For Perception:Perception must be mapped to a space variant gridLogrithmic in natureICRA 2010 ROS Tutorial44Steve Lehar

Pinhole‏Cameras‏are‏“Ideal”‏but…• PROBLEM:– Pinhole cameras cannot gather enough lightfor practical applications.• SOLUTION:– Use a lense to focus lots of light into a smallarea• TRADEOFF:– Lense distortionFrom:http://www.physics.uiowa.edu/~umallik/adventure/geo-optics/lightnw.htmICRA 2010 ROS Tutorial45

Aberrations2 types :1. geometrical2. chromaticgeometrical : small for paraxial raysstudy through 3 rd order opticschromatic : refractive index function ofwavelength46Marc Pollefeys

Geometrical aberrations spherical aberration astigmatism distortion comaAberrations are reduced by combining lenses-- This is why a good lens costs to much.47Marc Pollefeys

Geometric Distortion(The major source of distortions)RADIAL DISTORTION:magnification/focal length differentfor different angles of inclinationpincushion(tele-photo)barrel(wide-angle)Can be corrected! (if parameters are know)ICRA 2010 ROS Tutorial48Marc Pollefeys


Distortion Field• Points from an object in the world shouldproject to pinhole perspective locations.• Charts‏of‏errors‏from‏“Ideal”:Jean-Yves BouguetGary Bradski, CS223A, 50ICRA 2010 ROS Tutorial

But, Pinhole Cameras also Have Problems:ICRA 2010 ROS Tutorial

Summary• Camera Model– Complications necessary to collect more light• Simple camera, the pinhole, allows imageformation• Consequence is perspective mapping• Pinholes‏don’t‏allow‏enough‏light• Use lenses instead, but suffer from distortionsICRA 2010 ROS Tutorial52

Outline• Overview• Quick Tour• Pinhole Camera• Homography• Camera Calibration• Gradient Features• Demo– Object Recogntion Using Gradient Features– Node: Finding a Chessboard and its PoseICRA 2010 ROS Tutorial53

Homography• Maps one plane to another– In our case: A plane in the world to the camera plane– Great notes on this: Robert Collins CSE486• http://www.cse.psu.edu/~rcollins/CSE486/lecture16.pdf– Derivation details: Learning OpenCV 384-387Gary Bradski and Adrian Kaehler: Learning OpenCVICRA 2010 ROS TutorialPerspective Matrix Equation(camera coords Pt in world to pt on image) x' y ' z ' f 000f0p M001Pint C X0Y 0 Z0 1x y 54ffXZXZ

HomographyWant: Mapping of Pt on Object to pt on image:Projection Derivation (2):Robert Collins, CSE486Projection Derivation (1):Robert Collins, CSE486In reality, a plane 2x as big 2x asfar away yields the same image.Thus t z is arbitrary and is often fitto known coordinates by scaling.Thus H has only 8, not 9parameters.Robert Collins, CSE486ICRA 2010 ROS Tutorial55

Homography• We often use the chessboard detector tofind 4 non-colinear points– (X,Y * 4 = 8 constraints)– To solve for the 8 homography parmeters.• Code: Once again, OpenCV makes thiseasy– findHomography(…) or:– getPerspectiveTransform(…)ICRA 2010 ROS Tutorial56

Homography Uses• If you have a known planar object on theground plane,– you can use it to map any other ground pt inthe image to its (X,Y,Z) point on the groundWe used this in theDARPA GrandChallenge to mapthe image roadsegmentation to avieweyebird’sobsticle map:getPerspectiveTransform(objPts,imgPts,H);invert(H,H_invt);Gary Bradski and Adrian Kaehler: Learning OpenCV//This learns ground_pts->image_pts//So we need to invert this to get img_pts->ground_ptsICRA 2010 ROS TutorialParkinga robot57Gary Bradski and Adrian Kaehler: Learning OpenCV

Summary• Homography– Uses• Given 4 image points corresponding to 4non-colinear points on an external planarobject• We can compute a mapping from imagepoints to points on the external plane• Robots can use this to know where thingsare in an external ground plane.ICRA 2010 ROS Tutorial58

Outline• Overview• Quick Tour• Pinhole Camera• Homography• Camera Calibration• Gradient Features• Demo– Object Recogntion Using Gradient Features– Node: Finding a Chessboard and its PoseICRA 2010 ROS Tutorial59

Calibration Intuition• pinholeainpositions”‏ideal‏“theirfromdistortedarePoints (pure perspective) camera.• Take images of an object with known points.• Find out where those points are in the camera image.• Compute a tranformation that maps the points to the correct“ideal”positionstheyshouldhave.ICRA 2010 ROS Tutorial60

Relate Known Object to CameraPositionsGary Bradski and Adrian Kaehler: Learning OpenCVICRA 2010 ROS Tutorial61

Calibration Object• A chessboard makes a good calibration pattern.• We find its corners using the Harris corner detector.• Calculate its homography and plot it.ICRA 2010 ROS Tutorial

Camera Intrinsic and Extrinsic• Intrinsics:– Focal Length f– Pixel size sParameters– Image center o x , o y• Extrinsics:– Location and orientation of k-th calib. pattern: , , ,TICRA 2010 ROS Tutorial

Number of Images Needed• K images of chessboads with N corners• Have 4 intrinsic parameters (f,s,c(x,y)) and 6extrinsic (x,y,z, pan, tilt, yaw).• Solving then requires 2NK>6K+4– (N-3)K>=2– If N=5, K= 1– NO! Planar chessboard has only N=4 amount ofinformation, so K >= 2.– In practice, we use many images to reduce noise.ICRA 2010 ROS Tutorial64

Calibration Object Image CollectionGary Bradski and Adrian Kaehler: Learning OpenCV

yzWWWYWWWyxzWWWXWWWximimoTZYXTZYXsfoTZYXTZYXsfyxcossin0sincos0001cos0sin010sin0cos100cossin0sincos0001cos0sin010sin0cos0cossincossin0sincos0001cos0sin010sin0cos100cossin0sincos0001cos0sin010sin0cos0sincosSolve the Full Perspective CameraModel

Code• Explained‏in‏detail‏in‏“Learning‏OpenCV”• Calibration solved using OpenCV function:– CalibrateCamera()– OpenCV site:– http://opencv.willowgarage.comICRA 2010 ROS Tutorial67

Perspective n-Point Problem• Many useful things follow from our calibrationsolution.– Homography:• Relating one plane to another (next section)– PnP problem:• If we can find known points on a known 3D object,• torelativeorientationit’s‏)poseit’sfind(‏didjustwe‏)canwe the camera coordinate system).• OpenCV CODE: solvePnP(…)Gary Bradski, CS223A, 68ICRA 2010 ROS Tutorial

Training Set Assist• Find chessboard• Relate another part of the scene to itICRA 2010 ROS Tutorial69

Summary• Camera Calibration– How to compute– PnP problem solution• Use a known object• camerathetoprojection”‏raw‏“itsFind plane• Compute a mapping that moves the reallocation of features to the ideal locations.• Same techniques useful for finding thepose of an object given known points (PnPProblem)ICRA 2010 ROS Tutorial70

Outline• Overview• Quick Tour• Pinhole Camera• Homography• Camera Calibration• Gradient Features• Demo– Object Recogntion Using Gradient Features– Node: Finding a Chessboard and its PoseICRA 2010 ROS Tutorial71

Why is Vision Hard?The difference between seeing and perception.What to do? Search for featuresMaybe we should try edges ….Gary Bradski, 200572

But,‏What’s‏an‏Edge?• Depth discontinuity• Surface orientationdiscontinuity• Reflectancediscontinuity (i.e.,change in surfacematerial properties)• Illuminationdiscontinuity (e.g.,shadow)Slide credit: Christopher Rasmussen73

Gradient Features: SobelTypically gradients are found by convolving with a deriviatve operatorf[ m,n] I g I [ m k , n l ] g[k , l ] g[k , l ] 1k , lk , lThe Sobel derivative (approximates a Gaussian in onedimension, a difference operator in the other) is a popular choice:-1 -2 -1S 1 = 0 0 0 S 2 =1 2 1-1 0 1-2 0 2-1 0 1Edge Magnitude =2 2S 1 + S 1Edge Direction =tan -1 S 1S 2OpenCV Code: sobel(…), but we prefer the scharr(…) operator since it is just asfast but more accurate at angles near +/- 45 degrees.74

Binary Gradient GridsImages from: David G. Lowe, Object recognition from local scale-invariant features,International Conference on Computer Vision, Corfu, Greece (September 1999), pp. 1150-1157Many‏features‏‏(notably‏Lowe’s‏SIFT)‏start‏from‏collections‏of‏gradientsAnd process the gradients into summary vectors:I will demo direct use ofspatial layouts ofbinarized gradient grids:ICRA 2010 ROS Tutorial75

Summary• Vision is hard because of the 2D-3D ambiguity– Perception in contrast to seeing• Therefor, use statistical regularity– Expreess the image in terms of features• We focus on binary gradient gridsICRA 2010 ROS Tutorial76

Outline• Overview• Quick Tour• Pinhole Camera• Homography• Camera Calibration• Gradient Features• Demo– Object Recogntion Using Gradient Features– Node: Finding a Chessboard and its PoseICRA 2010 ROS Tutorial77

Binary Gradient GridWe organize gradients in a way that allows extremely rapid searchThis allows us to directly scale recognition with compute cyclesRaw Gradient Grid RecognizedGary Bradski, 2010DemoICRA 2010 ROS Tutorial

Chessboard Service Node# Get to the directoryroscd icra_ros_tutorials# Start roscore and the checkerboard-detector-to-pose serviceroslaunch launch/detectors.launch# Play some data (or get from the robot)while true; do rosbag play checkerboard_box.bag; sleep 0.1; done# What nodes are up?rosnode list# What services are offeredrosnode info /narrow_cb_detector#orrosservice listDemo# What arguments do I need to enter for that service?rosservice args /narrow_get_checkerboard_pose# Call the servicerosservice call narrow_get_checkerboard_pose 3 4 .108 .108Q1: How do you call the wide stereo?Q2: Are the results different? If so: Why?Q3: If you change around the 3 and 4, what happens? Show why.CS324 79ICRA 2010 ROS Tutorial

Summary• We made use of what we learned:– Recognition using binary gradient grids– A node that finds a chessboard and returns its poseICRA 2010 ROS Tutorial80

Questions?Photo: Gary Bradski81

Hooray! Your file is uploaded and ready to be published.

Saved successfully!

Ooh no, something went wrong!