Slides - ETHZ - Computer Vision Lab

Learning for Tracking andLessons Learned from itHelmut Grabner

Tracking CuesObject AppearanceBackgroundObject/BackgrounddiscriminationObject BoundaryMotion2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab4

[Grabner et al. VideoProc.CVPR 2006]2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab5

Tracking Requirements (model free tracking) Adaptive RobustnessAnd of course,REAL TIME! Generality2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab6

PART IOn-line Boosting basedTrackingCVPR’06, BMVC’06

Boosting and Vision Boosting[Freund and Schapire, JCSC, 1997] On-line boosting[Oza and Russel, AIS, 2001] Boosting for FeatureSelection[Tieu and Viola. CVPR 2000], [Violaand Jones, CVPR 2001] On-line Boosting forFeature Selection[Grabner and Bischof, CVPR 2006]2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab8

Off-line learning2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab9

On-line learningLabeled InformationLearning AlgorithmTeacher2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab10

Sir, could you please tellme what boosting is?Boosting needs just someguy who is a little bit betterthan guessing.2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab11

Off-line boostingStrong classifierWeak classifierReweighting thetraining examples2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab12

Boosting for Feature Selection Combination of SimpleImage Features fordistinguishing two classes Features = weak classifier Boosting to select a subset(strong classifier)2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab13

On-line Boosting2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab14

On-line BoostingConverges to theoff-line result!2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab15

On-Boosting for Feature Selection Combination of SimpleImage Features fordistinguishing two classes Features = weak classifier Boosting to select a subset(strong classifier)2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab16

On-Boosting for Feature Selection Combination of SimpleImage Features fordistinguishing two classesGeneral approachfor on-line featureselection. Features = weak classifier Boosting to select a subset(strong classifier)2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab17

Tracking as Classificationobjectvs.background2009/08/18, 2008/11/06 Scovis Southampton Meeting H. Department Grabner, Tracking of Information Learning Technology and Lessons and Electrical Learned Engineering from it - ETH-Zurich, – Computer Computer Vision Laboratory Vision 18

Tracking as Classificationobjectvs.background2009/08/18, 2008/11/06 Scovis Southampton Meeting H. Department Grabner, Tracking of Information Learning Technology and Lessons and Electrical Learned Engineering from it - ETH-Zurich, – Computer Computer Vision Laboratory Vision 19

from time t to t+1evaluate classifier on sub-patchesactual object positionsearch Regionanalyze map and set newobject positionupdate classifier (tracker)create confidence map--+- -2009/08/18, 2008/11/06 Scovis Southampton Meeting H. Department Grabner, Tracking of Information Learning Technology and Lessons and Electrical Learned Engineering from it - ETH-Zurich, – Computer Computer Vision Laboratory Vision 20

ObjectDetectorFixed Training setGeneral object detectorOff-line Boosting forFeature SelectionObject TrackerOn-line updateObject vs. BackgroundOn-Line Boosting forFeature Selection2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab21

LESSON LEARNED 1Tracking is a simple task!(When formulating it properly)

2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab23

“Tracking the Invisible”2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab25

Tracking Solved ☺Does it fail?If yes, when?2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab26

When does it fail…2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab27

When does it fail…Often, all toooften!2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab28

When does it fail…WHY?2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab29

from time t to t+1evaluate classifier on sub-patchesactual object positionsearch Regionupdate classifier (tracker)-+-analyze map andset new objectpositionSelf-learningcreate confidence map- -2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab30

Drifting due to self-learning policyTracked PatchesConfidence2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab31

LESSON LEARNED 2Self-learning → drifting!

PART IISemi-Supervised On-lineBoosting for TrackingECCV’08

Review: Supervised Tracking2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab34

Problems of…-Label Jitter+++Label Noise2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab35

Semi-Supervised TrackingUn-labeleddataLabeled data2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab36

Supervised learning+ -+ -Maximum margin2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab37

Can Unlabeled Data Help??? ?? ? ???? ? ?? ?? +? ?-??? ? ???? ?? ? +? ? - ????? ? ? ? ?decision? ?? ?boundary in lowdensity region2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab38

Semi-Supervised On-line BoostingPrior+ + +???-????2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab39

Semi-Supervised On-line BoostingPrior“stable”fixdynamicNobody is perfect!But, be a honest Teacher!H off (Prior) can be wrong with lowconfidence.2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab40

Tracking LoopPrioractual object positionevaluate classifier onsub-patchessearch Regionupdate classifier (tracker)create confidence map????2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab41

PriorLabeled dataUn-labeleddata2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab42

LESSON LEARNED 3On-line Semi-supervisedlearninig → limiteddrifting.

Occlusions2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab44

Object disapearance2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab45

Long term tracking (1h)Click here to start2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab46

Tracking Solved ☺Does it fail?If yes, when?2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab47

2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab48

Prior is toogeneric (e.g, dirftto similar objects)2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab49

2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab50

Prior restricts toomuch (e.g., partialocclusions)2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab51

LESSON LEARNED 4Prior is essential in semi-supervised learning.(c.f., Stability PlasticyDilemma)

PART IIIBeyond Semi-SupervisedTrackingICCV’09 WS on On-line Learning forComputer Vision

Review: DetectionNon adaptive at all!2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab54

Review: Supervised TrackingToo adaptive2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab55

Review: Semi-Supervised TrackingLimit drifting but toorestrictive / general2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab56

NEW PRINCIPLE: Active Sampling via Tracking2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab57

NEW PRINCIPLE: Active Sampling via Tracking2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab58

NEW APPROACH: Adaptive PriorAdditionalinformationfor prior updateDetectorDetector2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab59

Specializing (Simplifying)Updates No Few trustfulLearning taskApplicableObject class vs.everything elseAny time,everywhereParticular objectsvs. other objectsand backgroundCurrent sceneMany semisupervisedCurrent Objectappearance vs.local surroundingLocalneighborhood2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab60

Multiple classifier system#include “vision.h” use addition information, e.g., multipleobjects, background imageInformation aggregation2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab61

2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab62

2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab63

Performance evaluationpositionTruepositivesTooAdaptiveNotvisibleFalsepositivesFalsenegativesLessAdaptiveGround Truthtime2009/03/23 2009/08/18, Oxford Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab64

Implicit Occlusion handling2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab65

Really Long Term Tracking (24h)Click here to start2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab66

……2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab67

LESSON LEARNED 5Vision is more then pureMachine Learning!(keep problems simple)

Extension 1: re-IdentificationIdentifier 1Identifier 22009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab69

2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab70

Extension 2: Information AggregationWhole ImageScene / Current object /local surroundingSpecializationObject instance /current sceneSpecific imagelocation2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab71

Pedestrian Detection (PETS)Generic Detector & ContextProposed Approach2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab72

LESSON LEARNED 6Vision ≠ Detection +Tracking + Recognition(benefitting from a lot of– unlabeled – data)2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab73

ConclusionTracking issimpleSelf-learning →driftingSemi-SupervisedlearningPrior Vision > MachineLearningVision ≠ Det. +Trac. + Rec.2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab74

AcknowledgmentsLuc van GoolSeverin StalderEU-project SCOVIS under grantagreement no 216465.Institute for Computer Graphics and VisionGraz, University of Technology, AustriaCentere of Machine PreceptionCzech Technical UniversityHorst BischofMichael GrabnerChristian LeistnerJiri MatasJan Sochamn2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab75

Code/Demos & Tracker Evaluation2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab76

Additional Slides…2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab77

DetectorNo UpdatesValid samplesConfidence map2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab78

Detectorpos.updatesValid samplesConfidence mapNeg.updates2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab79

DetectorValid samplesConfidence mapUnlabeled updates(foreground & local background)2009/08/18, Southampton H. Grabner, Tracking for Learning and Lessons Learned from it - ETH-Zurich, Computer Vision Lab80