Object Detection from Aerial Image - Institute for Computer Graphics ...

icg.tugraz.at

Object Detection from Aerial Image - Institute for Computer Graphics ...

Object Detection from Aerial Image

Thuy Thi Nguyen

PhD Thesis Defense

Graz University of Technology

Institute for Computer Graphics and Vision

October 16 th 2009


Introduction

♦ Context that brought to the work of the thesis

I started my work at ICG, TU Graz Jan. 2006, the time of

some impressive events:

♦ Emergence of technology and application:

– UltraCamD: multiple large scale digital aerial images, highly redundancy.

– Issues of Aerial Photo Interpretation (API): Object detection and extraction

for various applications, e.g. cartography, land use classification, 3D models

of urban space (Microsoft virtual earth, Google earth).

The work of VRVis on 3D map generation from digital aerial images.

♦ Methodology: machine learning methods

– Online boosting and vision: novel, robust online learning algorithm (Grabner et

al, 2006)

– Models that can improve the performance of traditional state-of-the-art

methods, e.g. Conditional random field.

Object Detection from Aerial Image 2 Graz, 16 th Oct. 2009

Thuy Nguyen


Object Detection from Aerial Image

Complex urban scene of Graz city

Object Detection from Aerial Image 3 Graz, 16 th Oct. 2009

Thuy Nguyen


Objects in Aerial Image

Objects in aerial images: 2 categories

(1) Rather well-defined shape, stay in isolation: Cars, trucks -> detect object in a

bounding box, sliding window approach

(2) Complex, spread over the scene, may stay connected: Buildings, road net –>

pixelwise classification, probabilistic model

Object Detection from Aerial Image 4 Graz, 16 th Oct. 2009

Thuy Nguyen


Motivation

♦ Advance topics in real world applications: 3D city modeling, change

detection for updating maps, etc.

– Why the detection of cars is of interest?

• Noisy object to remove

• Context: car appearance – road or parking lots

– Why the detection of buildings is of interest?

• The most important object in aerial image: detection of building for applications such

as 3D modeling or updating cadastral maps

♦ Study novel machine learning methods for computer vision problems.

• Good domain to explore the problem of object detection, segmentation, and scene

understanding in computer vision,

• not only for objects in aerial image but also in terrestrial images.

Object Detection from Aerial Image 5 Graz, 16 th Oct. 2009

Thuy Nguyen


Approaches

1- Online Boosting

Learning for car

detection,

boundingbox

2- Hierarchical CRF

model for detection

and segmentation of

buildings, pixel level

No universal detector for all kind of object classes!

Object Detection from Aerial Image 6 Graz, 16 th Oct. 2009

Thuy Nguyen


Outline

♦ Introduction: context and motivation

♦ Online boosting based for car detection

♦ Hierarchical conditional random field for building detection

♦ Conclusion

♦ Future work

Object Detection from Aerial Image 7 Graz, 16 th Oct. 2009

Thuy Nguyen


Outline

♦ Introduction: context and motivation

♦ Online boosting based for car detection

• Introduction

• The online boosting based learning framework

• Experiment and evaluation

♦ Hierarchical pseudo-conditional random field (HpCRF) for

building detection

• Introduction

• The HpCRF model and learning algorithm

• Experiment and evaluation

♦ Conclusion

♦ Future work

Object Detection from Aerial Image 8 Graz, 16 th Oct. 2009

Thuy Nguyen


Approaches – Part 1

1- Online Boosting

Learning for car

detection,

boundingbox

2- Hierarchical CRF model

for detection and

segmentation of

buildings, pixelwise

Object Detection from Aerial Image 9 Graz, 16 th Oct. 2009

Thuy Nguyen


How many cars are there in the image?

Object Detection from Aerial Image 10 Graz, 16 th Oct. 2009

Thuy Nguyen


Motivation

To use novel on-line boosting method to build a robust

framework for efficient car detection from large scale

aerial images

‣ The detection is formulated as a classification

problem: Learn a classifier to discriminate object

from background.

‣ Active training, no labeled data is needed in

advance

Object Detection from Aerial Image 11 Graz, 16 th Oct. 2009

Thuy Nguyen


Boosting method

Why Boosting?

♦ A simple algorithm for learning robust classifiers

– Freund & Shapire, 1995; Friedman, Hastie, Tibshhirani, 1998

♦ Provides efficient algorithm sparse visual feature selection

– Tieu & Viola, 2000, Viola & Jones, 2003

On-line Boosting?

♦ Efficient generate training set, while incrementally train the

classifier, adaptive learning

− Grabner and Bischof, 2006

Object Detection from Aerial Image 12 Graz, 16 th Oct. 2009

Thuy Nguyen


Boosting Method

Boosting: Combining many „weak“ classifiers to produce a „strong“ one.

Given:

- set of training samples

- weight distribution

Train a weak classifier

- calculate weight

Combine

Object Detection from Aerial Image 13 Graz, 16 th Oct. 2009

Thuy Nguyen


Boosting for Feature Selection

♦ A weak classifier corresponds to a feature

h

weak

i

( x)

~ f ( x)

i

⇒ selects representative features while learning an efficient

classifiers (Tieu & Viola, 2000, Viola & Jones, 2003)

♦ Features

– Haar-like wavelets

– Orientation histograms

– Locally binary patterns

(LBP)

♦ Fast computation using efficient

data structures

– integral images

– integral histograms

Object Detection from Aerial Image 14 Graz, 16 th Oct. 2009

Thuy Nguyen


Online Boosting for Feature Selection

♦ Introducing “Selector”

– selects one feature from its local

feature pool

Boosting is performed on the

Selectors and not directly on the

weak classifiers.

(Grabner & Bischof, 2006)

Object Detection from Aerial Image 15 Graz, 16 th Oct. 2009

Thuy Nguyen


On-line Boosting for Car detection

♦ Active Learning

♦ On-line training process:

Initiate parameters for the

classifier

while non-stop-criteria do

– Evaluate the current

classifier and display

results

– Manually label one missed

classified sample (either

positive or negative)

– Update parameters for the

classifier

end while

Detections

Labels

Image

Discriminative model

AdaBoost

Classifier

Object Detection from Aerial Image 16 Graz, 16 th Oct. 2009

Thuy Nguyen


On-line Boosting for Car detection

♦ Active Learning: Initiate parameters for the classifier

Image

Discriminative model

AdaBoost

Object Detection from Aerial Image 17 Graz, 16 th Oct. 2009

Thuy Nguyen


Training process


Active Learning: Evaluate the current classifier and display results

Detections

Image

Discriminative model

AdaBoost

Object Detection from Aerial Image 18 Graz, 16 th Oct. 2009

Thuy Nguyen


Training process

♦ Active Learning

Detections

Image

Discriminative model

AdaBoost

Labels

Update parameters for the

classifier

Object Detection from Aerial Image 19 Graz, 16 th Oct. 2009

Thuy Nguyen


Training process


Active Learning: Evaluate the classifier and display results

Detections

Image

Discriminative model

AdaBoost

Labels

Object Detection from Aerial Image 20 Graz, 16 th Oct. 2009

Thuy Nguyen


Training process

♦ Active Learning

Detections

Image

Discriminative model

AdaBoost

Classifier

Object Detection from Aerial Image 21 Graz, 16 th Oct. 2009

Thuy Nguyen


Experiments

Object Detection from Aerial Image 22 Graz, 16 th Oct. 2009

Thuy Nguyen


Data Sets

♦ 2 Data sets

Aerial images taken by

UltraCamD from Graz

and Philadelphia cities

– Graz: 155 images, 8cm

ground sampling distance,

summer 2005

– Philadelphia: 158 images,

10cm ground sampling

distance, winter 2005

Object Detection from Aerial Image 23 Graz, 16 th Oct. 2009

Thuy Nguyen


Experiments

♦ Data set: 2

– Graz: 155 images,

– Philadelphia: 158 images

Image size: 11500x7500 pixels each

♦ 6 typical patches of 4000x4000 pixels are used for training and

test

– Test sets: Graz: 958 cars, Phil.: 1495 cars

♦ Parameters setting:

– 500 weak classifiers

– 250 selectors

– Car patch size: 35x70 pixel, the same for both data sets

Object Detection from Aerial Image 24 Graz, 16 th Oct. 2009

Thuy Nguyen


Training process

♦ Labeled 1420 samples: 410 positive, 1010 negative

- Few training samples

- # positive samples less than # negative samples

Object Detection from Aerial Image 25 Graz, 16 th Oct. 2009

Thuy Nguyen


Improving Performance

#TP

RR =

#TP + # FN

# TP

PR =

#TP + # FP

2. RR. PR

F _ m =

RR + PR

RR: Recall rate

PR: Precision rate

TP: True positive

FN: False negative

FP: False positive

F_m: F-measure

Object Detection from Aerial Image 26 Graz, 16 th Oct. 2009

Thuy Nguyen


Post processing

♦ Mean shift clustering

– To obtain one detection per location over multiple detections

of classifier output

Raw classifier output

After postprocessing

Object Detection from Aerial Image 27 Graz, 16 th Oct. 2009

Thuy Nguyen


Experimental Results

Object Detection from Aerial Image 28 Graz, 16 th Oct. 2009

Thuy Nguyen


Experimental Results


Performance evaluation: recall-precision curves (RPC)

Graz data set

Philadelphia data set

Object Detection from Aerial Image 29 Graz, 16 th Oct. 2009

Thuy Nguyen


♦ Road mask from street layer

Contextual information

Object Detection from Aerial Image 30 Graz, 16 th Oct. 2009

Thuy Nguyen


Experimental Results

♦ RPCs with contextual information (road mask of street layer)

Graz data set

Philadelphia data set

Object Detection from Aerial Image 31 Graz, 16 th Oct. 2009

Thuy Nguyen


Conclusion

♦ We have developed an efficient framework for car

detection from aerial images

– Adaboost, novel on-line boosting framework, interactive training

– Efficient image representation (Intergral), representative features

(Haar-like, orientation histogram, local binary pattern)

– No prior knowledge of image, no labeled data in advance

– Classifier is trained and improved during the online learning

♦ Performance: good, even superior

(Lack of public data set -> unfair comparision)

♦ Applications

– Estimation of transportation flow, road/street layer verification,

texture recovery for 3D city modeling, etc.

Object Detection from Aerial Image 32 Graz, 16 th Oct. 2009

Thuy Nguyen


An application: texture recovery

Object Detection from Aerial Image 33 Graz, 16 th Oct. 2009

Thuy Nguyen


Question?

Object Detection from Aerial Image 34 Graz, 16 th Oct. 2009

Thuy Nguyen


Approaches – Part 2

1- Online Boosting

Learning for car

detection, boundingbox

2- Hierarchical CRF

model for detection

and segmentation of

buildings, pixelwise

Object Detection from Aerial Image 35 Graz, 16 th Oct. 2009

Thuy Nguyen


HpCRF and the Buildings detection problem

Object Detection from Aerial Image 36 Graz, 16 th Oct. 2009

Thuy Nguyen


Problem

Detection/segmentation of buildings at pixel level

– Classify every image pixel into building or non-building

classes

♦ Buildings are complex object

– many architectural details,

– spread over the scene

– Rooftop reflectance properties, low contrast to ground,...

Object Detection from Aerial Image 37 Graz, 16 th Oct. 2009

Thuy Nguyen


Problem statement

Detection/segmentation of buildings at pixel level

– Classify every image pixel into building or non-building

classes

Original Image Output of a standard classifier Improvement

♦ Aim: use idea of conditional random field model to

improve the performance of the local classifier

Object Detection from Aerial Image 38 Graz, 16 th Oct. 2009

Thuy Nguyen


Background

♦ Classification:

– Logistic, SVM,

Boosting

Standard classifiers,

iid assumption

P( Y | X )

X: image data (pixel, site)

♦ MRF, CRF: improve the performance of iid classifiers by incorporating

contextual information

MRF

CRF

Local potential Interaction potential

the whole

Object Detection from Aerial Image 39 Graz, 16 th Oct. 2009

Thuy Nguyen


Motivation - Modeling

♦ Different visual cues should play different roles in discriminating object

classes (1)

♦ Mixture of features: f = [f color

, f texture

, f contour

...];

may cause over-fitting due to redundancy, correlation in the input data

♦ One feature type may inhibit the performance of the other: color and height

data of aerial image

Decompose input feature into different feature types

Given training set D = (X, Y)

X = {X t , t∈T},

D t = (X t , Y), T - set of feature types

(1) X. Ma, Learning Coupled Conditional Random Field for Image Decomposition: Theory and Application in

Object Categorization, PhD. Thesis, MIT, 2008

Object Detection from Aerial Image 40 Graz, 16 th Oct. 2009

Thuy Nguyen


Modeling

♦ Each kind of feature has its own context to exploit

Learn contextual models for each feature type separately: multiple CRFs at

1 st layer

Interactions between different feature channels and contexts: expand each

feature vector type with predictions from related instances (2, 3)

Learn multiple probabilistic classifiers at 2 nd layer

(2) Z. Kou and W. Cohen (2007): Stacked Graphical Models for Efficient Inference in Markov Random Fields,

in SDM 07

(3) Z. Tu. Auto-context and its application to high-level vision tasks. CVPR, 2008

Object Detection from Aerial Image 41 Graz, 16 th Oct. 2009

Thuy Nguyen


Modeling

♦ Combine potentials at different levels in a hierarchical model;

Fusion of classification potentials to infer the object class

♦ Training and inference of CRF is expensive, not applicable for

huge data of aerial images -> use pseudo CRF (4)

(4) C.-H. Lee, S. Wang, A. Murtha, M. R. Brown, and R. Greiner. Segmenting brain tumors using pseudoconditional

random fields. In MICCAI, 2008.

Object Detection from Aerial Image 42 Graz, 16 th Oct. 2009

Thuy Nguyen


HpCRF

♦ The hierarchical pseudo Conditional random field model

Object Detection from Aerial Image 43 Graz, 16 th Oct. 2009

Thuy Nguyen


HpCRF - Learning

♦ Learn local classifiers: any probabilistic classifier (SVM)

♦ For individual CRF at 1 st layer: compute classification confidences using

Pseudo-CRF

♦ Build new training set

♦ Learn individual CRFs at 2 nd layer: compute classification confidences

using Pseudo-CRF

♦ Top: Fusion of classification potentials

♦ Infer the object class

Object Detection from Aerial Image 44 Graz, 16 th Oct. 2009

Thuy Nguyen


Experiment

Object Detection from Aerial Image 45 Graz, 16 th Oct. 2009

Thuy Nguyen


Data set

RGB image

Height field data

Hand labeling

ground truth

Object Detection from Aerial Image 46 Graz, 16 th Oct. 2009

Thuy Nguyen


Data set

♦ Data set: 12 image patches of size 2000x2000

♦ Feature types:

Object Detection from Aerial Image 47 Graz, 16 th Oct. 2009

Thuy Nguyen


Performance

♦ Base classifier: SVM

Object Detection from Aerial Image 48 Graz, 16 th Oct. 2009

Thuy Nguyen


Performance

♦ Example of an image patch and the labeled building mask

Object Detection from Aerial Image 49 Graz, 16 th Oct. 2009

Thuy Nguyen


Performance

Color feature Texture feature Original image

Height field data Used all features

Ground truth

Object Detection from Aerial Image 50 Graz, 16 th Oct. 2009

Thuy Nguyen


Performance comparison

SVM (Height) SVM (all feature)

Original image

CRF HpCRF

Ground truth

Object Detection from Aerial Image 51 Graz, 16 th Oct. 2009

Thuy Nguyen


Conclusion and Outlook

♦ Conclusion

– New hierarchical CRFs model for learning complex objects, i.e.

buildings from aerial image

– Use different feature types and contextual interactions to train multiple

CRFs, incorporate into a unified HpCRF model.

– Learning and inference are efficient, general, can be easily applied to

some other learning tasks.

♦ Further work:

– Different contextual model for each feature type

– Multiple kernel learning to weight the contributions of each kind of

feature

– Apply to other object detection/recognition problems

– …

Object Detection from Aerial Image 52 Graz, 16 th Oct. 2009

Thuy Nguyen


Publications (chronologic)

The thesis is mainly based on the highlighted papers.

(i)- Thuy T. Nguyen, Stefan Kluckner and Horst Bischof, “A Stacked Model for Appearance and 3D Height with

Application to Building Detection from Aerial Image”, will be submitted to The Int. Conference on Computer Vision

Theory and Applications (VISAPP 2010). (new)

(ii)- “A Hierarchical Random Field Model for Building Detection from Aerial Image”, Accepted as Oral presentation at the

Workshop for Woman in Machine learning (WiML’09), Vancouver, Canada Dec. 2009. (new)

1- Thuy T. Nguyen and Horst Bischof, “ HpCRF: A Hierarchical Pseudo-Conditional Random Field Model for Buildings

Detection from Aerial Image”, Submitted to The IEEE Intl Conference on Computer Vision (ICCV’09). Japan, 2009.

2- B. D. Nguyen, Thuy T. Nguyen, ”Automatic Database Creation and Object Model Learning”, Lecture Notes in

Computer Science, Springer Verlag, Vol. 5465/2009, p. 27-39. May 2009.

3- Thuy T. Nguyen, B. D. Nguyen and Horst Bischof, ”An active boosting-based framework for real-time hand detection”,

in Proc. of The 8th IEEE Intl Conference on ”Automatic Face and Gesture Recognition (FG08)”, Amsterdam, Sep.

2008.

4- Thuy T. Nguyen, B. D. Nguyen and Horst Bischof, ”Efficient boosting-based active learning for specific object detection

problems”, in Proc. of The 5th Intl Conference on ”Computer Vision, Image and Signal Processing (CVISP 2008)”,

Praha, Jul. 2008.

5- B. D. Nguyen, Thuy T. Nguyen and Horst Bischof, ”On-Line Boosting Learning for Hand Tracking and Recognition”, in

Proc. of The 2008 Intl Conference on Image Processing, Computer Vision, and Pattern Recognition (IPCV’08), Las

Vegas, USA, Jul. 2008.

6- Helmut Grabner, Thuy T. Nguyen, Barbara Gruber and Horst Bischof, ”Boosting based car detection from aerial

images”, ISPRS, Intl. Journal of Photogrammetry and Remote Sensing, Vol. 63/3, p. 382-396. DOI information:

10.1016/j.isprsjprs.2007.10.005.

7- Thuy T. Nguyen, Helmut Grabner, Barbara Gruber and Horst Bischof, ”On-line boosting for car detection from aerial

images”, in Proc. of The 5th IEEE Intl Conference on ”Research, Innovation and Vision for the Future (RIVF’07)”,

Hanoi, Mar. 2007 (Best paper Award).

Object Detection from Aerial Image 53 Graz, 16 th Oct. 2009

Thuy Nguyen


Acknowledgments

♦ My advisor: Prof. Horst Bischof

♦ Thesis commitee:

Prof. Eugen Brenner, Prof. Vaclav Hlavac, Prof. Horst Bischof

♦ Co-authors

♦ ICG members, vision group,

♦ CMP at CVUT Prague.

♦ Data: Microsoft Photogrammetry (VRVis)

♦ Funding: OeAD, OeFG, TU Graz.

Object Detection from Aerial Image 54 Graz, 16 th Oct. 2009

Thuy Nguyen


Thank you.

Questions?

thuy@icg.tugraz.at

Object Detection from Aerial Image 55 Graz, 16 th Oct. 2009

Thuy Nguyen

More magazines by this user
Similar magazines