slides - Institute for Computer Graphics and Vision

icg.tugraz.at

slides - Institute for Computer Graphics and Vision

Institute for Computer Graphics and Vision

Group 6 Reichmann & Ruprecht


Institute for Computer Graphics and Vision

Group 6 Reichmann & Ruprecht


Institute for Computer Graphics and Vision

Group 6 Reichmann & Ruprecht


Institute for Computer Graphics and Vision

Presentation schedule

Presentation 16.1 Presentation 23.1

Group 1

Dzombic & Mandl

Group 2

Kühnel & Gärner

Group 3

Enzinger & Plaschka

Group 4

Meisel & Sternig

Group 5

Gugl & Kikelj

Group 6

Reichmann & Ruprecht

Group 7

Dokter & Küberl

Group 8

Knöbelreiter & Strauss


Institute for Computer Graphics and Vision

Mobile & Handheld Augmented Reality


Institute for Computer Graphics and Vision

Trend to Mobile/Ubiquitous Computing

Computer Form Factor User Relationship Users/Computer

Building Submit >100

Room Share >10

Desk Sit at 1

Box 1

Laptop ... carry around 1/2

PDA/mobile Hold 1 - 1/3

Clothing Wear 1/10


Institute for Computer Graphics and Vision

Implications of Wearability

(after S. Mann, B. Rhodes, T. Starner)

• Mobility

– usable/used indoors and

outdoors

• Intimacy

– sense the wearer’s body,

communicate privately

• Context sensitivity

– take into account changing

environment

• Constancy

– Permeation of UI into

wearer’s life

Wearable Computing Group @ MIT


Institute for Computer Graphics and Vision

Location-based Data


Institute for Computer Graphics and Vision

Location-based Computing

• WorldBoard (Spohrer ’99)

– The world as a repository

of information

– Place as index to

information

• Mobile Augmented Reality as User Interface for locationbased

information !


Institute for Computer Graphics and Vision

Ubiquitous Computing

• Mark Weiser, Xerox PARC (1952-1999)

• The third wave of computing:

Computing woven into fabric of our everyday live

Sales/year

time

• Augmented Reality as an interface for Ubiquitous Comp.?


Institute for Computer Graphics and Vision

Milgram-Weiser Continuum

Milgram

Weiser [Newman VR 2007]

11


Institute for Computer Graphics and Vision

Mobile AR Applications

• Navigational aids

• Tourism

• Entertainment

• Journalism

• Personal location-based

information DB

• Social Networking

• General UI for appliances

• Construction

• Maintenance

• Military training and warfighting


Institute for Computer Graphics and Vision

Challenges of Mobile AR

• Mobile computing

– Limited resources

– Size, weight, ruggedness

– Battery

• Mobile AR

– Tracking / Localization

– Models

• Outdoor environment

– Harsh conditions - light, weather

– no instrumentation possible


Institute for Computer Graphics and Vision

Early AR: Backpack Examples

1997

Columbia Touring Machine (2002) Rockwell vest (1999)

Tinmith (2002)


Institute for Computer Graphics and Vision

Mobile AR -

Backpacks

• Columbia University:

MARS ‘97

• Uni.SA: Tinmith

• Naval Research Lab:

BARS


Institute for Computer Graphics and Vision

Example: Studierstube Mobile AR System

Indoor + outdoor

DGPS

tracked

touchpad

inertial

sensor

camera

HMD

notebook

WLAN

GPRS modem

Wide Area Tracking

DGPS (outdoors)

ARToolKit (indoors)


Institute for Computer Graphics and Vision

Navigation & Browsing - Indoors

AR Library


Institute for Computer Graphics and Vision

Navigation & Browsing - Outdoors


Institute for Computer Graphics and Vision

Memory prod

AT&T Cambridge’s Sentient AR

Receivers

Sentient

Computing

Server

Receiver

chain

Maintenance

Inertial

tracker

VGA

Serial port

Everywhere GUI


Institute for Computer Graphics and Vision

Displays

• Head-mounted display

• Monocular, monoscopic, stereoscopic

• Optical/video see-thru, see-around

• Narrow field of view

• Problems in sunlight

MicroVision Nomad

Sony Glasstron

Stereo optical see-thru

MicroOptical EG 7


Institute for Computer Graphics and Vision

Hardware evolution - tablets

2003

2005

Backpack+HMD:

…5-8kg

… > $10K

Scale it down:

…Sony UMPC 1.1GHz

…1.5kg

…still >$5K


Institute for Computer Graphics and Vision

Tablet PC AR Setup

Camera

UMPC 1.1GHz

Kruijf et al.,’08

IMU


Institute for Computer Graphics and Vision

Example: (Smart) Vidente

• Geospatial Data +

Augmented Reality (AR)

• 3D real time visualization of

underground infrastructure

• High accuracy GPS


Institute for Computer Graphics and Vision

Example: Hydrosys

• Hydrology data visualization

• GPS, sensors


Institute for Computer Graphics and Vision

Ideal Mobile AR Device

• A pair of stylish sunglasses

• Hi-res stereo 3D graphics

• Wide field of view

• Built-in computer with wireless

network

• Highly accurate 6DOF tracking

• All for $99.90

• Doesn‘t exist (yet?)


Institute for Computer Graphics and Vision

Hardware evolution

2003

2005

2007

Backpack+HMD:

…5-8kg

… > $10K

Scale it down:

…Sony UMPC 1.1GHz

…1.5kg

…still >$5K

Scale it down more:

Smartphone…$500

…All-in-one

…0.1kg

…billions of units


Institute for Computer Graphics and Vision

Everything is coming together now

High performance

processors

Powerful graphics

Wireless networking

Multimedia

Compass direction

Camera

Orientation

GPS, Location


Institute for Computer Graphics and Vision

Mobile phone as platform

• Cheap

• Socially acceptable

• Well known

• Discrete

• Wide spread

• Intuitive to use


Institute for Computer Graphics and Vision

CPU/Memory Limitations of Mobiles

• Small and slow memory

– Not many GB

– Slow memory access, small caches

• Weak processing power

– still slower, Single, dual core (quad ?)

– (Typically no FPU (floating point ~40x slower than integer) - changing fast !)

• Code optimized for phones runs 5-10x slower on a highend

phone than on an average PC

• Not going to change quickly due to battery power


Institute for Computer Graphics and Vision

So what are the problems?

• Bad camera quality under low lighting

– Noise, motion blur

– Strongly varies with different phones

• Small memory

– Keeping large databases in memory is problematic

• Slow memory

– Processing large memory areas is prohibitive

– Low clock rate

• Typical CV building blocks (SVD) too demanding

• (No floating point unit

– Floating point code ~40x slower than integer code

– Fixed point problematic for many algorithms

– Requires good math library)


Institute for Computer Graphics and Vision

Platforms that are interesting for AR

(and their major strengths & weaknesses)

Windows

Mobile

Symbian

iPhone

Android

Easy to program

Lots of devices

(Largest installed basis)

Good devices

Very nice hardware

Hype factor

Good hardware,

Hype factor

Bad camera drivers, obsolete, or restricted

(7,8)

Outdated OS, hard to program, SDK

changes regularly, usually slow CPUs

only two different devices, Objective C as

main language

Changing quickly, currently most active

mixed Java/C/C++ programming

LiMo Full Linux support Few devices

Palm

WebOS

Nice hardware

Only very few devices so far

No native SDK so far

Blackberry Widely spread in the US Java only

Maemo or

whatever :)

Nice hardware, Linux, Open

Only one phone device (N900)

but we still use it !


Source: Millennial Media, 4/11

Institute for Computer Graphics and Vision

More market info...

Device OS Mix

Smartphone, Feature Phone &

Connected Device Impression Share

CHART A

Smartphone OS Mix

Ranked by Impressions

CHART B

16%

CONNECTED

DEVICES

16%

FEATURE

PHONES

68%

SMARTPHONES

28%

53%

1% 1% 1% Android

16%

iOS

RIM

Symbian

Windows

Other


Institute for Computer Graphics and Vision

Handheld Information Browsers

Wikitude

Geo-referenced Wikipedia information

Peak.ar

overlays names of mountains

Layar

Dedicated content layers


Put-A-Spell, ’09 Sony invizimals, ’09

Institute for Computer Graphics and Vision

Games

Invisible Train, ’04

MARQ, ’07


Institute for Computer Graphics and Vision

Visual Search

Foodtracer

Google Goggles, ’09


Institute for Computer Graphics and Vision

Navigation

Wikitude Drive

Navigationsinformation

acrossair

Nearest Tube


Institute for Computer Graphics and Vision

Indoor Navigation

• Sparse Localization with InfoPoints

• At InfoPoints: World-in-miniature plus orientation

augmentation

User walking:

Activity instructions

User standing at the info point:

3D world-in-miniature

37


Institute for Computer Graphics and Vision

Location-based Social Networking

Augmented ID

Junaio


Institute for Computer Graphics and Vision

AR Mashups Using Public Sources

• Geo-information relevant to AR

• Google Earth / MS Virtual Earth

• Massive amounts of data

• Added value through Mashups

• Free / paid by advertising


Institute for Computer Graphics and Vision

End-User Authoring for AR

• Desktop Authoring

– Most AR authoring to date on

desktop

– Efficient for complex content

preparation

– Efficient for large-scale overview

– Not efficient for detailed layout

– Not efficient for spontaneous

authoring

• In-situ authoring:

– Tracking requires model or online

modeling (point, area etc.)

– Annotation on phone: tags, audio?

Combination of tools on

phone and desktop


Institute for Computer Graphics and Vision

How is Content Made in the Web 2.0?

• Flickr, Wikipedia, Youtube & Co

• Social Networking

• End users provide content

• End users collaboratively

rate/tag content

• Classification by statistics rather than semantics

• Phenomenon of critical mass

• Mashups: End-user programming of Web 2.0 services


Institute for Computer Graphics and Vision

Why would you provide AR content?

• Give/take answers

• Friendship, prestige

• Relevance of information

• Filtering

• Speed of response

Oh,no,officer - it’s not graffiti. it’s an analog

real-time augmented reality application.


Institute for Computer Graphics and Vision

Augmented Reality with Panoramas

• Panoramas for

– Tracking

– Content creation and registration

– Modeling

– Localization


Institute for Computer Graphics and Vision

Panorama Creation on Mobile Phones

44


Institute for Computer Graphics and Vision

World Wide Signpost

• 2D map interface

– Navigation based on GPS

– Current position

– Position of annotations

• AR interface

– Browsing annotations

– Based on panorama

(orientation-)tracking

– Annotations stored as image patch


Institute for Computer Graphics and Vision

World Wide Signpost Video


Institute for Computer Graphics and Vision

Panorama Based Tracking

• Simultaneous Mapping and Tracking

• Orientation tracking (3DOF)

• Based on tracking keypoints in pano

• Panorama organized in tiles


Institute for Computer Graphics and Vision

Tracking and Mapping Principle

Video stream

New map pixels

Tracking

Mapping

Re-localization

Tracked pose

Map

Annotated features


Institute for Computer Graphics and Vision

World Wide Signpost - Matching

?

• Template matching

• Coarse test: Walsh transform

• Fine test: normalized cross correlation

• Executed on image tiles

49


Institute for Computer Graphics and Vision

Tracking, Mapping & Matching Principle

Video stream

Annotated

features

New map pixels

New map tiles

Tracking

Re-localization

Mapping

Matched

features

Matching

Tracked pose


Institute for Computer Graphics and Vision

Client-Server Communication

51


Institute for Computer Graphics and Vision

Matching under Temporal Variations

• Matching performance within same hour: 90%

• Matching performance after 1 day: 56% L

• Problem with shadows, illumination

• Need to avoid false positives

Shadows

Details

52


Institute for Computer Graphics and Vision

Matching with Sensor Fusion

NORTH

active search

?

• Features given with long/lat

info

• Convert to polar coordinates

• Active search window

constrained by compass/accel.

GPS (Long,Lat)


Institute for Computer Graphics and Vision

Matching with Global Transform

Library

Cafeteria

Institute

of

Physics

• Problem: Best match (green) does not always have highest score

• Wanted: inexpensive geometric constraints on matches

X


Institute for Computer Graphics and Vision

Global Transform Estimation

• Compute global rotation from candidate matches

• Use as RANSAC hypothesis

• Accept if max. error < 10 pixels

• Outliers: can compute and show virtual location

X


Institute for Computer Graphics and Vision

Reuse content: Situated Video Augmentations

Youtube...


Institute for Computer Graphics and Vision


Institute for Computer Graphics and Vision

Video object segmentation

• GrabCut segmentation

• Transfer into following frames for new segmentation

• Background for clean panorama

Track segment

Track segment


Institute for Computer Graphics and Vision

Segmented video + tracking

+ Tracking information

for each frame


Institute for Computer Graphics and Vision

Match offline and online panorama

RS

Source

RTS

RT

Target


Institute for Computer Graphics and Vision

Panoramas for Dense Outdoor construction

• Large scale reconstructions require thousands of

images...

• Simpler with a few wide-angle images


Institute for Computer Graphics and Vision

Wide-angle vs. Normal Images

• Few images to cover area

• Recorded by a single user

• Cheap to compute

• Little redundancy

• Many images

• High redundancy

• Lots of data and computation


Institute for Computer Graphics and Vision

Reconstruction

• Panorama or wide field-ofview

images


Institute for Computer Graphics and Vision

Reconstruction

• Match with multi-stage

process

1.Coarse NCC-based

alignment

2.SIFT in a constrained

window

3.RANSAC


Institute for Computer Graphics and Vision

Reconstruction

• Epipolar geometry

• 3D Triangulation


Institute for Computer Graphics and Vision

Reconstruction

• 3D Delaunay

Tetrahedralization

• Probabilistic space-carving


Institute for Computer Graphics and Vision

Reconstruction

• Texturing


Institute for Computer Graphics and Vision

More results


Institute for Computer Graphics and Vision

St. John’s College, Cambridge


Institute for Computer Graphics and Vision

Wide Area Localization from Panoramas

• Pano mapping & tracking: instant relative tracking

• Global absolute tracking: hard(er) on mobile phone

• One big problem: limited field of view of camera

– Often too few distinctive features in field of view

– Particularly difficult for large reconstructed models with lots of repetitive

features

• Approach

– Create partial panorama on the fly

– Global localization from panorama, not from single image

– Download 3D reconstruction data on demand from cloud server

– Continue orientation tracking while server works

64


Institute for Computer Graphics and Vision

System Overview

offline reconstruction

feature database

cloud storage

fast

camera

video

stream

(video rate)

tracking

new

map pixels

partial map

3DOF

relative pose

(dynamic)

slower

mapping

GPS

new

map tiles

slowest

prefetch

feature

data

localization

fusion

6DOF absolute pose (static)

6DOF absolute pose (dynamic)

mobile device

65


Institute for Computer Graphics and Vision

Example

66


Institute for Computer Graphics and Vision

Precision Results

67


Institute for Computer Graphics and Vision

Runtime Performance

68


Institute for Computer Graphics and Vision

Where next ?

• Many exiting ideas and

commercial applications

– Caveat: beware of inappropriate

sensing technology (barcodes,

compass only) or „concept videos“

• Look out for Visual Tracking

– Address current shortcomings

• AR as additional feature

– not standalone, but one of many

modes

– e.g. navigation, information browsers,

social networks...


Institute for Computer Graphics and Vision

Literature

[Arth2011] Arth, C., Klopschitz, M., Reitmayr, G., and Schmalstieg, D. (2011). Real-time self-localization from panoramic images on mobile

devices. In Proc. ISMAR 2011, pages 37–46, Basel, Switzerland.

[Feiner1997] Feiner, S., MacIntyre, B., H öllerer, T., and Webster, A. (1997). A touring machine: Prototyping 3D mobile augmented reality

systems for exploring the urban enviroment. In Proc. ISWC’97, pages 74–81, Cambridge, MA, USA.

[Julier2000] Julier, S., Baillot, Y., Lanzagorta, M., Brown, D., and Rosenblum, L. (2000). BARS: Battlefield augmented reality system. In

Proc. NATO Symposium on Information Processing Techniques for Military Systems, Istanbul, Turkey.

[Langlotz2011] Langlotz, T., Degendorfer, C., Mulloni, A., Schall, G., Reitmayr, G., and Schmalstieg, D. (2011). Robust detection and

tracking of annotations for outdoor augmented reality browsing. Computers & Graphics, 35(4):831–840.

[Langlotz2010] Langlotz, T., Mooslechner, S., Zollmann, S., Degendorfer, C., Reitmayr, G., and Schmalstieg, D. (2010). Sketching up the

world: In-situ authoring for mobile augmented reality. In Proc. Int. Workshop on Smartphone Applications and Services.

[Newman2007] Newman, J., Bornik, A., Pustka, D., Echtler, F., Huber, M., Schmalstieg, D., and Klinker, G. (2007). Tracking for distributed

mixed reality environments. In Proc. IEEE VR Worshop on Trends and Issues in Tracking for Virtual Environments, Charlotte NC, USA.

[Newman2001] Newman, J., Ingram, D., and Hopper, A. (2001). Augmented reality in a wide area sentient environment. In Proc. ISAR

2001, 77–86, New York, USA.

[Piekarski2002] Piekarski, W. and Thomas, B. H. (2002). The tinmith system - demonstrating new techniques for mobile augmented reality

modelling. In Proc. AUIC2002, Melbournce, Vic, Australia.

[Piekarski2004] Piekarski, W. and Thomas, B. H. (2004). Augmented reality working planes: A foundation for action and construction at a

distance. In Proc. ISMAR 2004, pages 162–171, Arlington, VA, USA.

[Reitmayr2004] Reitmayr, G. and Schmalstieg, D. (2004). Collaborative augmented reality for outdoor navigation and information browsing.

In Proc. Symposium Location Based Services and TeleCartography - Geowissenschaftliche Mitteilungen, volume 66, pages 53–62, Vienna,

Austria. Wiley.

[Schall2010] Schall, G., Schmalstieg, D., and Junghanns, S. (2010). Vidente - 3d visualization of underground infrastructure using handheld

augmented reality. In GeoHydroinformatics: Integrating GIS and Water Engineering. CRC Press.

[Spohrer1999] Spohrer, J. C. (1999). Information in places. IBM Systems Journal, 38(4):602–628.

[Starner1997] Starner, T., Mann, S., Rhodes, B., Levine, J., Healey, J., Kirsch, D., Picard, R., and Pentland, A. (1997). Augmented reality

through wearable computing. Presence - Teleoperators and Virtual Environments, Special Issue on Augmented Reality(4):386–398.

[Veas2008] Veas, E. and Kruijff, E. (2008). Vesp’r: design and evaluation of a handheld ar device. In Livingston, M. A., Bimber, O., and

Saito, H., editors, Proc. ISMAR 2008, pages 43–52, Cambridge, UK. IEEE Computer Society.

[Wagner2010] Wagner, D., Mulloni, A., Langlotz, T., and Schmalstieg, D. (2010). Real-time panoramic mapping and tracking on mobile

phones. In Proc. VR 2010, Waltham, Massachusetts, USA. IEEE.

[Weiser1991] Weiser, M. (1991). The computer of the twenty-first century. Scientific American, 265(3):94– 104.

[Pan2011] Pan, Q., Arth, C., Reitmayr, G., Rosten, E., and Drummond, T. W. (2011). Rapid scene reconstruction on mobile phones from

panoramic images. In Proc. ISMAR 2011, pages 55–64, Basel, Switzerland.

More magazines by this user
Similar magazines