Calibration-free Augmented Reality - Chair for Computer Aided ...

Calibration-free
Augmented Reality
Algorithms for Augmented and Virtual
Reality
by Miriam Wiegand

I. Introduction
What do we want to do?
Overlaying three-dimensional graphical objects on live
video of dynamic environments
Applications of this visualization technique:
- guiding trainees through complex manipulation and
maintenance tasks
- overlaying clinical 3D data with live video of patients
during surgical planning
- developing three-dimensional user interfaces
21.01.2005 Calibration-free Augmented Reality
2

Examples
Video seethrough
example on
a
consumer
cell-phone
21.01.2005 Calibration-free Augmented Reality
3

Typical approaches:
- use a stationary camera
- or rely on 6D position tracking devices and
precise camera calibration
21.01.2005 Calibration-free Augmented Reality
4

No calibration
Goal:
- Find a way to avoid camera calibration
- Development of simple and portable video-based AR
systems
- That are easy to initialise
- Impose minimal hardware requirements
21.01.2005 Calibration-free Augmented Reality
5

Reasons:
- End user should not have to perform a camera-
calibration step!
- Camera must be dynamically recalibrated whenever
its position or its intrinsic parameters change
- Additional equipment would be needed
21.01.2005 Calibration-free Augmented Reality
6

What's possible anyway?
- Shown pictures didn't use any camera calibration!
- No metric information about the calibration parameters
of the camera
- No information about the 3D locations and dimensions
of the environment's objects
21.01.2005 Calibration-free Augmented Reality
7

What we will do:
- Representing virtual objects so that their projection can
be computed as a linear combination of the projection of
the fiducial points
- Tracking regions and colour fiducial points at frame rate
- How does this work?
21.01.2005 Calibration-free Augmented Reality
8

II. Main part
Marker design
Mathematical foundations
Application
Tomographic Reflection
21.01.2005 Calibration-free Augmented Reality
9

Marker design
Version one
(by Kutulakos & Vallino)
- Fiducial points are specified by the
user during system initialisation
- Non-coplanar
- Only requirement: ability to track
across frames at least four fiducial
points
21.01.2005 Calibration-free Augmented Reality
10

Version two
(by Möhring, Lessing, Bimber)
- Layout has the shape of a common Cartesian
coordinate system with three axes
- Four basis points = end points of each axis and the
origin
- Axes of the marker defined by coloured line features
21.01.2005 Calibration-free Augmented Reality
11

Mathematical foundations
(Kutulakos & Vallino)
camera
coordinate
system
P
image plane
O
world
coordinate
system
Accurate projection of a virtual object requires knowing precisely the
combined effect of the object-to-world (O),(
world-to-camera (C),(
camera-to-
image (P)(
) transformations
21.01.2005 Calibration-free Augmented Reality
12
C

Projection in homogeneous coordinates:
Matrix modelling the
object's projection
onto the image plane
Object-to-world
transformation matrix
Its projection
Point on the
virtual object
World-to-camera
transformation
matrix
21.01.2005 Calibration-free Augmented Reality
13

- Main idea:
represent the object and the camera in a single, non-
Euclidean coordinate frame defined by the fiducials
- This change of representation has two effects
1. simplifies the projection equation:
models the combined effects of the change in the object's
representation as well as the object-to-world, world-to-camera
and projection transformations
21.01.2005 Calibration-free Augmented Reality
14

2. Allows the elements of the projection matrix
to simply be the image coordinates of the fiducial
points
fiducial points contain all the information needed
- To achieve these two effects, results from the theory
of affine-invariant object representations are used
21.01.2005 Calibration-free Augmented Reality
15

Affine Point Representations
- Collection of points; at least four non-coplanar
- Exist of three components:
1. origin
2. affine basis points
3. affine coordinates
- Affine representation does not change if the same non-
singular linear transformation is applied to all of the
points
- Useful properties:
21.01.2005 Calibration-free Augmented Reality
16

Re-Projection Property
When the projection of the origin and basis points is known
in an image
, we can compute the projection of a
point p from its affine coordinates:
or equivalently:
21.01.2005 Calibration-free Augmented Reality
17

projection process for any camera position:
completely determined by matrix collecting image
coordinates of the affine basis points
object's projection:
if its affine coordinates are known, trivially computed by
tracking the affine basis points
- This process can also be inverted in order to compute
the affine coordinates of a point (Affine Reconstruction
Property)
21.01.2005 Calibration-free Augmented Reality
18

Projection example:
Depth information?
21.01.2005 Calibration-free Augmented Reality
19

Affine Camera Coordinate Frame
- Let and be the vectors corresponding to the first
and second row of
- Affine image plane of the camera:
spanned by these vectors
- Viewing direction
of the camera:
given by the cross product
affine 3D coordinate frame
21.01.2005 Calibration-free Augmented Reality
20

Application
Previous section:
Affine coordinates of a virtual object determined relative to
four fiducials in the environment
the points` projection become trivial to compute
Central idea:
Ignore the original representation
perform all
graphics operations with the affine representation of the
object
21.01.2005 Calibration-free Augmented Reality
21

- This representation allows rendering operations (e.g.
z-buffering and clipping) to be performed accurately
- Because:
depth order, as well as the intersection of lines
and planes is preserved under affine transformations
21.01.2005 Calibration-free Augmented Reality
22

- Entire projection process: 4x4 homogeneous matrix
- Visible surface rendering of a point p
(on an affine object):
Image coordinates of p`s
projection
p`s assigned
depth-value
21.01.2005 Calibration-free Augmented Reality
23

Projection example:
- Similarity between Euclidean and affine view
transformation matrices
existing hardware engines for real-time
projection can be used
21.01.2005 Calibration-free Augmented Reality
24

Movies
21.01.2005 Calibration-free Augmented Reality
25

..\Cell_Phone_AR.avi
21.01.2005 Calibration-free Augmented Reality
26

Problems:
- Low resolution of live video streams
Cell-phone
- High optical distortion of the phones` simple
cameras
- Little graphics and memory capabilities
Goal:
- To realize a first running system on one of today's consumer phones
- Resource economical
- Simple, low-cost marker detection algorithm
- With a static and unchangeable camera focus
- Need for calibration steps by a novice end-user has to be kept at a
minimum
21.01.2005 Calibration-free Augmented Reality
27

Light-Weight Optical Tracking
- Determine four non-coplanar points on a 3D paper
marker
- Used as input parameters for rendering the overlaid 3D
geometry correctly
21.01.2005 Calibration-free Augmented Reality
28

Finding the base points
- Marker detected in one frame
search window is defined to constrain the image
processing in the following frame
21.01.2005 Calibration-free Augmented Reality
29

- Searching for features by tracing horizontal scan
lines
- Features detected based on a relative increase in
one of the R/G/B channels
- Two consecutive intersections with a feature
determined
detected feature = edge of one marker axis
21.01.2005 Calibration-free Augmented Reality
30

- Compute slope of the line that connects the two
feature intersections
- Until the two endpoints are reached
First estimate of the edge's gradient
21.01.2005 Calibration-free Augmented Reality
31

- Compute other two axes
- Apply scan tracks that orbit the endpoints:
- Endpoints and Edges for the remaining axes computed
as for the first one
Base points of the marker
coordinate system
21.01.2005 Calibration-free Augmented Reality
32

Perspective Rendering
- Geometry has to be transformed into a global affine
coordinate frame
- Defined by the four non-coplanar basis points
21.01.2005 Calibration-free Augmented Reality
33

Differences in comparison to Kutulakos & Vallino
here:
- Virtual object is directly placed into the origin
no translation needed
- Pure rendering
- Tracking easier because of the coloured features
21.01.2005 Calibration-free Augmented Reality
34

Another simplification…
21.01.2005 Calibration-free Augmented Reality
35

Tomographic Reflection
- To permit in situ visualization of
ultrasound images
direct hand-eye
coordination
Goal
Increased safety, ease, and
reliability in certain invasive
procedures
21.01.2005 Calibration-free Augmented Reality
36

Materials and Method
Flat-panel
monitor
Half-silvered
mirror
Ultrasound
Transducer
Ultrasound slice
Certain geometric relationships must exist between
the slice being scanned
the monitor displaying the slice
the mirror
21.01.2005 Calibration-free Augmented Reality
37

ultrasound image will thus appear at its physical
location, independent of viewer location
21.01.2005 Calibration-free Augmented Reality
38

Embodiment of RTTR
- Original apparatus too large and heavy
pencil grip prototype (“sonic(
flashlight”, , 2002)
21.01.2005 Calibration-free Augmented Reality
39

Scan of a fluid-filled
balloon containing a short
piece of rubber tubing
Ophthalmologist using
the sonic flashlight to
guide retro bulbar
injection on a cadaver
21.01.2005 Calibration-free Augmented Reality
40

III. Conclusion
21.01.2005 Calibration-free Augmented Reality
41

References / Papers I
Kiriakos N. Kutulakos and James R. Vallino
Calibration-Free Augmented Reality
IEEE Transactions on Visualization and Computer Graphics, vol. 4, no. 1,
pp. 1-20, 1998
Kiriakos N. Kutulakos and James R. Vallino
Affine Object Representations for Calibration-Free Augmented Reality
Proceedings of IEEE Virtual Reality Annual International Symposium
(VRAIS), pp. 25-36, 1996
M. Moehring, C. Lessig, and O. Bimber
Video See-Through AR on Consumer Cell Phones
In proceedings of International Symposium on Augmented and Mixed
Reality (ISMAR'04), 2004
M. Moehring, and C. Lessig, and O. Bimber
Optical Tracking and Video See-Through AR on Consumer Cell Phones
In proceedings of Workshop on Virtual and Augmented Reality of the GI-
Fachgruppe AR/VR, 2004
21.01.2005 Calibration-free Augmented Reality
42

References / Papers II
G. Stetten, D. Shelton, W. Chang, V. Chib, R. Tamburo, D. Hildebrand, L.
Lobes, J. Sumkin,
Towards a clinically useful Sonic Flashlight,
IEEE International Symposium on Biomedical Imaging 2002
G. Stetten, V. Chib,
Overlaying Ultrasound Images on Direct Vision,
Journal of Ultrasound in Medicine, vol. 20, no. 3, pp. 235-240, 2001
G. Stetten, V. Chib, R. Tamburo,
Tomographic Reflection to Merge Ultrasound Images with Direct Vision,
IEEE Proceedings of the Applied Imagery Pattern Recognition (AIPR)
annual workshop, 2000, pp. 200-205.
21.01.2005 Calibration-free Augmented Reality
43

Images / Movies taken from:
Kiriakos N. Kutulakos and James R. Vallino
Calibration-Free Augmented Reality
G. Stetten, V. Chib,
Overlaying Ultrasound Images on Direct Vision,
G. Stetten, D. Shelton, W. Chang, V. Chib, R. Tamburo, D. Hildebrand, L.
Lobes, J. Sumkin,
Towards a clinically useful Sonic Flashlight,
M. Moehring, and C. Lessig, and O. Bimber
Optical Tracking and Video See-Through AR on Consumer Cell Phones
http://www.stetten.com/
http://www.se.rit.edu/~jrv/research/ar/index.html
http://www.uni-weimar.de/~bimber/research.php
21.01.2005 Calibration-free Augmented Reality
44

Thanks for your attention!
21.01.2005 Calibration-free Augmented Reality
45