Calibration-free Augmented Reality - Chair for Computer Aided ...
Calibration-free Augmented Reality - Chair for Computer Aided ...
Calibration-free Augmented Reality - Chair for Computer Aided ...
You also want an ePaper? Increase the reach of your titles
YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.
<strong>Calibration</strong>-<strong>free</strong><br />
<strong>Augmented</strong> <strong>Reality</strong><br />
Algorithms <strong>for</strong> <strong>Augmented</strong> and Virtual<br />
<strong>Reality</strong><br />
by Miriam Wiegand
I. Introduction<br />
What do we want to do?<br />
<br />
Overlaying three-dimensional graphical objects on live<br />
video of dynamic environments<br />
<br />
Applications of this visualization technique:<br />
- guiding trainees through complex manipulation and<br />
maintenance tasks<br />
- overlaying clinical 3D data with live video of patients<br />
during surgical planning<br />
- developing three-dimensional user interfaces<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
2
Examples<br />
Video seethrough<br />
example on<br />
a<br />
consumer<br />
cell-phone<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
3
Typical approaches:<br />
- use a stationary camera<br />
- or rely on 6D position tracking devices and<br />
precise camera calibration<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
4
No calibration<br />
Goal:<br />
- Find a way to avoid camera calibration<br />
- Development of simple and portable video-based AR<br />
systems<br />
- That are easy to initialise<br />
- Impose minimal hardware requirements<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
5
Reasons:<br />
- End user should not have to per<strong>for</strong>m a camera-<br />
calibration step!<br />
- Camera must be dynamically recalibrated whenever<br />
its position or its intrinsic parameters change<br />
- Additional equipment would be needed<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
6
What's possible anyway?<br />
- Shown pictures didn't use any camera calibration!<br />
- No metric in<strong>for</strong>mation about the calibration parameters<br />
of the camera<br />
- No in<strong>for</strong>mation about the 3D locations and dimensions<br />
of the environment's objects<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
7
What we will do:<br />
- Representing virtual objects so that their projection can<br />
be computed as a linear combination of the projection of<br />
the fiducial points<br />
- Tracking regions and colour fiducial points at frame rate<br />
- How does this work?<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
8
II. Main part<br />
Marker design<br />
Mathematical foundations<br />
Application<br />
Tomographic Reflection<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
9
Marker design<br />
Version one<br />
(by Kutulakos & Vallino)<br />
- Fiducial points are specified by the<br />
user during system initialisation<br />
- Non-coplanar<br />
- Only requirement: ability to track<br />
across frames at least four fiducial<br />
points<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
10
Version two<br />
(by Möhring, Lessing, Bimber)<br />
- Layout has the shape of a common Cartesian<br />
coordinate system with three axes<br />
- Four basis points = end points of each axis and the<br />
origin<br />
- Axes of the marker defined by coloured line features<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
11
Mathematical foundations<br />
(Kutulakos & Vallino)<br />
camera<br />
coordinate<br />
system<br />
P<br />
image plane<br />
O<br />
world<br />
coordinate<br />
system<br />
Accurate projection of a virtual object requires knowing precisely the<br />
combined effect of the object-to-world (O),(<br />
world-to-camera (C),(<br />
camera-to-<br />
image (P)(<br />
) trans<strong>for</strong>mations<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
12<br />
C
Projection in homogeneous coordinates:<br />
Matrix modelling the<br />
object's projection<br />
onto the image plane<br />
Object-to-world<br />
trans<strong>for</strong>mation matrix<br />
Its projection<br />
Point on the<br />
virtual object<br />
World-to-camera<br />
trans<strong>for</strong>mation<br />
matrix<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
13
- Main idea:<br />
represent the object and the camera in a single, non-<br />
Euclidean coordinate frame defined by the fiducials<br />
- This change of representation has two effects<br />
1. simplifies the projection equation:<br />
models the combined effects of the change in the object's<br />
representation as well as the object-to-world, world-to-camera<br />
and projection trans<strong>for</strong>mations<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
14
2. Allows the elements of the projection matrix<br />
to simply be the image coordinates of the fiducial<br />
points<br />
fiducial points contain all the in<strong>for</strong>mation needed<br />
- To achieve these two effects, results from the theory<br />
of affine-invariant object representations are used<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
15
Affine Point Representations<br />
- Collection of points; at least four non-coplanar<br />
- Exist of three components:<br />
1. origin<br />
2. affine basis points<br />
3. affine coordinates<br />
- Affine representation does not change if the same non-<br />
singular linear trans<strong>for</strong>mation is applied to all of the<br />
points<br />
- Useful properties:<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
16
Re-Projection Property<br />
When the projection of the origin and basis points is known<br />
in an image<br />
, we can compute the projection of a<br />
point p from its affine coordinates:<br />
or equivalently:<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
17
projection process <strong>for</strong> any camera position:<br />
completely determined by matrix collecting image<br />
coordinates of the affine basis points<br />
object's projection:<br />
if its affine coordinates are known, trivially computed by<br />
tracking the affine basis points<br />
- This process can also be inverted in order to compute<br />
the affine coordinates of a point (Affine Reconstruction<br />
Property)<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
18
Projection example:<br />
Depth in<strong>for</strong>mation?<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
19
Affine Camera Coordinate Frame<br />
- Let and be the vectors corresponding to the first<br />
and second row of<br />
- Affine image plane of the camera:<br />
spanned by these vectors<br />
- Viewing direction<br />
of the camera:<br />
given by the cross product<br />
affine 3D coordinate frame<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
20
Application<br />
Previous section:<br />
Affine coordinates of a virtual object determined relative to<br />
four fiducials in the environment<br />
the points` projection become trivial to compute<br />
Central idea:<br />
Ignore the original representation<br />
per<strong>for</strong>m all<br />
graphics operations with the affine representation of the<br />
object<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
21
- This representation allows rendering operations (e.g.<br />
z-buffering and clipping) to be per<strong>for</strong>med accurately<br />
- Because:<br />
depth order, as well as the intersection of lines<br />
and planes is preserved under affine trans<strong>for</strong>mations<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
22
- Entire projection process: 4x4 homogeneous matrix<br />
- Visible surface rendering of a point p<br />
(on an affine object):<br />
Image coordinates of p`s<br />
projection<br />
p`s assigned<br />
depth-value<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
23
Projection example:<br />
- Similarity between Euclidean and affine view<br />
trans<strong>for</strong>mation matrices<br />
existing hardware engines <strong>for</strong> real-time<br />
projection can be used<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
24
Movies<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
25
..\Cell_Phone_AR.avi<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
26
Problems:<br />
- Low resolution of live video streams<br />
Cell-phone<br />
- High optical distortion of the phones` simple<br />
cameras<br />
- Little graphics and memory capabilities<br />
Goal:<br />
- To realize a first running system on one of today's consumer phones<br />
- Resource economical<br />
- Simple, low-cost marker detection algorithm<br />
- With a static and unchangeable camera focus<br />
- Need <strong>for</strong> calibration steps by a novice end-user has to be kept at a<br />
minimum<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
27
Light-Weight Optical Tracking<br />
- Determine four non-coplanar points on a 3D paper<br />
marker<br />
- Used as input parameters <strong>for</strong> rendering the overlaid 3D<br />
geometry correctly<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
28
Finding the base points<br />
- Marker detected in one frame<br />
search window is defined to constrain the image<br />
processing in the following frame<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
29
- Searching <strong>for</strong> features by tracing horizontal scan<br />
lines<br />
- Features detected based on a relative increase in<br />
one of the R/G/B channels<br />
- Two consecutive intersections with a feature<br />
determined<br />
detected feature = edge of one marker axis<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
30
- Compute slope of the line that connects the two<br />
feature intersections<br />
- Until the two endpoints are reached<br />
First estimate of the edge's gradient<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
31
- Compute other two axes<br />
- Apply scan tracks that orbit the endpoints:<br />
- Endpoints and Edges <strong>for</strong> the remaining axes computed<br />
as <strong>for</strong> the first one<br />
Base points of the marker<br />
coordinate system<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
32
Perspective Rendering<br />
- Geometry has to be trans<strong>for</strong>med into a global affine<br />
coordinate frame<br />
- Defined by the four non-coplanar basis points<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
33
Differences in comparison to Kutulakos & Vallino<br />
here:<br />
- Virtual object is directly placed into the origin<br />
no translation needed<br />
- Pure rendering<br />
- Tracking easier because of the coloured features<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
34
Another simplification…<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
35
Tomographic Reflection<br />
- To permit in situ visualization of<br />
ultrasound images<br />
direct hand-eye<br />
coordination<br />
Goal<br />
Increased safety, ease, and<br />
reliability in certain invasive<br />
procedures<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
36
Materials and Method<br />
Flat-panel<br />
monitor<br />
Half-silvered<br />
mirror<br />
Ultrasound<br />
Transducer<br />
Ultrasound slice<br />
Certain geometric relationships must exist between<br />
the slice being scanned<br />
the monitor displaying the slice<br />
the mirror<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
37
ultrasound image will thus appear at its physical<br />
location, independent of viewer location<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
38
Embodiment of RTTR<br />
- Original apparatus too large and heavy<br />
pencil grip prototype (“sonic(<br />
flashlight”, , 2002)<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
39
Scan of a fluid-filled<br />
balloon containing a short<br />
piece of rubber tubing<br />
Ophthalmologist using<br />
the sonic flashlight to<br />
guide retro bulbar<br />
injection on a cadaver<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
40
III. Conclusion<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
41
References / Papers I<br />
<br />
Kiriakos N. Kutulakos and James R. Vallino<br />
<strong>Calibration</strong>-Free <strong>Augmented</strong> <strong>Reality</strong><br />
IEEE Transactions on Visualization and <strong>Computer</strong> Graphics, vol. 4, no. 1,<br />
pp. 1-20, 1998<br />
<br />
Kiriakos N. Kutulakos and James R. Vallino<br />
Affine Object Representations <strong>for</strong> <strong>Calibration</strong>-Free <strong>Augmented</strong> <strong>Reality</strong><br />
Proceedings of IEEE Virtual <strong>Reality</strong> Annual International Symposium<br />
(VRAIS), pp. 25-36, 1996<br />
<br />
M. Moehring, C. Lessig, and O. Bimber<br />
Video See-Through AR on Consumer Cell Phones<br />
In proceedings of International Symposium on <strong>Augmented</strong> and Mixed<br />
<strong>Reality</strong> (ISMAR'04), 2004<br />
<br />
M. Moehring, and C. Lessig, and O. Bimber<br />
Optical Tracking and Video See-Through AR on Consumer Cell Phones<br />
In proceedings of Workshop on Virtual and <strong>Augmented</strong> <strong>Reality</strong> of the GI-<br />
Fachgruppe AR/VR, 2004<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
42
References / Papers II<br />
G. Stetten, D. Shelton, W. Chang, V. Chib, R. Tamburo, D. Hildebrand, L.<br />
Lobes, J. Sumkin,<br />
Towards a clinically useful Sonic Flashlight,<br />
IEEE International Symposium on Biomedical Imaging 2002<br />
<br />
G. Stetten, V. Chib,<br />
Overlaying Ultrasound Images on Direct Vision,<br />
Journal of Ultrasound in Medicine, vol. 20, no. 3, pp. 235-240, 2001<br />
<br />
G. Stetten, V. Chib, R. Tamburo,<br />
Tomographic Reflection to Merge Ultrasound Images with Direct Vision,<br />
IEEE Proceedings of the Applied Imagery Pattern Recognition (AIPR)<br />
annual workshop, 2000, pp. 200-205.<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
43
Images / Movies taken from:<br />
<br />
<br />
Kiriakos N. Kutulakos and James R. Vallino<br />
<strong>Calibration</strong>-Free <strong>Augmented</strong> <strong>Reality</strong><br />
G. Stetten, V. Chib,<br />
Overlaying Ultrasound Images on Direct Vision,<br />
G. Stetten, D. Shelton, W. Chang, V. Chib, R. Tamburo, D. Hildebrand, L.<br />
Lobes, J. Sumkin,<br />
Towards a clinically useful Sonic Flashlight,<br />
<br />
<br />
<br />
<br />
M. Moehring, and C. Lessig, and O. Bimber<br />
Optical Tracking and Video See-Through AR on Consumer Cell Phones<br />
http://www.stetten.com/<br />
http://www.se.rit.edu/~jrv/research/ar/index.html<br />
http://www.uni-weimar.de/~bimber/research.php<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
44
Thanks <strong>for</strong> your attention!<br />
21.01.2005 <strong>Calibration</strong>-<strong>free</strong> <strong>Augmented</strong> <strong>Reality</strong><br />
45