download LR pdf - AR Lab

01 april
2012
Augmented ReAlity, ARt And technology
intRoducing
Added woRlds
Yolande Kolstee
the
technology
Behind
Augmented
ReAlity
Pieter Jonker
Re-intRoducing
mosquitos
Maarten Lamers
how did we do it
Wim van Eck

2
AR[t]
Magazine about Augmented
Reality, art and technology
APRil 2012
3

CoLoPHoN
issn numBeR
2213-2481
contAct
The Augmented Reality Lab (AR Lab)
Royal Academy of Art, The Hague
(Koninklijke Academie van Beeldende Kunsten)
Prinsessegracht 4
2514 AN The Hague
The Netherlands
+31 (0)70 3154795
www.arlab.nl
info@arlab.nl
editoRiAl teAm
Yolande Kolstee, Hanna Schraffenberger,
Esmé Vahrmeijer (graphic design)
and Jouke Verlinden.
contRiButoRs
Wim van Eck, Jeroen van Erp, Pieter Jonker,
Maarten Lamers, Stephan Lukosch, Ferenc Molnár
(photography) and Robert Prevel.
coVeR
‘George’, an augmented reality headset designed
by Niels Mulder during his Post Graduate Course
Industrial Design (KABK), 2008
www.arlab.nl
TABLE oF CoNTENTS
32
welcome
to AR[t]
intRoducing Added woRlds
Yolande Kolstee
inteRView with
helen PAPAgiAnnis
Hanna Schraffenberger
the technology Behind AR
Pieter Jonker
Re-intRoducing mosquitos
Maarten Lamers
lieVen VAn VelthoVen —
the RAcing stAR
Hanna Schraffenberger
how did we do it
Wim van Eck
Pixels wAnt to Be fReed!
intRoducing Augmented
ReAlity enABling hARdwARe
technologies
Jouke Verlinden
07 60
ARtist in Residence
PoRtRAit: mARinA de hAAs
Hanna Schraffenberger
A mAgicAl leVeRAge —
in seARch of the
killeR APPlicAtion.
Jeroen van Erp
the Positioning
of ViRtuAl oBjects
Robert Prevel
mediAted ReAlity foR cRime
scene inVestigAtion
Stephan Lukosch
die wAlküRe
Wim van Eck, AR Lab Student Project
4 5
08
12
20
66
70
28 72
30
36
42
36
76

WELCoME...
to the first issue of AR[t],
the magazine about
Augmented Reality, art
and technology!
Starting with this issue, AR[t] is an aspiring
magazine series for the emerging AR community
inside and outside the Netherlands. The
magzine is run by a small and dedicated team
of researchers, artists and lecturers of the AR
Lab (based at the Royal Academy of Arts, The
Hague), Delft University of Technology (TU
Delft), Leiden University and SME. In AR[t], we
share our interest in Augmented Reality (AR),
discuss its applications in the arts and provide
insight into the underlying technology.
At the AR Lab, we aim to understand, develop,
refine and improve the amalgamation of the
physical world with the virtual. We do this
through a project-based approach and with the
help of research funding from RAAK-Pro. In the
magazine series, we invite writers from the industry,
interview artists working with Augmented
Reality and discuss the latest technological
developments.
It is our belief that AR and its associated
technologies are important to the field of new
media: media artists experiment with the intersection
of the physical and the virtual and probe
the limits of our sensory perception in order to
create new experiences. Managers of cultural
heritage are seeking after new possibilities for
worldwide access to their collections. Designers,
developers, architects and urban planners
are looking for new ways to better communicate
their designs to clients. Designers of games and
theme parks want to create immersive experiences
that integrate both the physical and the
virtual world. Marketing specialists are working
with new interactive forms of communication.
For all of them, AR can serve as a powerful tool
to realize their visions.
Media artists and designers who want to acquire
an interesting position within the domain of new
media have to gain knowledge about and experience
with AR. This magazine series is intended to
provide both theoretical knowledge as well as a
guide towards first practical experiences with AR.
our special focus lies on the diversity of contributions.
Consequently, everybody who wants
to know more about AR should be able to find
something of interest in this magzine, be they
art and design students, students from technical
backgrounds as well as engineers, developers,
inventors, philosophers or readers who just
happened to hear about AR and got curious.
We hope you enjoy the first issue and invite you
to check out the website www.arlab.nl to learn
more about Augmented Reality in the arts and
the work of the AR Lab.
www.arlab.nl
6 7

intRoducing Added woRlds:
Augmented ReAlity is heRe!
By yolande kolstee
Augmented Reality is a relatively recent computer
based technology that differs from the earlier
known concept of Virtual Reality. Virtual Reality is
a computer based reality where the actual, outer
world is not directly part of, whereas Augmented
Reality can be characterized by a combination of
the real and the virtual.
Augmented Reality is part of the broader concept of
Mixed Reality: environments that consist of the real
and the virtual. To make these differences and relations
more clear, industrial engineer Paul Milgram
and Fumio Kishino introduced the Mixed Reality
Continuum diagram in 1994, in which the real world
is placed on the one end and the virtual world is
placed on the other end.
Real
environment
Augmented
Reality (AR)
mixed ReAlity (mR)
Virtuality continuum by Paul Milgram and Fumio Kishino (1994)
8
Augmented
Virtuality (AV)
Virtual
environment
A shoRt oVeRView of AR
We define Augmented Reality as integrating 3-D
virtual objects or scenes into a 3-D environment
in real time (cf. Azuma, 1997).
WHERE 3D VIRTUAL oBJECTS
oR SCENES CoME FRoM
What is shown in the virtual world, is created
first. There are three ways of creating virtual
objects:
1. By hand: using 3d computer graphics
Designers create 3D drawings of objects, game
developers create 3D drawings of (human) figures,
(urban) architects create 3D drawings of buildings
and cities. This 3D modeling by (product)
designers, architects, and visual artists is done
by using specific software. Numerous software
programs are developed. While some software
packages can be downloaded for free, others
are pretty expensive. Well known examples are
Maya, Cinema 4D, Studio Max, Blender, Sketch up,
Rhinoceros, Solidworks, Revit, Zbrush, AutoCad,
Autodesk. By now at least 170 different software
programs are available.
2. By computer controlled imaging
equipment/3d scanners.
We can distinguish different types of threedimensional
scanners – the ones used in the
bio-medical world and the ones used for other
purposes – although there is some overlapping.
Inspecting a piece of medieval art or inspecting a
living human being is different but somehow also
alike. In recent years we see a vigorous expansion
of the use of image-producing bio-medical
equipment. We owe these developments to the
of engineer sir Godfrey Hounsfield and physicist
Allan Cormack, among others, who were
jointly awarded the Nobel Prize in 1979 for their
pioneering work on X-ray computed tomography
(CT). Another couple of Nobel Prize winners are
Paul C. Lauterbur and Peter Mansfield, who won
the prize in 2003 for their discoveries concerning
magnetic resonance imaging (MRI). Although
their original goals were different, in the field of
Augmented Reality one might use the 3D virtual
models that are produced by such systems. However,
they have to be processed prior to use in
AR because they might be too heavy. A 3D laser
scanner is a device that analyses a real-world object
or environment to collect data on its shape
and its appearance (i.e. colour). The collected
data can then be used to construct digital, three
dimensional models. These scanners are sometimes
called 3D digitizers. The difference is that
the above medical scanners are looking inside to
create a 3D model while the laser scanners are
creating a virtual image from the reflection of
the outside of an object.
3. Photo and/or film images
It is possible to use a (moving) 2D image like a
picture as a skin on a virtual 3D model. In this
way the 2D model gives a three-dimensional
impression.
INTEGRATING 3-D VIRTUAL oB-
JECTS IN THE REAL WoRLD IN
REAL TIME
There are different ways of integrating the virtual
objects or scenes into the real world. For all
three we need a display possibility. This might
be a screen or monitor, small screens in AR
glasses, or an object on which the 3D images are
projected. We distinguish three types of (visual)
Augmented Reality:
display type i : screen based
AR on a monitor, for example on a flatscreen or
on a smart phone (using e.g. LAYAR). With this
technology we see the real world and added at
the same time on a computer screen, monitor,
smartphone or tablet computer, the virtual
object. In that way, we can, for example,
9

Artist: kARolinA soBeckA | http://www.gravitytrap.com
add information to a book, by looking at the
book and the screen at the same time.
display type ii:
AR glasses (off-screen)
A far more sophisticated but not yet consumer
friendly method uses AR glasses or a head
mounted display (HMD), also called a head-up
display. With this device the extra information is
mixed with one’s own perception of the world.
The virtual images appear in the air, in the real
world, around you, and are not projected on a
screen. In type II there are two types of mixing
the real world with the virtual world:
Video see-through: a camera captures the real
world. The virtual images are mixed with the
captures (video images) of the real world and
this mix creates an Augmented Reality.
optical see-through: the real world is perceived
directly with one’s own eyes in real time. Via
small translucent mirrors in goggles, virtual
images are displayed on top of the perceived
Reality.
display type iii:
Projection based Augmented Reality
With projection based AR we project virtual 3D
scenes or objects on a surface of a building of an
object (or a person). To do this, we need to know
exactly the dimensions of the object we project
AR info onto. The projection is seen on the object
or building with remarkable precision. This can
generate very sophisticated or wild projections
on buildings. The Augmented Matter in Context
group, led by Jouke Verlinden at the Faculty of
Industrial Design Engineering, TU-Delft, uses
pro jection-based AR for manipulating the appear-
ance of products.
CoNNECTING ART AND
TECHNoLoGY
The 2011 IEEE International Symposium on Mixed
and Augmented Reality (ISMAR) was held in
Basel, Switzerland. In the track Arts, Media, and
Humanities, 40 articles were offered discussing
the connection of ‘hard’ physics and ‘soft’ art.
There are several ways in which art and Augmented
Reality technology can be connected:
we can, for example, make art with Augmented
Reality technology, create Augmented Reality
artworks or use Augmented Reality technology
to show and explain existing art (such as a
monument like the Greek Pantheon or paintings
from the grottos of Lascaux). Most of the contributions
of the conference concerned Augmented
Reality as a tool to present, explain or augment
existing art. However, some visual artists use AR
as a medium to create art.
The role of the artist in working with the emerg-
ing technology of Augmented Reality has been
discussed by Helen Papagiannis in her ISMAR
paper The Role of the Artist in Evolving AR as a
New Medium (2011). In her paper, Helen Papagi-
annis reviews how the use of technology as a cre-
ative medium has been discussed in recent years.
She points out, that in 1988 John Pearson wrote
about how the computer offers artists “new
means for expressing their ideas” (p.73., cited in
Papagiannis, 2011, p.61). According to Pearson,
“Technology has always been, the handmaiden of
the visual arts, as is obvious, a technical means is
always necessary for the visual communication of
ideas, of expression or the development of works
of art—tools and materials are required.” (p. 73)
However, he points out that new technologies
“were not developed by the artistic community
for artistic purposes, but by science and industry
to serve the pragmatic or utilitarian needs of
society.” (p.73., cited in Papagiannis, 2011, p.61)
As Helen Papagiannis concludes, it is then up to
the artist “to act as a pioneer, pushing forward
a new aesthetic that exploits the unique materi-
als of the novel technology” (2011, p.61). Like
Helen, we believe this holds also for the emerging
field of AR technologies and we hope, artists will
set out to create exciting new Augmented Reality
art and thereby contribute to the interplay
between art and technology. An interview with
Helen Papagiannis can be found on page 12 of this
magazine. A portrait of the artist Marina de Haas,
who did a residency at the AR Lab, can be found
on page 60.
RefeRences
■ Milgram P. and Kishino, F., “A Taxonomy of
Mixed Reality Visual Displays,” IEICE Trans.
Information Systems, vol. E77-D, no. 12, 1994,
pp. 1321-1329.
■ Azuma, Ronald T., “A Survey of Augmented
Reality”. In Presence: Teleoperators and
Virtual Environments 6, 4 (August 1997),
pp. 355-385.
■ Papagiannis, H., “The Role of the Artist
in Evolving AR as a New Medium”, 2011
IEEE International Symposium on Mixed and
Augmented Reality(ISMAR) – Arts, Media, and
Humanities (ISMAR-AMH), Basel, Switserland,
pp. 61-65.
■ Pearson, J., “The computer: Liberator or
Jailer of The creative Spirit.” Leonardo,
Supplemental Issue, Electronic Art, 1 (1988),
pp. 73-80.
10 11

BIoGRAPHY -
HELEN PAPAGIANNIS
helen Papagiannis is a designer, artist,
and Phd researcher specializing in Augmented
Reality (AR) in toronto, canada.
helen has been working with AR since
2005, exploring the creative possibilities
for AR with a focus on content development
and storytelling. she is a senior
Research Associate at the Augmented
Reality lab at york university, in the
department of film, faculty of fine Arts.
helen has presented her interactive
artwork and research at global juried
12
conferences and events including tedx
(technology, entertainment, design),
ismAR (international society for mixed
and Augmented Reality) and iseA (international
symposium for electronic Art).
Prior to her Augmented life, helen was a
member of the internationally renowned
Bruce mau design studio where she was
project lead on “massive change:
the future of global design." Read more
about helen’s work on her blog and follow
her on twitter: @ARstories.
www.augmentedstories.com
INTERVIEW WITH
HELEN PAPAGIANNIS
BY HANNA SCHRAFFENBERGER
What is Augmented Reality?
Augmented Reality (AR) is a real-time layering of
virtual digital elements including text, images,
video and 3D animations on top of our existing
reality, made visible through AR enabled devices
such as smart phones or tablets equipped with
a camera. I often compare AR to cinema when
it was first new, for we are at a similar moment
in AR’s evolution where there are currently no
conventions or set aesthetics; this is a time ripe
with possibilities for AR’s creative advancement.
Like cinema when it first emerged, AR has commenced
with a focus on the technology with
little consideration to content. AR content needs
to catch up with AR technology. As a community
of designers, artists, researchers and commercial
industry, we need to advance content in AR
and not stop with the technology, but look at
what unique stories and utility AR can present.
So far, AR technologies are still
new to many people and often
AR works cause a magical experience.
Do you think AR will lose
its magic once people get used to
the technology and have developed
an understanding of how AR
works? How have you worked with
this ‘magical element’ in your
work ‘The Amazing Cinemagician’?
I wholeheartedly agree that AR can create a
magical experience. In my TEDx 2010 talk, “How
Does Wonderment Guide the Creative Process”
(http://youtu.be/ScLgtkVTHDc), I discuss how
AR enables a sense of wonder, allowing us to see
our environments anew. I often feel like a magician
when presenting demos of my AR work live;
astonishment fills the eyes of the beholder questioning,
“How did you do that?” So what happens
when the magic trick is revealed, as you ask,
when the illusion loses its novelty and becomes
habitual? In Virtual Art: Illusion to Immersion
(2004), new media art-historian oliver Grau
discusses how audiences are first overwhelmed
by new and unaccustomed visual experiences,
but later, once “habituation chips away at the
illusion”, the new medium no longer possesses
“the power to captivate” (p. 152). Grau writes
that at this stage the medium becomes “stale
and the audience is hardened to its attempts
at illusion”; however, he notes, that it is at this
stage that “the observers are receptive to content
and media competence” (p. 152).
When the initial wonder and novelty of the
technology wear off, will it be then that AR is
explored as a possible media format for various
content and receive a wider public reception as
a mass medium? or is there an element of wonder
that need exist in the technology for it to
be effective and flourish?
13

Picture: PiPPin lee
“Pick a card. Place it here.
Prepare to be amazed and
entertained.”
14
I believe AR is currently entering the stage of
content development and storytelling, however,
I don’t feel AR has lost its “power to captivate”
or “become stale”, and that as artists, designers,
researchers and storytellers, we continue to
maintain wonderment in AR and allow it to guide
and inspire story and content. Let’s not forget
the enchantment and magic of the medium. I
often reference the work of French filmmaker
and magician George Méliès (1861-1938) as a
great inspiration and recently named him the
Patron Saint of AR in an article for The Creators
Project (http://www.thecreatorsproject.com/
blog/celebrating-georges-méliès-patron-saintof-augmented-reality)
on what would have been
Méliès’ 150th birthday. Méliès was first a stage
magician before being introduced to cinema at
a preview of the Lumiere brothers’ invention,
where he is said to have exclaimed, “That’s
for me, what a great trick”. Méliès became
famous for the “trick-film”, which employed a
stop- motion and substitution technique. Méliès
applied the newfound medium of cinema to
extend magic into novel, seemingly impossible
visualities on the screen.
I consider AR, too, to be very much about creat-
ing impossible visualities. We can think of AR as
a real-time stop-substitution, which layers content
dynamically atop the physical environment
and creates virtual actualities with shapeshifting
objects, magically appearing and disappearing—
as Méliès first did in cinema.
In tribute to Méliès, my Mixed Reality exhibit,
The Amazing Cinemagician integrates Radio
Frequency Identification (RFID) technology with
the FogScreen, a translucent projection screen
consisting of a thin curtain of dry fog. The
Amazing Cinemagician speaks to technology as
magic, linking the emerging technology of the
FogScreen with the pre-cinematic magic lantern
and phantasmagoria spectacles of the Victorian
era. The installation is based on a card-trick,
using physical playing cards as an interface
to interact with the FogScreen. RFID tags are
hidden within each physical playing card. Part
of the magic and illusion of this project was to
disguise the RFID tag as a normal object, out
of the viewer’s sight. Each of these tags corresponds
to a short film clip by Méliès, which is
projected onto the FogScreen once a selected
card is placed atop the RFID tag reader. The
RFID card reader is hidden within an antique
wooden podium (adding to the aura of the magic
performance and historical time period).
The following instructions were provided to the
participant: “Pick a card. Place it here. Prepare
to be amazed and entertained.” once the
participant placed a selected card atop the designated
area on the podium (atop the concealed
RFID reader), an image of the corresponding
card was revealed on the FogScreen, which was
then followed by one of Méliès’ films. The decision
was made to provide visual feedback of the
participant’s selected card to add to the magic
of the experience and to generate a sense of
wonder, similar to the witnessing and questioning
of a magic trick, with participants asking,
“How did you know that was my card? How did
you do that?” This curiosity inspired further
exploration of each of the cards (and in turn,
Méliès’ films) to determine if each of the participant’s
cards could be properly identified.
You are an artist and researcher.
Your scientific work as well as
your artistic work explores how
AR can be used as a creative
medium. What’s the difference
between your work as an artist /
designer and your work as a researcher?
Excellent question! I believe that artists and
designers are researchers. They propose novel
paths for innovation introducing detours into the
usual processes. In my most recent TEDx 2011
talk in Dubai, “Augmented Reality and the Power
of Imagination” (http://youtu.be/7QrB4cYxjmk),
15

Picture: HELEN PAPAGIANNIS
I discuss how as a designer/artist/PhD researcher
I am both a practitioner and a researcher, a maker
and a believer. As a practitioner, I do, create,
design; as a researcher I dream, aspire, hope.
I am a make-believer working with a technology
that is about make-believe, about imagining
possibilities atop actualities. Now, more than
ever, we need more creative adventurers and
make-believers to help AR continue to evolve
and become a wondrous new medium, unlike
anything we’ve ever seen before! I spoke to the
importance and power of imagination and makebelieve,
and how they pertain to AR at this critical
junction in the medium’s evolution. When
we make-believe and when we imagine, we are
in two places simultaneously; make-believe is
about projecting or layering our imagination
on top of a current situation or circumstance.
In many ways, this is what AR is too: layering
imagined worlds on top of our existing reality.
You’ve had quite a success with
your AR pop-up book ‘Who’s
Afraid of Bugs?’ In your blog you
talk about your inspiration for
the story behind the book: it was
inspired by AR psychotherapy
studies for the treatment of
phobias such as arachnophobia.
Can you tell us more?
Who’s Afraid of Bugs? was the world’s first Augmented
Reality (AR) Pop-up designed for iPad2
and iPhone 4. The book combines hand-crafted
paper-engineering and AR on mobile devices to
create a tactile and hands-on storybook that
explores the fear of bugs through narrative and
play. Integrating image tracking in the design,
as opposed to black and white glyphs commonly
seen in AR, the book can hence be enjoyed alone
as a regular pop-up book, or supplemented with
Augmented digital content when viewed through
a mobile device equipped with a camera. The
book is a playful exploration of fears using AR in
a meaningful and fun way. Rhyming text takes
the reader through the storybook where various
‘creepy crawlies’ (spider, ant, and butterfly) are
awaiting to be discovered, appearing virtually
as 3D models you can interact with. A tarantula
attacks when you touch it, an ant hyperlinks to
educational content with images and diagrams,
and a butterfly appears flapping its wings atop
a flower in a meadow. Hands are integrated
throughout the book design, whether its pla cing
one’s hand down to have the tarantula crawl
over you virtually, the hand holding the magnifying
lens that sees the ant, or the hands that popup
holding the flower upon which the butterfly
appears. It’s a method to involve the reader in
the narrative, but also comments on the unique
tactility AR presents, bridging the digital with
the physical. Further, the story for the AR
Pop-up Book was inspired by AR psychotherapy
studies for the treatment of phobias such as
arachnophobia. AR provides a safe, controlled
environment to conduct exposure therapy
within a patient’s physical surroundings, creating
a more believable scenario with heightened
presence (defined as the sense of really being in
an imagined or perceived place or scenario) and
provides greater immediacy than in Virtual Reality
(VR). A video of the book may be watched at
http://vimeo.com/25608606.
In your work, technology serves
as an inspiration. For example,
rather than starting with a story
which is then adapted to a certain
technology, you start out with
AR technology, investigate its
strengths and weaknesses and so
the story evolves. However, this
does not limit you to only use the
strength of a medium.
On the contrary, weaknesses such
as accidents and glitches have
for example influenced your work
‘Hallucinatory AR’. Can you tell us
a bit more about this work?
Hallucinatory Augmented Reality (AR), 2007,
was an experiment which investigated the
possibility of images which were not glyphs/AR
trackables to generate AR imagery. The projects
evolved out of accidents, incidents in earlier
experiments in which the AR software was mistaking
non-marker imagery for AR glyphs and
attempted to generate AR imagery. This confusion,
by the software, resulted in unexpected
and random flickering AR imagery. I decided to
explore the creative and artistic possibilities
of this effect further and conduct experiments
with non-traditional marker-based tracking.
The process entailed a study of what types of
non-marker images might generate such ‘hallucinations’
and a search for imagery that would
evoke or call upon multiple AR imagery/videos
from a single image/non-marker.
Upon multiple image searches, one image
emerged which proved to be quite extraordinary.
A cathedral stained glass window was
able to evoke four different AR videos, the only
instance, from among many other images, in
which multiple AR imagery appeared. Upon close
examination of the image, focusing in and out
with a web camera, a face began to emerge in
the black and white pattern. A fantastical image
of a man was encountered. Interestingly, it
was when the image was blurred into this face
using the web camera that the AR hallucinatory
imagery worked best, rapidly multiplying and
appearing more prominently. Although numerous
attempts were made with similar images,
no other such instances occurred; this image
appeared to be unique.
The challenge now rested in the choice of what
types of imagery to curate into this hallucinatory
viewing: what imagery would be best suited to
this phantasmagoric and dream-like form?
My criteria for imagery/videos were like-form
and shape, in an attempt to create a collage-like
set of visuals. As the sequence or duration of
the imagery in Hallucinatory AR could not be
predetermined, the goal was to identify imagery
16 17

that possessed similarities, through which the
possibility for visual synchronicities existed.
Themes of intrusions and chance encounters are
at play in Hallucinatory AR, inspired in part by
Surrealist artist Max Ernst. In What is the Mechanism
of Collage? (1936), Ernst writes:
one rainy day in 1919, finding myself on a village
on the Rhine, I was struck by the obsession
which held under my gaze the pages of an illustrated
catalogue showing objects designed for
anthropologic, microscopic, psychologic, mineralogic,
and paleontologic demonstration. There
I found brought together elements of figuration
so remote that the sheer absurdity of that collection
provoked a sudden intensification of
the visionary faculties in me and brought forth
an illusive succession of contradictory images,
double, triple, and multiple images, piling up
on each other with the persistence and rapidity
which are particular to love memories and visions
of half-sleep (p. 427).
of particular interest to my work in exploring
and experimenting with Hallucinatory AR was
Ernst’s description of an “illusive succession of
contradictory images” that were “brought forth”
(as though independent of the artist), rapidly
multiplying and “piling up” in a state of “halfsleep”.
Similarities can be drawn to the process
of the seemingly disparate AR images jarringly
coming in and out of view, layered atop one
another.
one wonders if these visual accidents are what
the future of AR might hold: of unwelcome
glitches in software systems as Bruce Sterling
describes on Beyond the Beyond in 2009; or
perhaps we might come to delight in the visual
poetry of these Augmented hallucinations that
are “As beautiful as the chance encounter of a
sewing machine and an umbrella on an operating
table.” 1
To a computer scientist, these ‘glitches’, as
applied in Hallucinatory AR, could potentially
be viewed or interpreted as a disaster, as an
18
example of the technology failing. To the artist,
however, there is poetry in these glitches, with
new possibilities of expression and new visual
forms emerging.
on the topic of glitches and accidents, I’d like to
return to Méliès. Méliès became famous for the
stop trick, or double exposure special effect,
a technique which evolved from an accident:
Méliès’ camera jammed while filming the streets
of Paris; upon playing back the film, he observed
an omnibus transforming into a hearse. Rather
than discounting this as a technical failure, or
glitch, he utilized it as a technique in his films.
Hallucinatory AR also evolved from an accident,
which was embraced and applied in attempt
to evolve a potentially new visual mode in the
medium of AR. Méliès introduced new formal
styles, conventions and techniques that were
specific to the medium of film; novel styles and
new conventions will also emerge from AR artists
and creative adventurers who fully embrace
the medium.
[1] Comte de Lautreamont’s often quoted allegory,
famous for inspiring both Max Ernst and Andrew
Breton, qtd. in: Williams, Robert. “Art Theory: An
Historical Introduction.” Malden, MA: Blackwell
Publishing, 2004: 197
“As beautiful as the chance
encounter of a sewing
machine and an umbrella
on an operating table.”
Comte de Lautréamont
Picture: PiPPin lee
19

THE TECHNoLoGY BEHIND
AUGMENTED REALITY
Augmented Reality (AR) is a field that is primarily
concerned with realistically adding computergenerated
images to the image one perceives
from the real world.
AR comes in several flavors. Best known is the
practice of using flatscreens or projectors,
but nowadays AR can be experienced even on
smartphones and tablet PCs. The crux is that 3D
digital data from another source is added to the
ordinary physical world, which is for example
seen through a camera. We can create this additional
data ourselves, e.g. using 3D drawing
programs such as 3D Studio Max, but we can
also add CT and MRI data or even live TV images
to the real world. Likewise, animated three
dimensional objects (avatars), which then can be
displayed in the real world, can be made using a
visualization program like Cinema 4D. Instead of
displaying information on conventional monitors,
the data can also be added to the vision of the
user by means of a head-mounted display (HMD)
or Head-Up Display. This is a second, less known
form of Augmented Reality. It is already known
to fighter pilots, among others. We distinguish
two types of HMDs, namely: optical See Through
(oST) headsets and Video See Through (VST)
headsets. oST headsets use semi-transparent
mirrors or prisms, through which one can keep
seeing the real world. At the same time, virtual
objects can be added to this view using small
displays that are placed on top of the prisms.
VSTs are in essence Virtual Reality goggles, so
the displays are placed directly in front of your
eyes. In order to see the real world, there are
two cameras attached on the other side of the
little displays. You can then see the Augmented
Reality by mixing the video signal coming from
the camera with the video signal containing the
virtual objects.
20 21

undeRlying technology
screens and glasses
unlike screen-based AR, hmds provide depth
perception as both eyes receive an image.
when objects are projected on a 2d screen,
one can convey an experience of depth by
letting the objects move. Recent 3d screens
allow you to view stationary objects in depth.
3d televisions that work with glasses quickly
alternate the right and left image - in sync with
this, the glasses use active shutters which let
the image in turn reach the left or the right
eye. this happens so fast that it looks like you
view both, the left and right image simultaneously.
3d television displays that work without
glasses make use of little lenses which are
placed directly on the screen. those refract
the left and right image, so that each eye can
only see the corresponding image. see for
example www.dimenco.eu/display-technology.
this is essentially the same method as used
on the well known 3d postcards on which a
beautiful lady winks when the card is slightly
turned. 3d film makes use of two projectors
that show the left and right images simultaneously,
however, each of them is polarized in a
different way. the left and right lenses of the
glasses have matching polarizations and only
let through the light of to the corresponding
projector. the important point with screens
is that you are always bound to the physical
location of the display while headset based
techniques allow you to roam freely. this is
called immersive visualization — you are immersed
in a virtual world. you can walk around
in the 3d world and move around and enter
virtual 3d objects.
Video-see-through AR will become popular
within a very short time and ultimately become
an extension of the smartphone. this is
because both display technology and camera
technology have made great strides with the
advent of smartphones. what currently still
might stand in the way of smartphone models
22
is computing power and energy consumption.
companies such as microsoft, google, sony,
Zeiss,... will enter the consumer market soon
with AR technology.
tracking technology
A current obstacle for major applications which
soon will be resolved is the tracking technology.
the problem with AR is embedding the virtual
objects in the real world. you can compare
this with color printing: the colors, e.g., cyan,
magenta, yellow and black have to be printed
properly aligned to each other. what you
often see in prints which are not cut yet, are
so called fiducial markers on the edge of the
printing plates that serve as a reference for
the alignment of the colors. these are also
necessary in AR. often, you see that markers
are used onto which a 3d virtual object is
projected. moving and rotating the marker, lets
you move and rotate the virtual object. such
a marker is comparable to the fiducial marker
in color printing. with the help of computer
vision technology, the camera of the headset
can identify the marker and based on it’s size,
shape and position, conclude the relative position
of the camera. if you move your head relative
to the marker (with the virtual object), the
computer knows how the image on the display
must be transformed so that the virtual object
remains stationary. And conversely, if your
head is stationary and you rotate the marker,
it knows how the virtual object should rotate
so that it remains on top of the marker.
AR smartphone applications such as layar use
the build in gPs and compass for the tracking.
this has an accuracy of meters and measures
angles of 5-10 degrees. camera-based tracking,
however, is accurate to the centimetre and can
measure angles of several degrees. nowadays,
using markers for the tracking is already out
of date and we use so called “natural feature
tracking” also called “keypoint tracking”.
here, the computer searches for conspicuous
(salient) key points in the left and right camera
image. if, for example, you twist your head, this
shift is determined on the basis of those key points
with more than 30 frames per second. this way, a
3d map of these keypoints can be built and the computer
knows the relationship (distance and angle)
between the keypoints and the stereo camera. this
method is more robust than marker based tracking
because you have many keypoints — widely spread
in the scene — and not just the four corners of the
marker close together in the scene. if someone
walks in front of the camera and blocks some of the
keypoints, there will still be enough keypoints left
and the tracking is not lost. moreover, you do not
have to stick markers all over the world.
collaboration with the Royal Academy of Arts
(kABk) in the hague in the AR lab (Royal
Academy, tu delft, leiden university, various
smes) in the realization of applications.
the tu delft has done research on AR since 1999.
since 2006, the university works with the art
academy in the hague. the idea is that AR is a new
technology with its own merits. Artists are very
good at finding out what is possible with the new
technology. here are some pictures of realized
projects. liseerde projecten
Fig 1. The current technology that replaces the markers
with natural feature tracking or so called keypoint
tracking. Instead of the four corners of the marker, the
computer itself determines which points in the left and
right images can be used as anchor points for calculating
the 3D pose of the camera in 3D space. From top:
1: you can use all points in the left and right images
to slowly build a complete 3D map. Such a map can,
for example, be used to relive your past experience
because you can again walk in the now virtual space.
2: the 3D keypoint space and the trace of the
camera position within it.
3: keypoints (the color indicates the suitability)
4: you can place virtual objects (eyes) on an existing
surface
23

Fig 2. Virtual furniture exhibition at the Salone di Mobile in Milan (2008); students of the Royal
Academy of Art, The Hague show their furnitures by means of AR headsets. This saves transportation
costs.
Fig 3. Virtual sculpture exhibition in Kröller-Müller (2009). From left:
1) visitors on adventure with laptops on walkers, 2) inside with a optical see-through headset,
3) large pivotable screen on a field of grass, 4) virtual image.
24
Fig 4. Exhibition in Museum Boijmans van Beuningen (2008-2009). From left: 1) Sgraffitto in 3D;
2) the 3D print version may be picked up by the spectator, 3) animated shards, the table covered
in ancient pottery can be seen via the headset, 4) scanning antique pottery with the CT scanner
delivers a 3D digital image.
Fig 5. The TUD, partially in collaboration with the Royal Academy (with the oldest industrial design
course in the Netherlands), has designed a number of headsets.This design of headsets is an ongoing
activity. From left: 1) first optical see-through headset with Sony headset and self-made inertia
tracker (2000), 2) on a construction helmet (2006), 3) SmartCam and tracker taped on a Cyber Mind
Visette headset (2007); 4) headset design with engines by Niels Mulder, a student at Royal Academy
of Art, The Hague (2007), based on Cybermind technology, 5) low cost prototype based on the Carl
Zeiss Cinemizer headset, 6) future AR Vizor?, 7) future AR lens?
25

there are many applications
that can be realized using AR;
they will find their way in the
coming decades:
1. head-up displays have already been used
for many years in the Air force for fighter
pilots; this can be extended to other
vehicles and civil applications.
2. the billboards during the broadcast of
a football game are essentially also AR;
more can be done by also ivolving the
game itself an allowing interaction of teh
user, such as off-side line projection.
3. in the professional sphere, you can, for
example, visualize where pipes under the
street lie or should lie. ditto for designing
ships, houses, planes, trucks and cars.
what’s outlined in a cAd drawing could
be drawn in the real world, allowing
you to see in 3d if and where there is a
mismatch.
4. you can easily find books you are looking
for in the library.
5. you can find out where restaurants are in
a city...
6. you can pimp theater / musical / opera /
pop concerts with (immersive) AR decor.
7. you can arrange virtual furniture or curtains
from the ikeA catalog and see how
they look in your home.
8. maintenance of complex devices will
become easier, e.g. you can virtually see
where the paper in the copier is jammed.
9. if you enter a restaurant or the hardware
store, a virtual avatar can show you the
place to find that special bolt or table.
showing the seRRA Room
in museum BoijmAns VAn
Beuningen duRing the
exhiBition sgRAffito in 3d
Picture: joAchim RotteVeel
26 27

28
RE-INTRoDUCING MoSQUIToS
MAARTEN LAMERS
ARoUND 2004, MY YoUNGER BRoTHER VALENTIJN INTRoDUCED ME
To THE FASCINATING WoRLD oF AUGMENTED REALITY. HE WAS A
MoBILE PHoNE SALESMAN AT THE TIME, AND SIEMENS HAD JUST
LAUNCHED THEIR FIRST “SMARTPHoNE”, THE BULKY SIEMENS SX1.
THIS PHoNE WAS QUITE MARVELoUS, WE THoUGHT – IT RAN THE
SYMBIAN oPERATING SYSTEM, HAD A BUILT-IN CAMERA, AND CAME
WITH… THREE GAMES.
one of these games was mozzies, a.k.a Virtual
mosquito hunt, which apparently won some
2003 Best mobile game Award and my brother
was eager to show it to me in the store where
he worked at that time. i was immediately
hooked… mozzies lets you kill virtual mosquitos
that fly around superimposed over the
live camera feed. By physically moving the
phone you could chase after the mosquitos
when they attempted to fly off the phone’s
display. those are all the ingredients for
Augmented Reality in my personal opinion:
something that interacts with my perception
and manipulation of the world around me, at
that location, at that time. And mozzies did
exactly that.
now almost eight years later, not much
has changed. whenever people around me
speak of AR, because they got tired of saying
“Augmented Reality”, they still refer to bulky
equipment (even bulkier than the siemens
sx1!) that projects stuff over a live camera
feed and lets you interact with whatever
that “stuff” is. in mozzies it was pesky little
mosquitos -- nowadays it is anything from
restaurant information to crime scene data.
But nothing really changed, right?
Right! technology became more advanced,
so we no longer need to hold the phone in
our hand, but get to wear it strapped to our
skull in the form of goggles. But the idea is
unchanged; you look at fake stuff in the real
world and physically move around to deal
with it. you still don’t get the tactile sensation
of swatting a mosquito or collecting
“virtually heavy” information. you still don’t
even hear the mosquito flying around you…
it’s time to focus on those matters also, in my
opinion. let’s take up the challenge and make
AR more than visual, exploring interaction
models for other senses. let’s enjoy the full
experience of seeing, hearing, and particularly
swatting mosquitos, but without the
itchy bites.
29

LIEVEN VAN
VELTHoVEN —
THE RACING STAR
“IT AIN’T FUN IF IT AIN’T REAL TIME”
BY HANNA SCHRAFFENBERGER
WHEN I ENTER LIEVEN VAN
VELTHoVEN’S RooM, THE PEoPLE
FRoM THE EFTELING HAVE JUST
LEFT. THEY ARE INTERESTED IN HIS
‘VIRTUAL GRoWTH’ INSTALLATIoN.
AND THEY ARE NoT THE oNLY oNES
INTERESTED IN LIEVEN’S WoRK. IN
THE LAST YEAR, HE HAS WoN THE
JURY AWARD FoR BEST NEW MEDIA
PRoDUCTIoN 2011 oF THE INTER-
NATIoNAL CINEKID YoUTH MEDIA
FESTIVAL AS WELL AS THE DUTCH
GAME AWARD 2011 FoR THE BEST
STUDENT GAME. THE WINNING
MIXED REALITY GAME ‘RooM RAC-
ERS’ HAS BEEN SHoWN AT THE
DISCoVERY FESTIVAL, MEDIAMATIC,
THE STRP FESTIVAL AND THE ZKM IN
KARLSRUHE. HIS VIRTUAL GRoWTH
INSTALLATIoN HAS EMBELLISHED
THE STREETS oF AMSTERDAM AT
NIGHT. NoW, HE IS GoING To SHoW
RooM RACERS To ME, IN HIS LIVING
RooM — WHERE IT ALL STARTED.
The room is packed with stuff and on first sight
it seems rather chaotic, with a lot of random
things laying on the floor. There are a few
plants, which probably don’t get enough light,
because Lieven likes the dark (that’s when his
projections look best). It is only when he turns
on the beamer, that I realize that his room is
actually not chaotic at all. The shoe, magnifying
class, video games, tape and stapler which cover
the floor are all part of the game.
“You create your own race game
tracks by placing real stuff on the
floor”
Lieven tells me. He hands me a controller and
soon we are racing the little projected cars
around the chocolate spread, marbles, a remote
control and a flash light. Trying not to crash the
car into a belt, I tell him what I remember about
when I first met him a few years ago at a Media
Technology course at Leiden University. Back
then, he was programming a virtual bird, which
would fly from one room to another, preferring
the room in which it was quiet. Loud and sudden
sounds would scare the bird away into another
room. The course for which he developed it was
called sound space interaction, and his installation
was solely based on sound. I ask him
whether the virtual bird was his first contact
with Augmented Reality. Lieven laughs.
“It’s interesting that you call it
AR, as it only uses sound!”
Indeed, most of Lieven’s work is based on
interactive projections and plays with visual
augmentations of our real environment. But like
the bird, all of them are interactive and work
in real-time. Looking back, the bird was not his
first AR work.
“My first encounter with AR was
during our first Media Technology
course — a visit to the Ars Electroncia
festival in 2007 — where
I saw Pablo Valbuena’s Augmented
Sculpture. It was amazing. I was
asking myself, can I do something
like this but interactive instead?”
Armed with a bachelor in technical computer
science from TU Delft and the new found possibility
to bring in his own curiosity and ideas at
the Media Technology Master program at Leiden
University, he set out to build his own interactive
projection based works.
30 31

Room RAceRs
Up to four players race their virtual cars around real objects
which are lying on the floor. Players can drop in or out of the
game at any time. Everything you can find can be placed on
the floor to change the route.
Room Racers makes use of projection-based mixed reality.
The structure of the floor is analysed in real-time using a
modified camera and self-written software. Virtual cars are
projected onto the real environment and interact with the
detected objects that are lying on the floor.
The game has won the Jury Award for Best New Media Produc-
tion 2011 of the international Cinekid Youth Media Festival,
and the Dutch Game Award 2011 for Best Student Game. Room
Racers shas been shown at several international media festivals.
You can play Room Racers at the 'Car Culture' exposition
at the Lentos Kunstmuseum in Linz, Austria until 4th of July
2012.
Picture: lieVen VAn VelthoVen, Room RAceRs At Zkm | centeR foR ARts And mediA
in kARlsRuhe, geRmAny on june 19th, 2011
32
33

“The first time, I experimented
with the combination of the real
and the virtual myself was in a
piece called shadow creatures
which I made with Lisa Dalhuijsen
during our first semester in 2007.”
More interactive projections followed in the
next semester and in 2008, the idea for Room
Racers was born. A first prototype was build in
a week: a projected car bumping into real world
things. After that followed months and months
of optimizations. Everything is done by Lieven
himself, mostly at night in front of the computer.
“My projects are never really
finished, they are always work in
progress, but if something works
fine in my room, it’s time to take
it out in the world.”
After having friends over and playing with the
cars until six o’clock in the morning, Lieven
knows it’s time to steer the cars out of his room
and show them to the outside world.
“I wanted to present Room Racers
but I didn’t know anyone, and
no one knew me. There was no
network I was part of.”
Uninhibited by this, Lieven took the initiative
and asked the Discovery Festival if they were
interested in his work. Luckily, they were — and
showed two of his interactive games at the Discovery
Festival 2010. After the festival requests
started coming and the cars kept rolling. When
I ask him about this continuing success he is
divided:
“It’s fun, but it takes a lot of time
— I have not been able to program
as much as I used to.”
His success does surprise him and he especially
did not expect the attention it gets in an art
context.
“I knew it was fun. That became
clear when I had friends over and
we played with it all night. But I
did not expect the awards. And I
did not expect it to be relevant
in the art scene. I do not think
it’s art, it’s just a game. I don’t
consider myself an artist. I am a
developer and I like to do interactive
projections. Room Racers is
my least arty project, nevertheless
it got a lot of response in the
art context.”
A piece which he actually considers more of an
artwork is Virtual Growth: a mobile installation
which projects autonomous growing structures
onto any environment you place it in, be it
buildings, people or nature.
“For me AR has to take place in
the real world. I don’t like screens.
I want to get away from them. I
have always been interested in
other ways of interacting with
computers, without mice, without
screens. There is a lot of screen
based AR, but for me AR is really
about projecting into the real
world. Put it in the real world,
identify real world objects, do it in
real-time, thats my philosophy. It
ain’t fun if it ain’t real-time. One
day, I want to go through a city
with a van and do projections on
buildings, trees, people and whatever
else I pass.”
For now, he is bound to a bike but that does
not stop him. Virtual Growth works fast and
stable, even on a bike. That has been witnessed
in Amsterdam, where the audiovisual bicycle
project ‘Volle Band’ put beamers on bikes and
invented Lieven to augmented the city with his
mobile installation. People who experienced
Virtual Growth on his journeys around Amsterdam,
at festivals and parties, are enthusiastic
about his (‘smashing!’) entertainment-art. As the
virtual structure grows, the audience members
not only start to interact with the piece but also
with each other.
“They put themselves in front
of the projector, have it projecting
onto themselves and pass on
the projection to other people
by touching them. I don’t explain
anything. I believe in simple
ideas, not complicated concepts.
The piece has to speak for itself.
If people try it, immediately get
it, enjoy it and tell other people
about it, it works!”
Virtual Growth works, that becomes clear from
the many happy smiling faces the projection
grows upon. And that’s also what counts for
Lieven.
“At first it was hard, I didn’t get
paid for doing these projects. But
when people see them and are enthusiastic,
that makes me happy.
If I see people enjoying my work,
and playing with it, that’s what
really counts.”
I wonder where he gets the energy to work that
much alongside being a student. He tells me,
what drives him, is that he enjoys it. He likes to
spend the evenings with the programming language
C#. But the fact that he enjoys working on
his ideas, does not only keep him motivated but
also has caused him to postpone a few courses
at university. While talking, he smokes his
cigarette and takes the ashtray from the floor.
With the road no longer blocked by it, the cars
take a different route now. Lieven might take a
different route soon as well. I ask him, if he will
still be working from his living room, realizing
his own ideas, once he has graduated.
“It’s actually funny. It all started
to fill my portfolio in order to get
a cool job. I wanted to have some
things to show besides a diploma.
That’s why I started realizing my
ideas. It got out of control and
soon I was realizing one idea after
the other. And maybe, I’ll just
continue doing it. But also, there
are quite some companies and
jobs I’d enjoy working for. First
I have to graduate anyway.”
If I have learned anything about Lieven and his
work, I am sure his graduation project will be
placed in the real world and work in in realtime.
More than that, it will be fun. It ain’t
Lieven, if it ain’t’ fun.
name: lieven van Velthoven
Born: 1984
study: media technology msc,
leiden university
Background: computer science,
tu delft
selected AR works: Room Racers,
Virtual growth
watch: http://www.youtube.com/
user/lievenvv
34 35

HoW DID WE Do IT:
ADDING VIRTUAL SCULPTURES
AT THE KRöLLER-MüLLER MUSEUM
By Wim van Eck
ALWAYS WANTED To CREATE YoUR oWN AUGMENTED REALITY PRo-
JECTS BUT NEVER KNEW HoW? DoN’T WoRRY, AR[T] IS GoING To
HELP YoU! HoWEVER, THERE ARE MANY HURDLES To TAKE WHEN
REALIZING AN AUGMENTED REALITY PRoJECT. IDEALLY YoU SHoULD
BE A SKILLFUL 3D ANIMAToR To CREATE YoUR oWN VIRTUAL oB-
JECTS, AND A GREAT PRoGRAMMER To MAKE THE PRoJECT TECHNI-
CALLY WoRK. PRoVIDING YoU DoN’T JUST WANT To MAKE A FANCY
TECH-DEMo, YoU ALSo NEED To CoME UP WITH A GREAT CoNCEPT!
My name is Wim van Eck and I work at the AR
Lab, based at the Royal Academy of Art. one of
my tasks is to help art-students realize their Augmented
Reality projects. These students have
great concepts, but often lack experience in 3d
animation and programming. Logically I should
tell them to follow animation and programming
courses, but since the average deadline for their
projects is counted in weeks instead of months
or years there is seldom time for that... In the
coming issues of AR[t] I will explain how the AR
Lab helps students to realize their projects and
how we try to overcome technical boundaries,
showing actual projects we worked on by example.
Since this is the first issue of our magazine
I will give a short overview of recommendable
programs for Augmented Reality development.
We will start with 3d animation programs, which
we need to create our 3d models. There are
many 3d animation packages, the more well
known ones include 3ds Max, Maya, Cinema 4d,
Softimage, Lightwave, Modo and the open source
Blender (www.blender.org). These are all great
programs, however at the AR Lab we mostly use
Cinema 4d (image 1) since it is very user friendly
and because of that easier to learn. It is a shame
that the free Blender still has a steep learning
curve since it is otherwise an excellent program.
You can download a demo of Cinema 4d at
http://www.maxon.net/downloads/demo-version.html,
these are some good tutorial sites to
get you started:
http://www.cineversity.com
http://www.c4dcafe.com
http://greyscalegorilla.com
image 1
image 2 image 3 | Picture by klaas A. mulder image 4
In case you don’t want to create your own 3d
models you can also download them from various
websites. Turbosquid (http://www.turbosquid.com),
for example, offers good quality but
often at a high price, while free sites such as
Artist-3d (http://artist-3d.com) have a more varied
quality. When a 3d model is not constructed
properly it might give problems when you import
it or visualize it. In coming issues of AR[t] we
will talk more about optimizing 3d models for
Augmented Reality usage. To actually add these
3d models to the real world you need Augmented
Reality software. Again there are many
options, with new software being added continuously.
Probably the easiest to use software is
BuildAR (http://www.buildar.co.nz) which is
available for Windows and oSX. It is easy to
import 3d models, video and sound and there is
a demo available. There are excellent tutorials
on their site to get you started. In case you want
to develop for ioS or Android the free Junaio
(http://www.junaio.com) is a good option. Their
online GLUE application is easy to use, though
their preferred .m2d format for 3d models is
not the most common. In my opinion the most
powerful Augmented Reality software right now
is Vuforia (https://developer.qualcomm.com/
develop/mobile-technologies/Augmented-reality)
in combination with the excellent game-engine
Unity (www.unity3d.com). This combination
offers high-quality visuals with easy to script
interaction on ioS and Android devices
Sweet summer nights
at the Kröller-Müller
Museum.
As mentioned before in the introduction we
will show the workflow of AR Lab projects with
these ‘How did we do it’ articles. In 2009 the AR
Lab was invited by the Kröller-Müller Museum to
present during the ‘Sweet Summer Nights’, an
evening full of cultural activities in the famous
sculpture garden of the museum. We were asked
to develop an Augmented Reality installation
aimed at the whole family and found a diverse
group of students to work on the project. Now
the most important part of the project started,
brainstorming!
our location in the sculpture garden was in-
between two sculptures, ‘Man and woman’, a
stone sculpture of a couple by Eugène Dodeigne
(image 2) and ‘Igloo di pietra’, a dome shaped
sculpture by Mario Merz (image 3). We decided
to read more about these works, and learned
that Dodeigne had originally intended to create
two couples instead of one, placed together in a
wild natural environment. We decided to virtually
add the second couple and also add a more
wild environment, just as Dodeigne initially had
in mind. To be able to see these additions we
placed a screen which can rotate 360 degrees
between the two sculptures (image 4).
36 37

A webcam was placed on top of the screen,
and a laptop running ARToolkit (http://www.
hitl.washington.edu/artoolkit) was mounted
on the back of the screen. A large marker was
placed near the sculpture as a reference point
for ARToolkit.
Now it was time to create the 3d models of the
extra couple and environment. The students
working on this part of the project didn’t have
much experience with 3d animation, and there
wasn’t much time to teach them, so manually
modeling the sculptures would be a difficult task.
Soon options such as 3d scanning the sculpture
were opted, but it still needs quite some skill
to actually prepare a 3d scan for Augmented
Reality usage. We will talk more about that in
a coming issue of this magazine.
But when we look carefully at our setup (image
5) we can draw some interesting conclusions.
our screen is immobile, we will always see our
added 3d model from the same angle. So since
we will never be able to see the back of the 3d
model there is no need to actually model this
part. This is a common practice while making 3d
models, you can compare it with set construction
for Hollywood movies where they also only
38
image 5
actually build what the camera will see. This will
already save us quite some work. We can also
see the screen is positioned quite far away from
the sculpture, and when an object is viewed
from a distance it will optically lose its depth.
When you are one meter away from an object
and take one step aside you will see the side of
the object, but if the same object is a hundred
meter away you will hardly see a change in perspective
when changing your position (see image
6). From that distance people will hardly see the
difference between an actual 3d model and a
plain 2d image. This means we could actually use
photographs or drawings instead of a complex 3d
model, making the whole process easier again.
We decided to follow this route.
image 6
image 7
image 8
image 9
image 10
image 11
=
39

original photograph by klaas A. mulder
image 12
To be able to place the photograph of the
sculpture in our 3d scene we have to assign
it to a placeholder, a single polygon, image 7
shows how this could look.
This actually looks quite awful, we see the
statue but also all the white around it from the
image. To solve this we need to make usage of
something called an alpha channel, an option
you can find in every 3d animation package
(image 8 shows where it is located in the material
editor of Cinema 4d). An alpha channel is
a grayscale image which declares which parts
of an image are visible, white is opaque, black
is transparent. Detailed tutorials about alpha
channels are easily found on the internet.
As you can see this looks much better (image 9).
We followed the same procedure for the second
statue and the grass (image 10), using many
separate polygons to create enough randomness
for the grass. As long as you see these models
from the right angle they look quite realistic
(image 11). In this case this 2.5d approach probably
gives even better results than a ‘normal’ 3d
model, and it is much easier to create. Another
advantage is that the 2.5d approach is very easy
to compute since it uses few polygons, so you
don’t need a very powerful computer to run it
or you can have many models on screen at the
same time. Image 12 shows the final setup.
For the iglo sculpture by Mario Merz we used
a similar approach. A graphic design student
imagined what could be living inside the iglo,
and started drawing a variety of plants and
creatures. Using the same 2.5d approach as
described before we used these drawings and
placed them around the iglo, and an animation
was shown of a plant growing out of the iglo
(image 12).
The Lab collaborated in this project with students from different departments
of the KABK: Ferenc Molnar, Mit Koevoets, Jing Foon Yu, Marcel
Kerkmans and Alrik Stelling. The AR Lab team consisted of: Yolande
Kolstee, Wim van Eck, Melissa Coleman en Pawel Pokutycki, supported by
Martin Sjardijn and Joachim Rotteveel.
We can conclude that it is good practice to
analyze your scene before you start making your
3d models. You don’t always need to model all
the detail, and using photographs or drawings
can be a very good alternative. The next issue
of AR[t] will feature a new ‘How did we do it’, in
case you have any questions you can contact me
at w.vaneck@kabk.nl
40 41

Pixels wAnt to Be fReed!
intRoducing Augmented ReAlity
enABling hARdwARe technologies
By jouke VeRlinden
1. introduction
From the early head-up display in the movie
“Robocop” to the present, Augmented Reality
(AR) has evolved to a manageable ICT environment
that must be considered by product designers
of the 21st century.
Instead of focusing on a variety of applications
and software solutions, this article will discuss
the essential hardware of Augmented Reality
(AR): display techniques and tracking techniques.
We argue that these two fields differentiate AR
from regular human-user interfaces and tuning
these is essential in realizing an AR experience.
As often, there is a vast body of knowledge behind
each of the principles discussed below,
hence a large variety of literature references is
given.
Furthermore, the first author of this article
found it important to elude his own preferences
and experiences throughout this discussion.
We hope that this material strikes a chord and
makes you consider employing AR in your designs.
After all, why should digital information
always be confined to a dull, rectangular screen?
42 43

2. display technologies
to categorise AR display technologies, two
important characteristics should be identified:
imaging generation principle and physical
layout.
generic AR technology surveys describe a
large variety of display technologies that support
imaging generation (Azuma, 1997; Azuma
et al., 2001); these principles can be categorised
into:
1. Video-mixing. A camera is mounted some-
where on the product; computer graphics
are combined with captured video frames
in real time. the result is displayed on an
oblique surface, for example, an immersive
head-mounted display (hmd).
2. see-through: Augmentation by this
principle typically employs half-silvered
mirrors to superimpose computer graphics
onto the user’s view, as found in head-up
displays of modern fighter jets.
3. Projector-based systems: one or more
projectors cast digital imagery directly
on the physical environment.
As Raskar and Bimber (2004, p.72) argued, an
important consideration in deploying an Augmented
system is the physical layout of the
image generation. for each imaging generation
principle mentioned above, the imaging
display can be arranged between user and
physical object in three distinct ways:
a) head-attached, which presents digital
images directly in front of the viewer’s
eyes, establishing a personal information
display.
b) hand-held, carried by a user and does not
cover the whole field of view
c) spatial, which is fixed to the environment.
the resulting imaging and arrangement combinations
are summarised in table 1.
1.Video-mixing 2. see-through 3. Projection-based
A. head-attached head-mounted display (hmd)
B. hand-held handheld devices
c. spatial embedded display
Table 1. Image generation principles for Augmented Reality
see-through boards spatial projection-based
when the AR image generation and layout principles are combined, the following collection of
display technologies are identified: hmd, handheld devices, embedded screens, see-through
boards and spatial projection-based AR. these are briefly discussed in the following sections.
44
2.1 head-mounted display
Head-attached systems refer to HMD solutions,
which can employ either of the three image
generation technologies. Even the first headmounted
displays developed by virtue of the
Virtual Reality already considered a see-through
system with half-silvered mirrors to merge
virtual line drawings with the physical environment
(Sutherland, 1967). Since then, the variety
of head-attached imaging systems has been
expanded and encompasses all three principles
for AR: video-mixing, see-through and direct
projection on the physical world (Azuma et al.,
2001). A benefit of this approach is its handsfree
nature. Secondly, it offers personalised
content, enabling each user to have a private
view of the scene with customised and sensitive
data that das not have to be shared. For
most applications, HMDs have been considered
inadequate, both in the case of see-through and
video-mixing imaging. According to Klinker et al.
(2002), HMDs introduce a large barrier between
the user and the object and their resolution is
insufficient for IAP — typically 800 × 600 pixels
for the complete field of view (rendering the
user “legally blind”by American standards).
Similar reasoning was found in Bochenek et al.
(2001), in which both the objective and subjective
assessment of HMDs were less than those of
hand-held or spatial imaging devices. However,
new developments (specifically high-resolution
oLED displays) show promising new devices, specifically
for the professional market (Carl Zeiss)
and enterntainment (Sony), see figure right.
2.2 handheld display
Hand-held video-mixing solutions are based on
smartphones, PDAs or other mobile devices
equipped with a screen and camera. With the
advent of powerful mobile electronics, handheld
Figure 1. Recent heAd mounted disPlAys (ABoVe: kABk the
hAgue And undeR: cARl Zeiss).
Augmented Reality technologies are emerging.
By employing built-in cameras on smartphones
or PDAs, video mixing is enabled while concurrent
use is being supported by communication
45

figure 2. the VesP´R deVice foR undeRgRound infRAstRuctuRe VisuAliZAtion (schAll et Al., 2008).
through wireless networks (Schmalstieg and
Wagner, 2008). The resulting device acts as a
hand-held window of a mixed reality. An example
of such a solution is shown in Figure 2, which
is a combination of an Ultra Mobile Personal
Computer (UMPC), a Global Positioning System
‘such systems
are found in
each modern
smartphone’
(GPS) antenna for global position tracking, a
camera for local position and orientation sensing
along with video mixing. As of today, such sys-
GPS Antenna
Camera + IMU
Joystick Handles
UMPC
tems are found in each modern smartphone,
and apps such as Layar (www.layar.com) and
Junaio (www.junaio.com) offer such functions
for free to the user — allowing different layers of
content to the user (often social-media based).
The advantage of using a video-mixing approach
is that the lag times in processing are less influential
than with the see-through or projector-based
systems — the live video feed is also delayed and,
thus, establishes a consistent combined image.
This hand-held solution works well for occasional,
mobile use. Long-term use can cause strain in the
arms. The challenges in employing this principle
are the limited screen coverage/resolution (typically
with a 4-in diameter and a resolution of 320
× 240 pixels). Furthermore, memory, processing
power and graphics processing is limited to rendering
relatively simple 3D scenes, although these
capabilities are rapidly improving by the upcoming
dual-core and quad-core mobile CPUs.
2.3 embedded display
Another AR display option is to include a number
of small LCD screens in the observed object in
order to display the virtual elements directly on
the physical object. Although arguably an augmentation
solution, embedded screens do add
digital information on product surfaces.
This practice is found in the later stages of prototyping
mobile phones and similar information
appliances. Such screens typically have a similar
resolution as that of PDAs and mobile phones,
which is QVGA: 320 × 240 pixels. Such devices
are connected to a workstation by a specialised
cable, which can be omitted if autonomously
components are used, such as a smartphone.
Regular embedded screens can only be used on
planar surfaces and their size is limited while
their weight impedes larger use. With the advent
of novel, flexible e-Paper and organic Light-
Emitting Diode (oLED) technologies, it might
be possible to cover a part of a physical model
figure 3. imPRession of the luminex mAteRiAl
with such screens. To our knowledge, no such
systems have been developed or commercialised
so far. Although it does not support changing
light effects, the Luminex material approximates
this by using an LED/fibreglass based fabric (see
Figure 4). A Dutch company recently presented
a fully interactive light-emitting fabric based on
integrated RGB LEDs labelled ‘lumalive’. These
initiatives can manifest as new ways to support
prototyping scenarios that require a high local
resolution and complete unobstructedness. However,
the fit to the underlying geometry remains
a challenge, as well as embedding the associated
control electronics/wiring. An elegant solution
to the second challenge was given by (Saakes et
al 2010) entitled “the slow display: by temporarily
changing the color of photochromatic paint
properties by UV laser projection. This effect
lasts for a couple of minutes and demonstrates
how fashion and AR could meet.
46 47

2.4 see-through board
See-through boards vary in size between desk-
top and hand-held versions. The Augmented
engineering system (Bimber et al., 2001) and
the AR extension of the haptic sculpting project
(Bordegoni and Covarrubias, 2007) are examples
of the use of see-through technologies, which
typically employ a half-silvered mirror to mix
virtual models with a physical object (Figure
4). Similar to the Pepper’s ghost phenomenon,
standard stereoscopic Virtual Reality (VR) workbench
systems such as the Barco Baron are used
to project the virtual information. In addition
to the need to wear shutter glasses to view stereoscopic
graphics, head tracking is required to
align the virtual image between the object and
the viewer. An advantage of this approach is
that digital images are not occluded by the users’
hand or environment and that graphics can
be displayed outside the physical object (i.e.,
to display the environment or annotations and
tools). Furthermore, the user does not have to
wear heavy equipment and the resolution of the
projection can be extremely high — enabling a
2.5 spatial projection-based displays
This technique is also known as Shader Lamps
by (Raskar et al., 2001) and was extended in
(Raskar&Bimber, 2004) to a variety of imaging solutions,
including projections on irregular surface
textures and combinations of projections with
(static) holograms. In the field of advertising and
performance arts, this technique recently gained
popularity labelled as Projection Mapping: to
project on buildings, cars or other large objects,
replacing traditional screens as display means, cf.
Figure 5. In such cases, theatre projector systems
are used that are prohibitively expensive (>30.000
euros). The principle of spatial projection-based
technologies is shown in Figure 6. Casting an image
to a physical object is considered comple-
48
compelling display system for exhibits and trade
fairs. However, see-through boards obstruct user
interaction with the physical object. Multiple
viewers cannot share the same device, although
a limited solution is offered by the virtual
showcase by establishing a faceted and curved
mirroring surface (Bimber, 2002).
Figure 4. the Augmented engineeRing see-thRough
disPlAy (BimBeR et Al., 2001).
mentary to constructing a perspective image
of a virtual object by a pinhole camera. If the
physical object is of the same geometry as the
virtual object, a straightforward 3D perspective
transformation (described by a 4 × 4 matrix)
is sufficient to predistort the digital image. To
obtain this transformation, it suffices to indicate
6 corresponding points in the physical world
and virtual world: an algorithm entitled Linear
Camera Calibration can then be applied (see
Appendix). If the physical and virtual shapes differ,
the projection is viewpoint-dependent and
the head position needs to be tracked. Important
projector characteristics involve weight
and size versus the power (in lumens) of the
figure 5. two PRojections on A chuRch chAPel in utRecht (hoeBen, 2010).
49

figure 6. PRojection-BAsed disPlAy PRinciPle
(AdAPted fRom (RAskAR And low, 2001)), on the
Right the dynAmic shAdeR lAmPs demonstRAtion
50
(BAndyoPAdhyAy et Al., 2001)).
projector. There are initiatives to employ LED
lasers for direct holographic projection, which
also decreases power consumption compared to
traditional video projectors and ensures that the
projection is always in focus without requiring
optics (Eisenberg, 2004). Both fixed and handheld
spatial projection-based systems have been
demonstrated. At present, hand-held projectors
measure 10 × 5 × 2 cm and weigh 150 g, including
the processing unit and battery. However,
the light output is little (15–45 lumens).
The advantage of spatial projection-based tech-
nologies is that they support the perception of
all visual and tactile/haptic depth cues without
the need for shutter glasses or HMDs. Furthermore,
the display can be shared by multiple
co-located users. It requires less expensive
equipment, which are often already available at
design studios. Challenges to projector-based AR
approaches include optics and occlusion. First,
only a limited field of view and focus depth can
be achieved. To reduce these problems, multiple
video projectors can be used. An alternative
so lution is to employ a portable projector, as
proposed in the iLamps and the I/o Pad concepts
(Raskar et al., 2003) (Verlinden et al., 2008).
other issues include occlusion and shadows,
which are cast on the surface by the user or
other parts of the system. Projection on nonconvex
geometries depends on the granularity
and orientation of the projector. The perceived
quality is sensitive to projection errors (also
known as registration errors), especially projection
overshoot (Verlinden et al., 2003b).
A solution for this problem is either to include an
offset (dilatation) of the physical model or introduce
pixel masking in the rendering pipeline. As
projectors are now being embedded in consumer
cameras and smartphones, we are expecting this
type of augmentation in the years to come.
3. input technologies
In order to merge the digital and physical, position
and orientation tracking of the physical
components is required. Here, we will discuss
two different types of input technologies: tracking
and event sensing. Furthermore, we will
briefly discuss other input modalities.
3.1 Position tracking
Welch and Foxlin (2002) presented a comprehensive
overview of the tracking principles
that are currently available. In the ideal case,
the measurement should be as unobtrusive and
invisible as possible while still offering accurate
and rapid data. They concluded that there is
currently no ideal solution (‘silver bullet’) for
position tracking in general, but some respectable
alternatives are available. Table 2 summarises
the most important characteristics of
these tracking methods for Augmented Reality
purposes. The data have been gathered from
commercially available equipment (the Ascension
Flock of Birds, ARToolkit, optotrack,
tracking type
size of
tracker
(mm)
magnetic 16x16x16 2
optical
passive
optical
active
80x80x0.01 >10
10x10x5 >10
ultrasound 20x20x10 1
mechanical
linkage
laser
scanning
defined by
working
envelope
typical
number of
trackers
table 2. summARy of tRAcking technologies.
1
none infinite
Logitech 3D Tracker, Microscribe and Minolta VI-
900). All these should be considered for object
tracking in Augmented prototyping scenarios.
There are significant differences in the tracker/
marker size, action radius and accuracy. As
the physical model might consist of a number
of parts or a global shape and some additional
components (e.g., buttons), the number of items
to be tracked is also of importance. For simple
tracking scenarios, either magnetic or passive
optical technologies are often used.
In some experiments we found out that a projector
could not be equipped with a standard Flock
of Birds 3D magnetic tracker due to interference.
other tracking techniques should be used
for this paradigm. For example, the ARToolkit
employs complex patterns and a regular webcamera
to determine the position, orientation
and identification of the marker. This is done by
measuring the size, 2D position and perspective
distortion of a known rectangular marker, cf.
Figure 7 (Kato and Billinghurst, 1999).
Passive markers enable a relatively untethered
system, as no wiring is necessary. The optical
markers are obtrusive when markers are visible
to the user while handling the object. Although
computationally intensive, marker-less optical
Action
radius/
accuracy
1.5 m
(1 mm)
3 m
(1 mm)
3 m
(0.5 mm)
1 m
(3 mm)
0.7 m
(0.1 mm)
2 m
( 0.2mm)
dof issues
6 Ferro-magnetic interference
6 line of sight
3 line of sight, wired connections
6 line of sight
5
6
limited degrees of freedom,
inertia
line of sight, frequency, object
recognition
51

figure 7. woRkflow of the ARtoolkit oPticAl tRAcking AlgoRithm,
http://www.hitl.washington.edu/artoolkit/documentation/userarwork.html
tracking has been proposed (Prince et al.,2002).
The employment of Laser-Based tracking systems
is demonstrated by the illuminating Clay
system by Piper et al. (2002): a slab of Plasticine
acts as an interactive surface — the user
influences a 3D simulation by sculpting the clay,
while the simulation results are projected on the
surface. A laser-based Minolta Vivid 3D scanner
is employed to continuously scan the clay
surface. In the article, this principle was applied
to geodesic analysis, yet it can be adapted to
design applications, e.g., the sculpting of car
bodies. This method has a number of challenges
when used as a real-time tracking means, including
the recognition of objects and their posture.
However, with the emergence of depth cameras
for gaming such as the Kinect (Microsoft), similar
systems are now being devised with a very small
technological threshold.
In particular cases, a global measuring system is
combined with a different local tracking principle
to increase the level of detail, for example, to
track the position and arrangement of buttons on
figure 8. illuminAting clAy system with A PRojectoR/lAseR scAnneR (PiPeR et Al., 2002).
the object’s surface. Such local positioning systems
might have less advanced technical requirements;
for example, the sampling frequency can
be decreased to only once a minute. one local
tracking system is based on magnetic resonance,
as used in digital drawing tablets. The Sensetable
demonstrates this by equipping an altered commercial
digital drawing tablet with custom-made
wireless interaction devices (Patten et al., 2001).
The Senseboard (Jacob et al., 2002) has similar
functions and an intricate grid of RFID receivers
to determine the (2D) location of an RFID tag on
a board. In practice, these systems rely on a rigid
tracking table, but it is possible to extend this to
a flexible sensing grid. A different technology was
proposed by Hudson (2004) to use LED pixels as
light emitters and sensors. By operating one pixel
as a sensor whilst its neighbours are illuminated,
it is possible to detect light reflected from a
fingertip close to the surface. This principle could
be applied to embedded displays, as mentioned
in Section 2.3.
3.2 event sensing
Apart from location and orientation tracking,
Augmented prototyping applications require
inter action with parts of the physical object,
for example, to mimic the interaction with the
artefact. This interaction differs per AR scenario,
so a variety of events should be sensed
to cater to these applications.
Physical sensors
The employment of traditional sensors labelled
‘physical widgets’ (phidgets) has been studied
extensively in the Computer-Human Interface
(CHI) community. Greenberg and Fitchett (2001)
introduced a simple electronics hardware and
software library to interface PCs with sensors
(and actuators) that can be used to discern
user interaction. The sensors include switches,
sliders, rotation knobs and sensors to measure
force, touch and light. More elaborate components
like a mini joystick, Infrared (IR) motion
sensor, air pressure and temperature sensor are
commercially available. Similar initiatives are
iStuff (Ballagas et al., 2003), which also hosts a
number of wireless connections to sensors. Some
systems embed switches with short-range wireless
connections, for example, the Switcheroo
and Calder systems (Avrahami and Hudson, 2002;
Lee et al., 2004) (cf. Figure 9). This allows a
greater freedom in modifying the location of the
interactive components while prototyping. The
Switcheroo system uses custom-made RFID tags.
A receiver antenna has to be located nearby
(within a 10-cm distance), so the movement envelope
is rather small, while the physical model
is wired to a workstation. The Calder toolkit
(Lee et al., 2004) uses a capacitive coupling
technique that has a smaller range (6 cm with
small antennae), but is able to receive and transmit
for long periods on a small 12 mm coin cell.
other active wireless technologies would draw
more power, leading to a system that would
only fit a few hours. Although the costs for this
system have not been specified, only standard
electronics components are required to build
such a receiver.
hand tracking
Instead of attaching sensors to the physical
environment, fingertip and hand tracking
technologies can also be used to generate user
events. Embedded skins represent a type of
interactive surface technology that allows the
accurate measurement of touch on the object’s
surface (Paradiso et al., 2000). For example, the
Smartskin by Reikimoto (2002) consists of a flexible
grid of antennae. The proximity or touch of
human fingers changes the capacity locally in the
grid and establishes a multi-finger tracking cloth,
which can be wrapped around an object. Such a
solution could be combined with embedded displays,
as discussed in Section 2.3. Direct electric
52 53

figure 9. mockuP equiPPed with wiReless switches thAt cAn Be RelocAted to exPloRe usABility
(lee et Al., 2004).
contact can also be used to track user interac-
tion; the Paper Buttons concept (Pedersen et
al., 2000) embeds electronics on the objects and
equips the finger with a two-wire plug that supplies
power and allows bidirectional communication
with the embedded components when they
are touched. Magic Touch (Pedersen, 2001) uses
a similar wireless system; the user wears an RFID
reader on his or her finger and can interact by
touching the components, which have hidden
RFID tags. This method has been adapted to
Augmented Reality for design by Kanai et al.
(2007). optical tracking can be used for finger -
tip and hand tracking as well. A simple example
is the light widgets system (Fails and olsen,
2002) that traces skin colour and determines
finger/hand position by 2D blobs. The openNI
library enables hand and body tracking of depth
range cameras such as the Kinect (openNi.org).
A more elaborate example is the virtual drawing
tablet by Ukita and Kidode (2004); fingertips
are recognised on a rectangular sheet by a
head-mounted infrared camera. Traditional VR
gloves can also be used for this type of tracking
(Schäfer et al., 1997).
3.3 other input modalities
Speech and gesture recognition require consideration
in AR as well. In particular, pen-based
interaction would be a natural extension to the
expressiveness of today’s designer skills. oviatt
et al. (2000) offer an comprehensive overview of
the so-called Recognition-Based User Interfaces
(RUIs), including the issues and Human Factors
aspects of these modalities. Furthermore,
speech-based interaction can also be useful to
activate operations while the hands are used for
selection.
4. conclusions and further
reading
This article introduces two important hardware
systems for AR: displays and input technologies.
To superimpose virtual images onto physical
models, head mounted-displays (HMDs), seethrough
boards, projection-based techniques
and embedded displays have been employed.
An important observation is that HMDs, though
best known by the public, have serious limita-
display imaging principle Video Mixing
display arrangment
tions and constraints in terms of the field of
view and resolution and lend themselves to a
kind of isolation. For all display technologies,
the current challenges include an untethered
interface, the enhancement of graphics capabilities,
visual coverage of the display and improvement
of resolution. LED-based laser projection
and oLEDs are expected to play an important
role in the next generation of IAP devices
because this technology can be employed by
see-through or projection-based displays.
To interactively merge the digital and physical
parts of Augmented prototypes, position and
orientation tracking of the physical components
is needed, as well as additional user input
means. For global position tracking, a variety of
principles exist. optical tracking and scanning
suffer from the issues concerning line of sight
and occlusion. Magnetic, mechanical linkage and
ultrasound-based position trackers are obtrusive
and only a limited number of trackers can be
used concurrently.
The resulting palette of solutions is summarized
in Table 3 as a morphological chart. In devising a
solution for your AR system, you can use this as
a checklist or inspiration of display and input.
54 55
input technologies
Position
tracking
event
sensing
Headattached
Projectorbased
Handheld/
wearable
See-through
Spatial
Magnetic Passive
optical
Active 3D laser
markers markers scanning
Physical sensors Virtual
Wired
connection
Wireless Surface
tracking
Table 3. Morphological chart of AR enabling technologies.
3D
tracking
Ultrasound Mechanical

further reading
For those interested in research in this area,
the following publication means offer a range of
detailed solutions:
■ International Symposium on Mixed and
Augmented Reality (ISMAR) – ACM-sponsored
annual convention on AR, covering both specific
applications as emerging technologies.
accesible through http://dl.acm.org
■ Augmented Reality Times — a daily update
on demos and trends in commercial and academic
AR systems: http://artimes.rouli.net
■ Procams workshop — annual workshop on
projector-camera systems, coinciding with
the IEEE conference on Image Recognition
and Robot Vision. The resulting proceedings
are freely accessible at http://www.procams.
org
■ Raskar, R. and Bimber, o. (2004) Spatial
Augmented Reality, A.K. Peters, ISBN:
1568812302 – personal copy can be downloaded
for free at http://140.78.90.140/medien/ar/SpatialAR/download.php
■ BuildAR – download simple webcam-based
application that uses markers, http://www.
buildar.co.nz/buildar-free-version
Appendix: linear camera
calibration
This procedure has been published in (Raskar
and Bimber, 2004) to some degree, but is slightly
adapted to be more accessible for those with
less knowledge of the field of image processing.
C source code that implements this mathematical
procedure can be found in appendix A1 of
(Faugeras, 1993). It basically uses point correspondences
between original x,y,z coordinates
and their projected u,v, counterparts to resolve
internal and external camera parameters.
In general cases, 6 point correspondences are
sufficient (Faugeras 1993, Proposition 3.11).
Let I and E be the internal and external param-
56
eters of the projector, respectively. Then a point
P in 3D-space is transformed to:
p=[I·E] ·P (1)
where p is a point in the projector’s coordinate
system. If we decompose rotation and translation
components in this matrix transformation
we obtain:
p=[R t] ·P (2)
In which R is a 3x3 matrix corresponding to the
rotational components of the transformation and
t the 3x1 translation vector. Then we split the
rotation columns into row vectors R1, R2, and R3
of formula 3. Applying the perspective division
results in the following two formulae:
(3)
(4)
in which the 2D point pi is split into (ui,vi).
Given n measured point-point correspondences
(p ; P ); (i = 1::n), we obtain 2n equations:
i i
R 1 ·P i – u i ·R 3 ·P i + t x - u i ·t z = 0 (5)
R 2 ·P i – v i ·R 3 ·P i + t y - u i ·t z = 0 (6)
We can rewrite these 2n equations as a matrix
multiplication with a vector of 12 unknown
variables, comprising the original transformation
components R and t of formula 3. Due to measurement
errors, a solution is usually non-singular;
we wish to estimate this transformation with
a minimal estimation deviation. In the algorithm
presented at (Bimber & Raskar, 2004), the minimax
theorem is used to extract these based on
determining the singular values. In a straightforward
matter, internal and external transformations
I and E of formula 1 can be extracted from
the resulting transformation.
References
■ Avrahami, d. and hudson, s.e. (2002)
‘Forming interactivity: a tool for rapid prototyping
of physical interactive products’,
Proceedings of DIS ‘02, pp.141–146.
■ Azuma, R. (1997)
‘A survey of augmented reality’, Presence:
Teleoperators and Virtual Environments,
Vol. 6, No. 4, pp.355–385.
■ Azuma, R., Baillot, y., Behringer, R., feiner,
s., julier, s. and macintyre, B. (2001)
‘Recent advances in augmented reality’, IEEE
Computer Graphics and Applications, Vol. 21,
No. 6, pp.34–47.
■ Ballagas, R., Ringel, m., stone,
m. and Borchers, j. (2003)
‘iStuff: a physical user interface toolkit for
ubiquitous computing environments’, Proceedings
of CHI 2003, pp.537–544.
■ Bandyopadhyay, d., Raskar, R. and fuchs,
h. (2001)
‘Dynamic shader lamps: painting on movable
objects’, International Symposium on Augmented
Reality (ISMAR), pp.207–216.
■ Bimber, o. (2002)
‘Interactive rendering for projection-based
augmented reality displays’, PhD dissertation,
Darmstadt University of Technology.
■ Bimber, o., stork, A. and Branco, P. (2001)
‘Projection-based augmented engineering’,
Proceedings of International Conference on
Human-Computer Interaction (HCI’2001),
Vol. 1, pp.787–791.
■ Bochenek, g.m., Ragusa, j.m. and malone,
l.c. (2001)
‘Integrating virtual 3-D display systems into
product design reviews: some insights from
empirical testing’, Int. J. Technology Management,
Vol. 21, Nos. 3–4, pp.340–352.
■ Bordegoni, m. and covarrubias, m. (2007)
‘Augmented visualization system for a haptic
interface’, HCI International 2007 Poster.
■ eisenberg, A. (2004)
‘For your viewing pleasure, a projector in
your pocket’, New York Times, 4 November.
■ faugeras, o. (1993)
‘Three-Dimensional Computer Vision:
a Geometric Viewpoint’, MIT press.
■ fails, j.A. and olsen, d.R. (2002)
‘LightWidgets: interacting in everyday
spaces’, Proceedings of IUI ‘02, pp.63–69.
■ greenberg, s. and fitchett, c. (2001)
‘Phidgets: easy development of physical interfaces
through physical widgets’, Proceedings
of UIST ‘01, pp.209–218.
■ hoeben, A. (2010)
“Using a projected Trompe L’oeil to highlight
a church interior from the outside”, EVA
2010
■ hudson, s. (2004)
‘Using light emitting diode arrays as touchsensitive
input and output devices’, Proceedings
of the ACM Symposium on User Interface
Software and Technology, pp.287–290.
■ jacob, R.j., ishii, h., Pangaro, g. and Patten,
j. (2002)
‘A tangible interface for organizing information
using a grid’, Proceedings of CHI ‘02,
pp.339–346.
57

■ kanai, s., horiuchi, s., shiroma, y., yo-
koyama, A. and kikuta, y. (2007)
‘An integrated environment for testing and
assessing the usability of information appliances
using digital and physical mock-ups’,
Lecture Notes in Computer Science, Vol.
4563, pp.478–487.
■ kato, h. and Billinghurst, m. (1999)
‘Marker tracking and HMD calibration for a
video-based augmented reality conferencing
system’, Proceedings of International
Workshop on Augmented Reality (IWAR 99),
pp.85–94.
■ klinker, g., dutoit, A.h., Bauer, m., Bayer,
j., novak, V. and matzke, d. (2002)
‘Fata Morgana – a presentation system for
product design’, Proceedings of ISMAR ‘02,
pp.76–85.
■ oviatt, s.l., cohen, P.R., wu, l., Vergo,
j., duncan, l., suhm, B., Bers, j., et al.
(2000)
‘Designing the user interface for multimodal
speech and gesture applications: state-ofthe-art
systems and research directions’,
Human Computer Interaction, Vol. 15, No. 4,
pp.263–322.
■ Paradiso, j.A., hsiao, k., strickon, j.,
lifton, j. and Adler, A. (2000)
‘Sensor systems for interactive surfaces’,
IBM Systems Journal, Vol. 39, Nos. 3–4,
pp.892–914.
■ Patten, j., ishii, h., hines, j. and Pangaro,
g. (2001)
‘Sensetable: a wireless object tracking platform
for tangible user interfaces’, Proceedings
of CHI ‘01, pp.253–260.
■ Pedersen, e.R., sokoler, t. and nelson, l.
(2000)
‘PaperButtons: expanding a tangible user interface’,
Proceedings of DIS ’00, pp.216–223.
58
■ Pederson, t. (2001)
‘Magic touch: a simple object location tracking
system enabling the development of
physical- virtual artefacts in office environ-
ments’, Personal Ubiquitous Comput., Janu-
ary, Vol. 5, No. 1, pp.54–57.
■ Piper, B., Ratti, c. and ishii, h. (2002)
‘Illuminating clay: a 3-D tangible interface
for landscape analysis’, Proceedings of CHI
‘02, pp.355–362.
■ Prince, s.j., xu, k. and cheok, A.d. (2002)
‘Augmented reality camera tracking with
homographies’, IEEE Comput. Graph. Appl.,
November, Vol. 22, No. 6, pp.39–45.
■ Raskar, R., welch, g., low, k-l. and
Bandyopadhyay, d. (2001)
‘Shader lamps: animating real objects with
image based illumination’, Proceedings
of Eurographics Workshop on Rendering,
pp.89–102.
■ Raskar, R. and low, k-l. (2001)
‘Interacting with spatially augmented reality’,
ACM International Conference on Virtual
Reality, Computer Graphics and Visualization
in Africa (AFRIGRAPH), pp.101–108.
■ Raskar, R., van Baar, j., Beardsley, P.,
willwacher, t., Rao, s. and forlines, c.
(2003)
‘iLamps: geometrically aware and self-configuring
projectors’, SIGGRAPH, pp.809–818.
■ Raskar, R. and Bimber, o. (2004)
Spatial Augmented Reality, A.K. Peters,
ISBN: 1568812302.
■ Reikimoto, j. (2002)
‘SmartSkin: an infrastructure for freehand
manipulation on interactive surfaces’,
Proceedings of CHI ‘02, pp.113–120.
■ saakes, d.P., chui, k., hutchison, t.,
Buczyk, B.m., koizumi, n., inami, m.
and Raskar, R. (2010)
’ Slow Display’. In SIGGRAPH 2010 emerging
technologies: Proceedings of the 37th
annual conference on Computer graphics and
interactive techniques, July 2010.
■ schäfer, k., Brauer, V. and Bruns, w.
(1997)
‘A new approach to human-computer
interaction – synchronous modelling in real
and virtual spaces’, Proceedings of DIS ‘97,
pp.335–344.
■ schall, g., mendez, e., kruijff, e., Veas, e.,
sebastian, j., Reitinger, B. and schmalstieg,
d. (2008)
‘Handheld augmented reality for underground
infrastructure visualization’, Journal
of Personal and Ubiquitous Computing,
Springer, DoI 10.1007/s00779-008-0204-5.
■ schmalstieg, d. and wagner, d. (2008)
‘Mobile phones as a platform for augmented
reality’, Proceedings of the IEEE VR 2008
Workshop on Software Engineering and Architectures
for Realtime Interactive Systems,
pp.43–44.
■ sutherland, i.e. (1968)
‘A head-mounted three-dimensional display’,
Proceedings of AFIPS, Part I, Vol. 33,
pp.757–764.
■ ukita, n. and kidode, m. (2004)
‘Wearable virtual tablet: fingertip drawing
on a portable plane-object using an activeinfrared
camera’, Proceedings of IUI 2004,
pp.169–175.
■ Verlinden, j.c., de smit, A., horváth, i.,
epema, e. and de jong, m. (2003)
‘Time compression characteristics of the
augmented prototyping pipeline’, Proceedings
of Euro-uRapid’03, p.A/1.
■ Verlinden, j., horvath, i. (2008)
”Enabling interactive augmented prototyping
by portable hardware and a plugin-based
software architecture” Journal of Mechanical
Engineering, Slovenia, Vol 54(6), pp.
458-470.
■ welch, g. and foxlin, e. (2002)
‘Motion tracking: no silver bullet, but a respectable
arsenal’, IEEE Computer Graphics
and Applications, Vol. 22, No. 6, pp.24–38.
.
59

LIKE RIDING A BIKE. LIKE PARKING A CAR.
PoRTRAIT oF THE ARTIST IN RESIDENCE
MARINA DE HAAS
BY HANNA SCHRAFFENBERGER
"Hi Marina. Nice to meet you!
I have heard a lot about you."
I usually avoid this kind of phrases. Judging from
my experience, telling people that you have
heard a lot about them makes them feel uncomfortable.
But this time I say it. After all, it’s no
secret that Marina and the AR Lab in The Hague
share a history which dates back much longer
than her current residency at the AR Lab. At the
lab, she is known as one of the first students
who overcame the initial resistance of the fine
arts program and started working with AR. With
support of the lab, she has realized the AR artworks
Out of the blue and Drops of white in the
course of her study. In 2008 she graduated with
an AR installation that shows her 3d animated
portfolio. Then, having worked with AR for three
years, she decided to take a break from technology
and returned to photography, drawing and
painting. Now, after yet another three years,
she is back in the mixed reality world. Convinced
by her concepts for future works, the AR Lab
has invited her as an Artist in Residence. That is
what I have heard about her, and made me want
to meet her for an artist-portrait. Knowing quite
60 61

a lot about her past, I am interested in what she
is currently working on, in the context of her
residency. When she starts talking, it becomes
clear that she has never really stopped thinking
about AR. There’s a handwritten notebook
full of concepts and sketches for future works.
Right now, she is working on animations of two
animals. once she is done animating, she'll use
AR technology to place the animals — an insect
and a dove — in the hands of the audience.
“I usually start
out with my own
photographs and a
certain space I want
to augment.”
"They will hold a little funeral
monument in the shape of a tile
in their hands. Using AR technol-
ogy, the audience will then see a
dying dove or dying crane fly with
a missing foot.”
Marina tells me her current piece is about impermanence
and mortality, but also about the fact
that death can be the beginning of something
new. Likewise, the piece is not only about death
but also intended as an introduction and beginning
for a forthcoming work. The AR Lab makes
this beginning possible through financial support
but also provides technical assistance and serves
as a place for mutual inspiration and exchange.
Despite her long break from the digital arts, the
young artist feels confident about working with
AR again:
“It’s a bit like biking, once you’ve
learned it, you never unlearn it. It’s
the same with me and AR, of course
I had to practice a bit, but I still
have the feel for it. I think working
with AR is just a part of me.”
After having paused for three years, Marina is
positively surprised about how AR technology
has emerged in the meantime:
“AR is out there, it’s alive, it’s
growing and finally, it can be
markerless. I don’t like the use of
markers. They are not part of my
art and people see them, when
they don’t wear AR glasses. I am
also glad that so many people
know AR from their mobile phones
or at least have heard about it
before. Essentially, I don’t want
the audience to wonder about the
technology, I want them to look at
the pictures and animations I create.
The more people are used to
the technology the more they will
focus on the content. I am really
happy and excited how AR has
evolved in the last years!”
I ask, how working with brush and paint differs
from working with AR, but there seems to be
surprisingly little difference.
“The main difference is that with
AR I am working with a pen-tablet,
a computer and a screen. I
control the software, but if I work
with a brush I have the same kind
of control over it. In the past, I
used to think that there was a
difference, but now I think of the
computer as just another medium
to work with. There is no real
difference between working with
a brush and working with a computer.
My love for technology is
similar to my love for paint.”
Marina discovered her love for technology
at a young age:
“When I was a child I found a
book with code and so I programmed
some games. That was
fun, I just understood it. It’s the
same with creating AR works
now. My way of thinking perfectly
matches with how AR works. It
feels completely natural to me.”
Nevertheless, working with technology also has
its downside:
“The most annoying thing about
working with AR is that you are
always facing technical limitations
and there is so much that
can go wrong. No matter how well
you do it, there is always the risk
that something won’t work.
I hope for technology to get more
stable in the future.”
When realizing her artistic augmentations,
Marina sticks to an established workflow:
“I usually start out with my own
photographs and a certain space
I want to augment. Preferably I
measure the dimensions of the
space, and then I work with that
62 63

oom in my head. I have it in my
inner vision and I think in pictures.
There is a photo register in my
head which I can access. It’s a bit
like parking a car. I can park a car
in a very small space extremely
well. I can feel the car around me
and I can feel the space I want to
put it in. It’s the same with the
art I create. Once I have clear idea
of the artwork I want to create, I
use Cinema4D software to make
3d models. Then I use BuildAR to
place my 3d models it the real
space. If everything goes well,
things happen that you could not
have imagined.”
A result of this process is, for example, the AR
installation Out of the blue which was shown
at Today’s Art festival in The Hague in 2007:
“The idea behind ‘Out of the
blue’ came from a photograph
I took in an elevator. I took the
picture so that the lights in the
elevator looked like white ellipses
on a black background. I
took this basic elliptical shape
as a basis for working in a very
big space. I was very curious if I
could use such a simple shape and
still convince the audience that it
really existed in the space. And
it worked — people tried to touch
it with their hands and were very
surprised when that wasn’t possible.”
The fact that people believe in the existence of
her virtual objects is also important for Marina’s
personal understanding of AR:
“For me, Augmented Reality
means using digital images to
create something which is not
real. However, by giving meaning
to it, it becomes real and people
realize that it might as well
exist.”
I wonder whether there is a specific place or
space she’d like to augment in the future and
Marina has quite some places in mind. They have
one thing in common: they are all known museums
that show modern art.
“I would love to create works
for the big museums such as the
TATE Modern or MoMa. In the
Netherlands, I’d love to augment
spaces at the Stedelijk Museum in
Amsterdam or Boijmans museum
in Rotterdam. That’s my world.
Going to a museum means a lot
to me. Of course, one can place
AR artworks everywhere, also in
public spaces. But it is important
to me that people who experience
my work have actively chosen to
go somewhere to see art. I don’t
want them to just see it by accident
at a bus stop or in a park.”
Rather than placing her virtual models in a specific
physical space, her current work follows a
different approach. This time, Marina will place
the animated dying animals in the hands of the
audiences. The artist has some ideas about how
to design this physical contact with the digital
animals.
“In order for my piece to work,
the viewer needs to feel like he
is holding something in his hand.
Ideally, he will feel the weight of
the animal. The funeral monuments
will therefor have a certain
weight.”
It is still open where and when we will be able
to experience the piece:
“My residency lasts 10 weeks. But
of course that’s not enough time
to finish. In the past, a piece was
finished when the time to work on
it was up. Now, a piece is finished
when it feels complete. It’s something
I decide myself, I want to
have control over it. I don’t want
any more restrictions. I avoid
deadlines.”
Coming from a fine arts background, Marina has
a tip for art students who want to to follow in
her footsteps and are curious about working
with AR:
“I know it can be difficult to combine
technology with art, but it is
worth the effort. Open yourself
up to for art in all its possibilities,
including AR. AR is a chance
to take a step in a direction of
which you have no idea where
you’ll find yourself. You have to
be open for it and look beyond
the technology. AR is special —
I couldn’t live without it anymore...”
64 65

BIoGRAPHY -
JERoEN VAN ERP
jeRoen VAn eRP gRAduAted
fRom the fAculty of industRiAl
design At the technicAl
uniVeRsity of delft in
1988. in 1992, he wAs one of
the foundeRs of fABRique
in delft, which Positioned
itself As A multidisciPlinARy
design BuReAu. he estABlished
the inteRActiVe
mediA dePARtment in 1994,
focusing PRimARily on de-
VeloPing weBsites foR the
woRld wide weB - BRAnd
new At thAt time.
66
under jeroen’s joint leadership, fabrique
has grown through the years into a multi-
faceted design bureau. it currently employs
more than 100 artists, engineers and story-
tellers working for a wide range of customers:
from supermarket chain Albert heijn to the
Rijksmuseum.
fabrique develops visions, helps its clients
think about strategies, branding and innova-
tion and realises designs. Preferably cutting
straight through the design disciplines, so
that the traditional borders between graphic
design, industrial design, spatial design and
interactive media are sometimes barely recognisable.
in the bureau’s vision, this cross
media approach will be the only way to create
apparently simple solutions for complex
and relevant issues. the bureau also opened
a studio in Amsterdam in 2008.
jeroen is currently cco (chief creative
officer) and a partner, and in this role he
is responsible for the creative policy of the
company. he has also been closely involved
in various projects as art director and designer.
he is a guest lecturer for various
courses and is a board member at nAgo
(the netherlands graphic design Archive)
and the design & emotion society.
www.fabrique.nl
A MAGICAL LEVERAGE
IN SEARCH oF THE KILLER APPLICATIoN
BY JERoEN VAN ERP
the moment i was confronted with the technology of Augmented Reality,
back in 2006 at the Royal Academy of Arts in the hague, i was thrilled.
despite the heavy helmet, the clumsy equipment, the shaky images and
the lack of a well-defined purpose, it immediately had a profound impact
on me. from the start, it was clear that this technology had a lot of po-
tential, although at first it was hard to grasp why. Almost six years later,
the fog that initially surrounded this new technology has gradually faded
away. to start with, the technology itself is developing rapidly, as is the
equipment. But more importantly: companies and cul-
tural institutions are starting to understand how they can benefit from
this technology. At the moment there are a variety of applications avail-
able (mainly mobile applications for tablets or smart phones) that create
added value for the user or consumer. this is great, because it not only
allows the audience to gain experience in the field of this still-developing
technology, but also the industry. But to make Augmented Reality a real
success, the next step will be of vital importance.
67

innoVAting oR innoVAting?
let’s have a look at different forms of innovat-
ing in figure 1. on the left we see innovations
with a bottom-up approach, and on the right a
top-down approach to innovating. A bottomup
approach means that we have a promising
new technique, concept or idea although the
exact goal or matching business model aren’t
clear yet. in general, bottom-up developments
are technological or art-based, and are therefore
what i would call autonomous: the means
are clear, but the exact goal has still to be
defined. the usual strategy to take it further
is to set up a start-up company in order to
develop the technique and hopefully to create
a market.
this is not always that simple. innovating from
a top-down approach means that the innovation
is steered on the basis of a more or less
clearly defined goal. in contrast with bottomup
innovations, the goal is well-defined and
the designer or developer has to choose the
right means, and design a solution that fits
the goal. this can be a business goal, but also
figure 1.
a social goal. A business goal is often derived
from a benefit for the user or the consumer,
which is expected to generate an economic
benefit for the company. A marketing specialist
would state that there is already a market.
this approach means that you have to innovate
with an intended goal in mind. A business
goal-driven innovation can be a product innovation
(either physical products, services or a
combination of the two) or a brand innovation
(storytelling, positioning), but always with an
intended economical or social benefit in mind.
As there is an expected benefit, people are
willing to invest.
it’s interesting to note the difference on the
vertical axis between radical innovations and
incremental changes (Robert Verganti – design
drive innovation). incremental changes are
improvements of existing concepts or products.
this is happening a lot, for instance in
the automotive industry. in general, a radical
innovation changes the experience of the
product in a fundamental way, and as a result
of this often changes an entire business.
this is something Apple has achieved several
times, but it has also been achieved by tomtom,
and by Philips and douwe egberts with
their senseo coffee machine.
how ABout AR?
what about the position of Augmented Reality?
to start with, the Augmented Reality
technique is not a standalone innovation. it’s
not a standalone product but a technique or
feature that can be incorporated into products
or services with a magical leverage. At its core
it is a technique that was developed — and is
still developing — with specialist purposes in
mind. in principle there was no big demand
from ‘the market’. essentially, it is a bottomup
technological development that needs a
concept, product or service.
you can argue about whether it is an incre-
mental innovation or a radical one. A virtual
reality expert will probably tell you that it is
an improvement (incremental innovation) of
the VR technique. But if you look from an application
perspective, there is a radical aspect
to it. i prefer to keep the truth in the middle.
At this moment in time, AR is in the blue area
(figure 1).
it is clear that bottom-up innovation and topdown
innovation are different species. But
when it comes to economic leverage, it is a
challenge to be part of the top-down game.
this provides a guarantee for further development,
and broad acceptation of the technique
and principles. so the major challenge for AR
is to make the big step to the right part of figure
1 as indicated by the red arrow. Although
the principles of Augmented Reality are very
promising, it’s clear we aren’t there yet. An
example: we recently received a request to
‘do something’ with Augmented Reality. the
idea was to project the result of an AR application
onto a big wall. suddenly it occurred to
me that the experience of AR wasn’t suitable
at all for this form of publishing. AR doesn’t do
well on a projection screen. it does well in the
user’s head, where time, place, reality and
imagination can play an intriguing game with
our senses. it is unlikely that the technique of
Augmented Reality will lead to mass consumption
as in ‘experiencing the same thing with
a lot of people at the same time’. no, by their
nature, AR applications are intimate and intense,
and this is one of its biggest assets.
futuRe
we have come a long way, and the things we
can do with AR are becoming more amazing by
the day. the big challenge is to make it applicable
in relevant solutions. there’s no discussion
about the value of AR in specialist areas,
such as the military industry. institutions in
the field of art and culture have discovered
the endless possibilities, and now it is the
time to make the big leap towards solutions
with social or economic value (the green area
in figure 1). this will give the technique the
chance to develop further in order to flourish
at the end. from that perspective, it wouldn’t
surprise me if the first really good, efficient
and economically profitable application will
emerge for educational purposes.
let’s not forget we are talking about a tech-
nology that is still in its infant years. when i
look back at the websites we made 15 years
ago, i realize the gigantic steps we have made,
and i am aware of the fact that we could
hardly imagine then what the impact of the
internet would be on society today. of course,
it’s hard to compare the concept of Augmented
Reality with that of the internet, but it is
a valid comparison, because it gives the same
powerless feeling of not being able to predict
its future. But it will probably be bigger than
you can imagine.
68 69

THE PoSITIoNING
oF VIRTUAL oBJECTS
RoBERT PREVEL
WHEN USING AUGMENTED
REALITY (AR) FoR VISIoN,
VIRTUAL oBJECTS ARE ADDED
To THE REAL WoRLD AND DIS-
PLAYED IN SoME WAY To THE
USER; BE THAT VIA A MoNIToR,
PRoJECToR, oR HEAD-MoUNT-
ED DISPLAY (HMD). oFTEN IT IS
DESIRABLE, oR EVEN UNAVoID-
ABLE, FoR THE VIEWPoINT oF
THE USER To MoVE ARoUND
THE ENVIRoNMENT (THIS IS
PARTICULARLY THE CASE IF
THE USER IS WEARING A HMD).
THIS PRESENTS A PRoBLEM,
REGARDLESS oF THE TYPE oF
DISPLAY USED: HoW CAN THE
VIEWPoINT BE DECoUPLED
FRoM THE AUGMENTED VIR-
TUAL oBJECTS?
To recap, virtual objects are blended with the
real world view in order to achieve an Augmented
world view. From our initial viewpoint
we can determine what the virtual object’s
70
position and orientation (pose) in 3D space,
and its scale, should be. However, if the view
point changes, then how we view the virtual
object should also change. For example, if
I walk around to face the back of a virtual
object, I expect to be able to see the rear
of that object.
The solution to this problem is to keep track
of the user’s viewpoint and, in the event that
the viewpoint changes, to update the pose of
any virtual content accordingly. There are a
number of ways in which this can be achieved,
by using, for example: positional sensors
(such as inertia trackers), a global positioning
system, computer vision techniques, etc. Typically
the best results are those systems that
take the data from many tracking systems and
blend them together.
At TU Delft, we have been researching and
developing techniques to track position using
computer vision techniques. often it is
the case that video cameras are used in AR
systems; indeed, in the case where the AR
system uses video see-through, the use of
cameras is necessary. Using computer vision
techniques, we can identify landmarks in the
environment, and, using these landmarks, we
can determine the pose of our camera with
basic geometry. If the camera is not used
directly as the viewpoint (as is the case in
optical see-through systems), then we can still
keep track of the viewpoint by attaching the
camera to it. Say, for example, that we have
an optical see-through HMD with an attached
video camera. Then, if we calculate the pose
of the camera, we can then determine the
pose of the viewpoint, provided that the
camera’s position relative to the viewpoint
remains fixed.
The problem then, has been reduced to
identifying landmarks in the environment.
Historically, this has been achieved by the
use of fiducial markers, which act as points
of reference in the image. Fiducial markers
provide us with a means of determining the
scale of the visible environment, provided
that: enough points of reference are visible,
we know their relative positions, and these
relative positions don’t change. A typical
marker often used in AR applications consists
of a card with a black rectangle in the centre,
a white border, and an additional mark to
determine which edge of the card is considered
the bottom. As we know that the corners
of the black rectangle are all 90 degrees, and
we know the distance between corners, we
can identify the marker and determine the
pose of the camera with regard to the points
of reference (in this case the four corners of
the card).
A large number of simple ‘desktop’ AR applica-
tions make use of individual markers to track
camera pose, or conversely, to track the position
of the markers relative to our viewpoint.
Larger applications require multiple markers
linked together, normally distinguishable by
a unique pattern or barcode in the centre
of the marker. Typically the more points of
reference that are visible in a scene, the better
the results when determining the camera’s
pose. The key advantage to using markers
for tracking the pose of the camera is that
an environment can be carefully prepared
in advance, and provided the environment
does not change, should deliver the same AR
experience each time. Sometimes however,
it is not feasible to prepare an environment
with markers. often it is desirable to use an
AR application in an unknown or unprepared
environment. In these cases, an alternative
to using markers is to identify the natural
features found in the environment.
The term ‘natural features’ can be used to
describe the parts of an image that stand out.
Examples include: edges, corners, areas of
high contrast, etc. In order to be able to use
the natural features to track the camera position
in an unknown environment, we need to
be able to first identify the natural features,
and then determine their relative positions
in the environment. Whereas you could place
20 markers in an environment and still only
have 80 identifiable corners, there are often
hundreds of natural features in any one image.
This makes using natural features a more robust
solution than using markers, as there are
far more landmarks we can use to navigate,
not all of which need to be visible. one of the
key advantages to using natural features over
markers is that: as we already need to identify
and keep track of those natural features seen
from our initial view point, we can use the
same method to continually update a 3D map
of features as we change our view point.
This allows our working environment to grow,
which could not be achieved in a prepared
environment.
Although we are able to determine the relative
distance between features, the question
remains: how can we determine the absolute
position of features in an environment without
some known measurement? The short answer
is that we cannot; either we need to estimate
the distance or we can introduce a known
measurement. In a future edition we will
discuss the use of multiple video cameras and
how, given the absolute distance between the
cameras, we can determine the absolute position
of our identified features.
71

MEDIATED REALITY FoR CRIME
SCENE INVESTIGATIoN1 STEPHAN LUKoSCH
cRime scene inVestigAtion in the netheRlAnds is PRimARily
the ResPonsiBility of the locAl Police. foR seVeRe cRimes,
A nAtionAl teAm suPPoRted By the netheRlAnds foRensic
institute (nfi) is cAlled in. initiAlly cAPtuRing All detAils
of A cRime scene is of PRime imPoRtAnce (so thAt eVidence
is not Accidently destRoyed). nfi’s dePARtment of digitAl
imAge AnAlysis uses the infoRmAtion collected foR 3d
cRime scene ReconstRuction And AnAlysis.
within the csi the hague project (http://
www.csithehague.com) several companies and
research institutes cooperate under the guidance
of the netherlands forensic institute in
order to explore new technologies to improve
crime scene investigation by combining different
technologies to digitize, visualize and investigate
the crime scene. the major motivation
for the csi the hague project is that one
can investigate a crime scene only once. if you
do not secure all possible evidence during this
investigation, it will not be available for solving
the committed crime. the digitalization
of the crime scene provides opportunities for
testing hypotheses and witness statements,
but can also be used to train future investigators.
for the csi the hague project, two
groups at the delft university of technology,
systems engineering2 and Biomechanical engineering3
, joined their efforts to explore the
potential of mediated and Augmented Reality
for future crime scene investigation and to
tackle current issues in crime scene investigation.
in Augmented Reality, virtual data is
spatially overlaid on top of physical reality. with
this technology the flexibility of virtual reality
can be used and is grounded in physical reality
(Azuma, 1997). mediated reality refers to the
ability to add to, subtract information from, or
otherwise manipulate one’s perception of reality
through the use of a wearable computer or
hand-held device (mann and Barfield, 2003).
in order to reveal the current challenges for
supporting spatial analysis in crime scene
investigation, structured interviews with five
international experts in the area of 3d crime
scene reconstruction were conducted. the
interviews showed a particular interest for
current challenges in spatial reconstruction
and the interaction with the reconstruction
data. the identified challenges are:
1This article is based upon (Poelman et al., 2012).
2http://www.sk.tbm.tudelft.nl 3 http://3me.tudelft.nl/en/about-the-faculty/departments/biomechanical-engineering/research/dbl-delft-biorobotics-lab/people/
72
figure 1. mediAted ReAlity heAd mounted deVice in use duRing the exPeRiment in the dutch foRensic field lAB.
■ Time needed for reconstruction: data capture,
alignment, data clean-up, geometric
modelling and analyses are manual steps.
■ Expertise required to deploy dedicated software
and secure evidence at the crime scene.
■ Complexity: Situations differ significantly.
■ Time freeze: Data capture is often conducted
once after a scene has been contaminated.
The interview sessions ended with an open
discussion on how mediated reality can support
crime scene investigation in the future. Based
on these open discussions, the following requirements
for a mediated reality system that is to
support crime scene investigation were identified:
■ Lightweight head-mounted display (HMD):
It became clear that the investigators whom
arrive first on the crime scene currently carry
a digital camera. Weight and ease of use are
important design criteria. Experts would like
those close to a pair of glasses.
■ Contactless augmentation alignment (no
markers on the crime scene): The first
investigator who arrives on a crime scene has
to keep the crime scene as untouched as possible.
Technology that involves preparing the
scene is therefore unacceptable.
■ Bare hands gestures for user interface opera-
tion: The hands of the CSIs have to be free to
physically interact with the crime scene when
needed, e.g. to secure evidence, open doors,
climb, etc. Additional hardware such as data
gloves or physically touching an interface
such as a mobile device is not acceptable.
■ Remote connection to and collaboration with
experts: Expert crime scene investigators are
a scarce resource and are not often available
at location on request. Setting up a remote
connection to guide a novice investigator
through the crime scene and to collaboratively
analyze the crime scene has the potential
to improve the investigation quality.
To address the above requirements, a novel
mediated reality system for collaborative spatial
analysis on location has been designed, developed
and evaluated together with experts in the
field and the NFI. This system supports collaboration
between crime scene investigators (CSIs)
on location who wear a HMD (see Figure 1) and
expert colleagues at a distance.
The mediated reality system builds a 3D map of
the environment in real-time, allows remote users
to virtually join and interact together in shared
Augmented space with the wearer of the HMD,
and uses bare hand gestures to operate the 3D
multi-touch user interface. The resulting medi-
73

figure 2. heAd mounted disPlAy, modified
cinemiZeR oled (cARl Zeiss) with two micRosoft
hd-5000 weBcAms.
figure 3. sPARse 3d feAtuRe mAP geneRAted By
the Pose estimAtion module.
figure 4. gRAPhicAl useR inteRfAce oPtions menu.
74
ated reality system supports a lightweight headmounted
display (HMD), contactless augmentation
alignment, and a remote connection to and collaboration
with expert crime scene investigators.
The video see-through of a modified Carl Zeiss
Cinemizer oLED (cf. Figure 2) for displaying
content fulfills the requirement for a lightweight
HMD, as its total weight is ~180 grams. Two
Microsoft HD-5000 webcams are stripped and
mounted in front of the Cinemizer providing a
full stereoscopic 720p resolution pipeline. Both
cameras record at ~30hz in 720p, images are
projected in our engine, and render 720p stereoscopic
images to the Cinemizer.
As for all mediated reality systems, robust realtime
pose estimation is one of the most crucial
parts, as the 3D pose of the camera in the physical
world is needed to render virtual objects
correctly on required positions. We use a heavily
modified version of PTAM (Parallel Tracking and
Mapping) (Klein and Murray, 2007), in which
a single camera setup is replaced by a stereo
camera setup using 3D natural feature matching
and estimation based on natural features. Using
this algorithm, a sparse metric map (cf. Figure
3) of the environment is created. This sparse
metric map can be used for pose estimation in
our Augmented Reality system.
In addition to the sparse metric map, a dense
3D map of the crime scene is created. The
dense metric map provides a detailed copy of
the crime scene enabling detailed analysis and
is created from a continuous stream of disparity
maps that are generated while the user moves
around the scene. Each new disparity map is
registered (combined) using the pose information
from the PE module to construct or extend
the 3D map of the scene. The point clouds are
used for occlusion and collision checks, and for
snapping digital objects to physical locations.
By using an innovative hand tracking system,
the mediated reality system can recognize bare
hands gestures for user interface operation.
This hand tracking system utilizes the stereo
camera rig to detect the hand movements in 3D.
The cameras are part of the HMD and an adaptive
algorithm has been designed to determine
whether to rely on the color, disparity or on both
depending on the lighting conditions. This is the
core technology to fulfill the requirement of
bare hand interfacing. The user interface and
the virtual scene are general-purpose parts of
the mediated reality system. They can be used
for CSI, but also for any other mediated reality
application. The tool set, however, needs to be
tailored for the application domain. The current
mediated reality system supports the following
tasks for CSIs: recording the scene, placing tags,
loading 3D models, bullet trajectories and placing
restricted area ribbons. Figure 4 shows the
corresponding menu attached to a user’s hand.
The mediated reality system has been evalu-
ated on a staged crime scene at the NFI’s Lab
with three observers, one expert and one
layman with only limited background in CSI.
Within the experiment the layman, facilitated
by the expert, conducted three spatial tasks,
i.e. tagging a specific part of the scene with
information tags, using barrier tape and poles
to spatially secure the body in the crime scene
and analyzing a bullet trajectory analysis with
ricochet. The experiment was analyzed along
seven dimensions (Burkhardt et al., 2007): fluidity
of collaboration, sustaining mutual understanding,
information exchanges for problem
solving, argumentation and reaching consensus,
task and time management, cooperative orientation,
and individual task orientation. The
results show that the mediated reality system
supports remote spatial interaction with the
physical scene as well as collaboration in shared
augmented space while tackling current issues in
crime scene investigation. The results also show
that there is a need for more support to identify
whose turn it is and who wants the next turn,
etc. Additionally, the results show the need to
represent the expert in the scene to increase
the awareness and trust of working in a team
and to counterbalance the feeling of being observed.
Knowing the expert’s focus and current
activity could possibly help to overcome this issue.
Whether traditional patterns for computermediated
interaction (Schümmer and Lukosch,
2007) support awareness in mediated reality
or rather new forms of awareness need to be
designed, will be the subject of future research.
Further tasks for future research include the
design and evaluation of alternative interaction
possibilities, e.g. by using physical objects that
are readily available in the environment, sensor
fusion, image feeds from spectral cameras or
previously recorded laser scans, to provide more
situational awareness and the privacy, security
and validity of captured data. Finally, though
IT is being tested and used for educational
purposes within the CSI Lab of the Netherlands
Forensic Institute (NFI), only the application and
test of the mediated reality system in real settings
can show the added value for crime scene
investigation.
RefeRences
■ R. Azuma, A Survey of Augmented Reality,
Presence 6, Vol 4, 1997, 355-385
■ J. Burkhardt, F. Détienne, A. Hébert, L. Perron,
S. Safin, P. Leclercq, An approach to assess the
quality of collaboration in technology-mediated
design situation, European Conference on Cognitive
Ergonomics: Designing beyond the Product - Understanding
Activity and User Experience in Ubiquitous
Environments, 2009, 1-9
■ G. Klein, D. Murray, Parallel Tracking and Mapping
for Small AR Workspaces, Proc. International
Symposium on Mixed and Augmented Reality, 2007,
225-234
■ S. Mann, W. Barfield, Introduction to Mediated
Reality, International Journal of Human-Computer
Interaction, 2003, 205-208
■ R. Poelman, o. Akman, S. Lukosch, P. Jonker, As
if Being There: Mediated Reality for Crime Scene
Investigation, CSCW ‘12: Proceedings of the 2012
ACM conference on Computer Supported Cooperative
Work, ACM New York, NY, USA, 2012, 1267-1276,
http://dx.doi.org/10.1145/2145204.2145394
■ T. Schümmer, S. Lukosch, Patterns for Computer-
Mediated Interaction, John Wiley & Sons, Ltd. 2007
75

on fRidAy decemBeR 16th 2011
the symPhony oRchestRA
of the RoyAl conseRVAtoiRe
PlAyed die wAlküRe (Act 1)
By RichARd wAgneR, At the
BeAutiful conceRt hAll ‘de
VeReeniging’ in nijmegen.
the AR lAB wAs inVited By
the RoyAl conseRVAtoiRe to
PRoVide VisuAls duRing this
liVe PeRfoRmAnce. togetheR
with students fRom diffeRent
dePARtments of the RoyAl
AcAdemy of ARt, we designed
A scReen consisting of 68
Pieces of tRAnsPARent cloth
(400x20 cm), hAnging in fouR
lAyeRs ABoVe the oRchestRA.
By PRojecting on this cloth
we cReAted VisuAls giVing the
illusion of dePth.
we chose 7 leitmotiVs (RecuR-
Ring theme, AssociAted with A
PARticulAR PeRson, PlAce, oR
ideA), And cReAted AnimAtions
RePResenting these using
colouR, shAPe And moVement.
these AnimAtions weRe PlAyed
At key-moments of the PeRfoRmAnce.
76 77

CoNTRIBUToRS
wim VAn eck
Royal Academy of Art (KABK)
w.vaneck@kabk.nl
Wim van Eck is the 3D animation specialist of the
AR Lab. His main tasks are developing Augmented
Reality projects, supporting and supervising
students and creating 3d content. His interests
are, among others, real-time 3d animation, game
design and creative research.
jeRoen VAn eRP
Fabrique
jeroen@fabrique.nl
Jeroen van Erp co-founded Fabrique, a multidisciplinary
design agency in which the different
design disciplines (graphic, industrial, spatial
and new media) are closely interwoven. As a
designer he was recently involved in the flagship
store of Giant Bicycles, the website for the
Dutch National Ballet and the automatic passport
control at Schiphol airport, among others.
PieteR jonkeR
Delft University of Technology
P.P.Jonker@tudelft.nl
Pieter Jonker is Professor at Delft University
of Technology, Faculty Mechanical, Maritime
and Materials Engineering (3ME). His main
interests and fields of research are: real-time
embedded image processing, parallel image
processing architectures, robot vision, robot
learning and Augmented Reality.
yolAnde kolstee
Royal Academy of Art (KABK)
Y.Kolstee@kabk.nl
Yolande Kolstee is head of the AR Lab since
2006. She holds the post of Lector (Dutch for
researcher in professional universities) in the
field of ‘Innovative Visualisation Techniques in
higher Art Education’ for the Royal Academy of
Art, The Hague.
mAARten lAmeRs
Leiden University
lamers@liacs.nl
Maarten Lamers is assistant professor at the
Leiden Institute of Advanced Computer Science
(LIACS) and board member of the Media Technology
MSc program. Specializations include social
robotics, bio-hybrid computer games, scientific
creativity, and models for perceptualization.
stePhAn lukosch
Delft University of Technology
S.g.lukosch@tudelft.nl
Stephan Lukosch is associate professor at the
Delft University of Technology. His current
research focuses on collaborative design and
engineering in traditional as well as emerging
interaction spaces such as augmented reality.
In this research, he combines recent results from
intelligent and context-adaptive collaboration
support, collaborative storytelling for knowledge
elicitation and decision-making, and design
patterns for computer-mediated interaction.
feRenc molnáR
Photographer
info@baseground.nl
Ferenc Molnár is a multimedia artist based in
The Hague since 1991. In 2006 he has returned
to the KABK to study photography and that’s
where he started to experiment with AR. His
focus is on the possibilities and on the impact of
this new technology as a communication platform
in our visual culture.
RoBeRt PReVel
Delft University of Technology
r.g.prevel@tudelft.nl
Robert Prevel is working on a PhD focusing on
localisation and mapping in Augmented Reality
applications at the Delft Biorobotics Lab, Delft
University of Technology under the supervision
of Prof.dr.ir P.P.Jonker.
hAnnA
schRAffenBeRgeR
Leiden University
hkschraf@liacs.nl
Hanna Schraffenberger works as a researcher
and PhD student at the Leiden Institute of
Advanced Computer Science (LIACS) and at
the AR Lab in The Hague. Her research interests
include interaction in interactive
art and (non-visual) Augmented Reality.
esmé VAhRmeijeR
Royal Academy of Art (KABK)
e.vahrmeijer@kabk.nl
Esmé Vahrmeijer is graphic designer and webmaster
of the AR Lab. Besides her work at the
AR Lab, she is a part time student at the Royal
Academy of Art (KABK) and runs her own graphic
design studio ooxo. Her interests are in graphic
design, typography, web design, photography
and education.
jouke VeRlinden
Delft University of Technology
j.c.verlinden@tudelft.nl
Jouke Verlinden is assistant professor at the
section of computer aided design engineering
at the Faculty of Industrial Design Engineering.
With a background in virtual reality and interaction
design, he leads the “Augmented Matter in
Context” lab that focuses on blend between bits
and atoms for design and creativity. Co-founder
and lead of the minor on advanced prototyping
programme and editor of the International
Journal of Interactive Design, Engineering and
Manufacturing.
SPECIAL THANKS
We would like to thank Reba Wesdorp, Edwin
van der Heide, Tama McGlinn, Ronald Poelman,
Karolina Sobecka, Klaas A. Mulder, Joachim
Rotteveel and last but not least the Stichting
Innovatie Alliantie (SIA) and the RAAK (Regionale
Aandacht en Actie voor Kenniscirculatie) initiative
of the Dutch Ministry of Education, Culture
and Science.
NEXT ISSUE
The next issue of AR[t] will be out in october
2012.
78 79