Development of an Augmented Reality system using ARToolKit

Supervised by:
Research Assignment
Development of an Augmented Reality system using
ARToolKit and user invisible markers
Dennis Joele, student number: 1047450
Master Programme Media & Knowledge Engineering
Dr.ir. C.A.P.G. van der Mast, Dr. María Carmen Juan-Lizandra,
Unit of Man-Machine Interaction, Medical Image Computing Laboratory,
Department of Mediamatics, Department of Infomation Systems and
Delft University of Technology, Computation,
The Netherlands Technical University of Valencia,
Spain
Valencia, May 26 th 2005

Development of an Augmented Reality system using ARToolKit and user invisible markers
Table of contents
Table of contents..............................................................................................................1
1. Introduction .................................................................................................................3
1.1 Research Goals...................................................................................................... 3
1.2 Research Approach .............................................................................................. 4
1.3 Research Outline .................................................................................................. 4
2. Augmented Reality and its applications....................................................................5
2.1 AR in relation to VR............................................................................................. 5
2.2 Display technologies ............................................................................................. 7
2.2.1 Head-attached displays ................................................................................. 8
2.2.1.1 HMD ........................................................................................................ 8
2.2.1.2 Head-mounted projector........................................................................ 9
2.2.1.3 Retinal displays..................................................................................... 10
2.2.2 Hand-held displays ...................................................................................... 10
2.2.2.1 PDA........................................................................................................ 11
2.2.2.2 Mobile Phone ........................................................................................ 11
2.2.2.3 Hand-held mirror displays .................................................................. 11
2.2.2.4 Hand-held video projectors ................................................................. 12
2.2.3 Spatial displays ............................................................................................ 12
2.2.3.1 Screen based video see-through displays............................................ 13
2.2.3.2 Spatial optical see-through displays.................................................... 13
2.2.3.3 Projection based spatial displays ........................................................ 14
2.3 Tracking .............................................................................................................. 14
2.3.1 Inertial tracking........................................................................................... 15
2.3.2 Acoustic tracking ......................................................................................... 16
2.3.3 Magnetic tracking........................................................................................ 17
2.3.4 Mechanical tracking.................................................................................... 17
2.3.5 Optical tracking ........................................................................................... 18
2.3.6 Hybrid tracking, outdoor tracking and collaborative AR tracking ....... 18
2.4 Registration ......................................................................................................... 19
2.4.1 Static errors.................................................................................................. 20
2.4.2 Dynamic errors ............................................................................................ 21
2.5 Calibration .......................................................................................................... 22
2.5.1 Manual calibration ...................................................................................... 22
2.5.2 Calibration-free AR..................................................................................... 23
2.5.3 Autocalibration ............................................................................................ 23
2.6 Interaction ........................................................................................................... 23
2.6.1 3D Data browsing ........................................................................................ 24
2.6.2 3D AR interfaces.......................................................................................... 24
2.6.3 Augmented surfaces .................................................................................... 24
2.6.4 Tangible AR ................................................................................................. 24
2.7 AR applications................................................................................................... 25
2.7.1 Medical ......................................................................................................... 25
2.7.1.1. Image guided surgery.......................................................................... 25
2.7.1.2. Surgery planning ................................................................................. 26
2.7.1.3. Surgical training .................................................................................. 27
2.7.1.4. Psychiatry............................................................................................. 27
2.7.2 Military......................................................................................................... 28
1

Development of an Augmented Reality system using ARToolKit and user invisible markers
2.7.3 Education...................................................................................................... 29
2.7.4 Engineering design ...................................................................................... 29
2.7.5 Manufacturing, maintenance and repair .................................................. 30
2.7.6 Entertainment .............................................................................................. 30
3. ARToolKit and the AR development process.........................................................32
3.1 Tracking and registration.................................................................................. 32
3.2 Optical see-through calibration......................................................................... 33
3.3 Video see-through calibration ........................................................................... 33
3.4 Developing AR applications............................................................................... 34
3.5 Training marker patterns.................................................................................. 35
4. Hardware components for AR with user invisible markers..................................36
4.1 Invisible marker tracking medium ................................................................... 37
4.1.1 Ink Pen.......................................................................................................... 39
4.1.2 IR Powder..................................................................................................... 39
4.2 Optical filters ...................................................................................................... 40
4.2.1 IR filters........................................................................................................ 40
4.2.2 Colour correction filter ............................................................................... 41
4.3 IR Cameras ......................................................................................................... 41
4.3.2.1 USB ........................................................................................................ 43
4.3.2.2 IEEE1394 Firewire............................................................................... 44
4.4 Firewire board camera....................................................................................... 46
4.4.1 Lenses............................................................................................................ 47
4.4.3 Round IR filter............................................................................................. 50
4.5 AR HMD system ................................................................................................. 51
4.5.1 Video see-through........................................................................................ 51
4.5.2 Optical see-through ..................................................................................... 53
5. Proposal for system components..............................................................................58
5.1 Invisible medium ................................................................................................ 58
5.2 IR Camera........................................................................................................... 58
5.3 Connection to computer..................................................................................... 59
5.4 HMD .................................................................................................................... 59
5.5 Final choice on system components and perspective....................................... 61
6. References ..................................................................................................................62
7. Abbreviations.............................................................................................................65
2

Development of an Augmented Reality system using ARToolKit and user invisible markers
1. Introduction
Virtual Reality (VR) has been around for quite some time and consequently has found
its way into many application areas. One area that Delft University of Technology is
investigating with their project Virtual Reality and Phobias is the application of VR to
the treatment of phobias. Results have been encouraging; virtual reality exposure has
shown to be as effective as in vivo exposure to fear of heigths, even with low budget
hardware [Emm02] [Sch00]. Currently is being investigated the use of VR for treatment
of claustrophobia and fear of flying.
Recently Augmented Reality (AR) also has been tried to be used in treatment of
phobias. At MedICLab, Technical University of Valencia, has been developed an AR
system for the treatment of phobia of small animals. Up until now only a limited
number of patients with fear of spiders and cockroaches were treated successfully
[Jua04]. However this gives a good perspective to try to apply AR to other type of
phobias as well. Currently is being investigated the use of AR for treatment of fear of
heights.
Many AR systems use a Head Mounted Display (HMD) to show a combination of the
virtual and the real world to the user. ARToolKit is a software library to create AR
applications that use physical markers in the real environment to determine where
exactly to overlay the virtual objects [Kat99]. Also the AR system at MedICLab is
based upon those two components. This has proved to function well, but the desire has
grown to have an AR system to work with user invisible markers. The main advantage
of this is that users are not able to perceive the markers which introduce a surprise effect
when virtual objects are overlaid. The effectiveness and reality factor of AR treatment
are expected to increase.
1.1 Research Goals
The research assignment is carried out as a preparation for the Master of Science Thesis
project in MedICLab, Technical University of Valencia. The project’s ultimate goal is
aimed at developing an AR system that uses invisible markers and is based upon
ARToolKit.
To actually be able to conduct the project itself, the condition that primarily needs to be
met is that the needed information is collected, processed and structured. The research
assignment will suffice to provide the background that is required and will mould the
obtained results into a structured report.
To be able in doing so, a clear problem definition needs to be articulated. The problem
definition, consisting of several elements, that will serve as guidance in the research
assignment is the following:
• Research the current state of the art on AR, and get acquainted with
ARToolKit.
• Find suitable hardware components for the implementation of detecting of
user invisible markers, that is, a suitable invisible medium, a camera to detect
that medium and a connection between the camera and a computer. The
camera should work together with ARToolKit.
3

Development of an Augmented Reality system using ARToolKit and user invisible markers
• Find a suitable stereoscopic video see-through HMD that incorporates both
capture and visualization. No requirements are stated on intrinsic properties
of the HMD.
• Find a suitable optical see-through HMD. Also for this HMD no requirements
are stated on intrinsic properties. Moreover the stereoscopic requirement is
relaxed.
• Propose the hardware components to create an AR system that uses invisible
marker tracking in conjunction with ARToolKit.
In order to provide an answer to this problem definition, several goals are attached to
the research assignment. The goals identified are the following:
• Learning about AR and applications of AR
• Learning the functionality of ARToolKit
• Proposing a setup of hardware components for future AR system, that meets the
requirements stated in the problem formulation.
1.2 Research Approach
For answering the problem definition, doing research on related topics is a necessity. Of
course there are many research methods available, and a selection needs to put the focus
on a limited number. For this research the following five research methods are applied:
1. The first step in the taken approach consists of doing literature study, which will
provide all the relevant information on the subject. The literature study existed
of searching for relevant literature, papers and other information sources.
2. Secondly ARToolKit, the development toolkit for AR applications will be
studied to provide insight into the AR development process.
3. After the first two preparative steps, a search intending to find a set of hardware
components will be carried out.
4. The last step will consist of an evaluation of found hardware components which
eventually lead to a proposed hardware setup.
1.3 Research Outline
In this chapter an introduction on the research was given, together with the research
goals and the research approach. The second chapter will discuss AR and current state
of the art AR applications. ARToolKit is described in detail in the third chapter. The
results of searching for hardware components are summarized in the next chapter. The
fifth chapter, which is the last chapter, contains proposals for a hardware setup. To
conclude, in this chapter is also presented which hardware components will be obtained
in the end, and what the perspective will be.
4

Development of an Augmented Reality system using ARToolKit and user invisible markers
2. Augmented Reality and its applications
Augmented Reality has received growing attention during the last few years. More and
more researchers have started to pick up the technology. Despite its novelty already
great advances have been made, as is evidenced by the known examples of AR
applications. This is not limited to one particular area but nowadays there are several
areas found appropriate for application of AR. And developments continue as more
areas are pioneered. Not only expanding areas of applications deserves attention, but
more important is refining the technology itself. Therefore the three basic aspects that
constitute the technology need to be researched and improved. These aspects are
tracking, registration and calibration [Isd00].
Most definitions of AR narrow the view by focusing on HMD as display technology. In
[Mil94] is referred to one definition that defines AR “as being a form of Virtual Reality
where the participant's head-mounted display is transparent, allowing a clear view of the
real world”. But a definition that does not contain any reference to a certain display
technology, allow for a broader view. This keeps open more possibilities for
developments. Such a definition is the following one, found in[Azu01]:
An AR system is defined as having the following three properties:
• It combines real and virtual objects in a real environment
• It runs interactively and in real-time
• It registers real and virtual objects with each other
Firstly will be positioned AR against VR. After that will be discussed display
technologies for establishing AR. Then, the three basic aspects that constitute AR are
treated. To complete the discussion, an extensive overview of former AR applications is
given, mainly focusing on the way were implemented before mentioned aspects.
2.1 AR in relation to VR
The concept of Virtual Reality has been around for quite some time. Researchers have
constituted a common view on what VR actually is. A VR environment is defined as a
completely synthetic environment in which the observer is totally immersed. In contrast
to VR environments exists the real world in which only real objects are present and in
which the observer is not at all immersed. These two types of environments have been
defined as the opposite ends of a continuum which is referred to as the Virtual-Reality
Continuum (VRC) [Mil94]. This is shown in Figure 2.1.
Figure 2.1 The Virtual-Reality Continuum
Figure 2.2 An alternative VRC representation
5

Development of an Augmented Reality system using ARToolKit and user invisible markers
Another concept, known as Mixed Reality (MR), is introduced to encompass all the
environments between both extremes of the VRC. In such environments real world and
virtual world objects are concurrently presented together within one single display. Two
categories to create MR are identified; Augmented Reality and Augmented Virtuality
(AV). In AR environments the environment being observed is principally real, and is
added computer generated enhancements. In AV environments the surrounding
environment is principally virtual, but augmented through the use of real imaging data.
Figure 2.2 shows another way to visualize the RV continuum by differentiating between
display technologies that is used. Depending on that, different levels of reality are
achieved. For the left extreme, defined as reality, no supplemental display technology is
required. VR clearly has the need to be supported by display technology. This usually is
done by using immersive head mounted displays. For MR to be able to present real
world and virtual objects at the same time, also display technology is taken advantage
of. Such MR display technology can be put together in a taxonomy which is based upon
three properties; reality, immersion and directness [Mil94].
• Reality, that is, some environments are primarily virtual, in the sense that they
have been created artificially, by computer, while others are primarily real.
• Immersion, that is, virtual and real environments can each be displayed without
the need for the observer to be completely immersed within them.
• Directness, that is, whether primary world objects are viewed directly or by
means of some electronic synthesis process.
This taxanomy identifies seven most prominent display concepts for MR. As it focuses
on the MR class in total it is not as general as is needed for distinguishing between the
several display technologies available for AR. Others have tried to provide a
categorization only focused on AR, as is discussed in the next section.
The main advantage that AR offers in favour of VR is presented in the reality factor;
only the augmented objects are virtual, while the environment is real. This offers
excellent possibilities for giving the user an enhanced experience. Users can use
familiar tools to manipulate virtual objects, which makes it an intuitive interface. Next
to this, users are not fully immersed which keeps open social communication channels.
This creates a secure feeling and does not hinder natural movements. AR offers a
technology to present 3D information in a smooth and ergonomic way. Lower demands
are put upon rendering performance since only the augmented information needs to be
rendered.
AR is expected to have a great potential although its effectiveness is not supported by
many case studies. One such known study measures the effectiveness of AR applied to
an assembly task [Tan03]. The shown results are promising; error rates for the chosen
task are reduced by 82%, and the measured mental effort indicates a decrease. There is a
need for more research on the effectiveness of AR to enable it to become a more
generally used technique. What also needs to be given attention is the effectiveness of
AR compared to VR. Because of the above mentioned advantages in favour of AR, the
assumption exists that AR is more effective. One example acknowledges this assumed
greater effectiveness for cybertherapy [Jua04]. One single AR treatment session reduced
a patient’s fear of cockroaches from 10 to 0 SUD, while treatment with VR required
several sessions to establish the same result. The SUD scale provides a way to measure
6

Development of an Augmented Reality system using ARToolKit and user invisible markers
how strong or unpleasant experienced emotions are felt. A total of 11 patients were
treated; 4 having spider phobia and 7 with fear of cockroaches. Of course, one such
example does not constitute a founded basis for the assumption.
2.2 Display technologies
As described in the introduction AR is not necessarily limited to a certain type of
display technology. There are more possibilities available to choose from. To keep
things clear it has been tried to categorize these. Two such categorizations can be found
in [Azu01] and [Bim04a]. There are minor differences between them, but what they
have in common is the main division in three categories; head-attached displays, hand
held displays and spatial displays.
Display devices are used for forming an augmented image that eventually is presented
to the user. The two types of images that can be formed are either planar or curved,
depending on the used optics. The process of image-forming can take place anywhere
on the optical path between the eyes of the observer and the physical object. These two
properties make it possible to categorize display technologies. Figure 2.3 shows the
three before mentioned categories.
Figure 2.3 Categorization of AR display technologies
Retinal displays and projector-based devices form a curved image; either directly on the
retina of the observer or on the real objects. The other display devices form an image on
a defined plane. Head-attached devices are mounted on the head of the user. Hand-held
devices are positioned at arm reach from the user, while spatial devices are integrated in
the environment and are completely detached from the user.
Now will be shortly discussed the basics of the three categories. The main concepts are
described, advantages and disadvantages are mentioned and examples are given for each
category.
7

Development of an Augmented Reality system using ARToolKit and user invisible markers
2.2.1 Head-attached displays
Head-attached displays are mounted on the head of the user. The technology for
generating images can be implemented in three ways. This choice determines the three
different types of head-attached displays that exist. Miniature displays in front the eyes
can be used. Then a head-attached display of type HMD is created. When miniature
LCD panels or miniature projectors are used the term head-mounted projector is
utilized. Finally, retinal displays take advantage of low power lasers for the image
generation.
2.2.1.1 HMD
This type of displays comes into two kinds; video see-through and optical see-through
[Sch01]. Figure 2.4 and Figure 2.5 show the concept of both approaches.
The video see-through approach uses two cameras to record a real scene. Virtual objects
are generated by a graphic system of a computer. These two are merged together by a
video composer to create an augmented view. The images are projected onto the opaque
micro displays of the HMD. In this way a closed view system is created.
Optical see-through HMDs use a semi-transparent display for allowing the real world to
be seen by the user. Virtual objects are generated by a graphics system of a computer.
Optical merging projects the generated virtual objects onto the view of the user. In this
way the user has an augmented view on the real scene.
Figure 2.4 Video see-through HMD concept Figure 2.5 Optical see-through HMD concept
Bimber and Raskar [Bim04a] identify several shortcomings to the use of HMDs as
display devices for AR. Here are not mentioned all of them, but only the most important
ones. For both kinds of HMDs the Field Of View (FOV) is limited to the optics that is
used. But either kind also has its own particular disadvantages which are mentioned
now.
Video see-through HMDs limit the resolution of both the real world and virtual objects
to the resolution of the used video optics. Only one focal plane is used for both
displaying the real world and virtual objects. Tracking is an important aspect, but has a
lower importance compared to optical see-through HMDs, since image processing
techniques are used to overlay virtual objects. This allows for integrating virtual objects
on a pixel basis. The calibration aspect therefore is not a stringent requirement.
Optical see-through HMDs only limit the resolution of virtual objects to the resolution
of the optics system. The visual perception is coping with the fact that different
8

Development of an Augmented Reality system using ARToolKit and user invisible markers
projection depths are used for the real world and the virtual objects; either the eyes have
to switch between those two or perceive one depth as unsharp. Registration of virtual
objects heavenly relies on precise head tracking. Every time optical see-through HMDs
are used they need to be calibrated which is difficult and a time consuming task.
2.2.1.2 Head-mounted projector
As HMDs can be divided into two kinds, this also holds for this type of displays. The
two types that can be distinguished are head-mounted projective displays and projective
head-mounted displays.
The first type makes use of retro-reflective screens that are mounted in front of the eyes
of the user [Hua04]. Such surfaces have the property to reflect light back along its
incident direction. A projective lens in combination with a beam-splitter is used to
redirect an image from a miniature LCD display onto these surfaces. The incoming light
is retro-reflected back to the exit pupil to have the user observe the projected image.
This concept is presented in Figure 2.6.
Figure 2.6 Head-mounted projective Figure 2.7 Projective head-mounted
display concept display concept
The approach that uses projective head-mounted displays does not, as opposed to the
earlier described one, use retro-reflective screens in front of the user. Instead, images
are projected along the viewer’s line of sight by a ceiling attached projector [Kij97].
Two half mirrors are applied to combine the back directed projected images and the
user’s view of the environment. This concept is shown in Figure 2.7.
Compared to HMDs both kinds of head-mounted projectors offer a larger FOV. Headmounted
projective also displays decrease the effect of inconsistency of accommodation
and convergence. However, several shortcomings are related to the use of headmounted
projectors [Bim04a]. The technology as developed up until now is not
ergonomic in use and currently available prototypes are cumbersome. Limited
resolution and brightness can be achieved by the used miniature LCD screens and
projectors. For the projective head-mounted displays an extra requirement is the
presence of a ceiling, which limits its use to indoor environments. Head-mounted
projective displays require special retro-reflective surfaces for bright imagery.
9

Development of an Augmented Reality system using ARToolKit and user invisible markers
2.2.1.3 Retinal displays
As the earlier described display devices use small displays in front of the eyes to project
images, this category directly projects an image on the retina of the eye. This is done by
aiming a small laser spot at the retina and sweeping it over using a raster pattern. To be
used without causing any damage the lasers have very low power. Despite this bright
images can be created, with a potential greater FOV compared to screen-based displays.
This concept is shown in Figure 2.8.
Figure 2.8 Retinal display concept
As current retinal display technology is not yet as developed as other display
technologies, additional disadvantages are introduced [Bim04a]. Only monochrome red
images can be presented, since cheap blue or green lasers do not yet exist. Stereoscopy
can also not be supported by existing retinal displays.
2.2.2 Hand-held displays
Hand-held displays combine multiple components into one single device; processor,
memory, display and interaction technology. Usually these devices are small, lightweight,
wire-less and mobile. Images are generated at arm length. Such devices can be
used as a window or a magnifying glass. The video see-through approach is most
promising to be used with this kind of devices. A camera captures a real time video
stream of the environment and before displaying such video material to the user
graphical augmentations are overlaid. The position and orientation of the camera are
tracked so that result correct overlays.
There are available different kinds of hand-held displays. Among those are Personal
Digital Assistant (PDA), mobile phone, hand-held mirror display and hand-held video
projector. For the category of head-attached display the necessity existed to explain the
underlying concept. For the now treated displays these are assumed to be known.
Taking this into account, only examples of hand-held displays applied in AR are given.
Thereby needs to be noted that the presented examples only serve as illustration; they
are not unique in their kind.
But first are discussed the difficulties to overcome when using hand-held displays for
AR [Bim04a]. The image processing suffers from high delays and a low frame rate
when PDAs or mobile phones are used, since limited processor capacity and rendering
components are available. The cameras nowadays installed in mobile phones offer a
fixed focus, which only makes them effective in a certain depth range. A comparable
difficulty is related to hand held-hand video projectors; only projection at a certain
depth range is supported. Additionally, all hand-held displays do not provide completely
10

Development of an Augmented Reality system using ARToolKit and user invisible markers
hands-free working, as compared to head-attached displays. The FOV also is restricted
by the limited screen size of hand-held devices.
2.2.2.1 PDA
In [Bell02] is described an AR system based upon PDA technology, which provides
information at a glance to aid a mobile user to explore an unfamiliar environment.
Labels are overlaid in such a way that they are automatically positioned and sized to
avoid overlapping. Labels are connected to its belonging object by arrows. Users also
can ask for additional information concerning an object. A pop-up window supplies a
textual description of a selected object together with a close view of the object. Figure
2.9 and Figure 2.10 show two PDA images of the system in use.
Figure 2.9 Annotations on a PDA Figure 2.10 Pop-up window on a PDA
2.2.2.2 Mobile Phone
In [Sin04] is shown the use of standard mobile phones readily available in the consumer
market as user interface for AR. Such devices are camera equipped and are Bluetooth
enabled. Captured images are communicated to a server through the Bluetooth protocol.
The server carries out the necessary image processing tasks for augmentation. This
augmentation is based upon the recognition of fiducial markers. Several markers in a
scene can be overlaid concurrently with three-dimensional virtual objects. Two such
examples are shown in Figure 2.11 and Figure 2.12.
Figure 2.11 Recognition of several markers Figure 2.12 User-dependent augmentation
2.2.2.3 Hand-held mirror displays
In [Loo04] is discussed an interaction technique for AR based upon the application of
hand-held magnifying glass. Since the lenses have the property to be semi-transparent
they can be applied as an interface element to change the representation of data shown
11

Development of an Augmented Reality system using ARToolKit and user invisible markers
beneath it. Two types of lenses are available; volumetric and flat lenses. Fundamental
tasks that can be carried out using either of those lenses are magnification, information
filtering and object selection and manipulation. In Figure 2.14 and Figure 2.15 are
shown the application of volumetric and flat lenses for magnifying a certain area within
an AR environment. Figure 2.13 shows the initial virtual scene.
Figure 2.13 A virtual scene Figure 2.14 Volumetric lens Figure 2.15 Flat lens application
application
2.2.2.4 Hand-held video projectors
In [Ras03] is described an AR system that uses hand-held projectors for the
augmentation of objects. Object recognition is done through fiducials that are attached
to objects of interest. They do not only provide object identity, but are also used to
determine the position and orientation of the camera. Since the system is fully calibrated
this information can be related to the position and orientation of the projector. In this
way content can be overlaid on an object as required. Figure 2.16 shows the use of a
hand-held video projector for guiding a user through an environment containing storage
bins.
2.2.3 Spatial displays
Figure 2.16 AR using a hand-held video
projector
The two before described main categories of display devices take a body-attached
approach. Also the possibility exists to detach technology from the user, and have most
of it integrated in the environment. This is the way spatial displays try to offer
possibilities for AR. The augmentation of the environment can be done in three ways;
using video see-through, optical see-through or direct augmentation. Depending on
which is used a different kind of spatial display is created.
12

Development of an Augmented Reality system using ARToolKit and user invisible markers
2.2.3.1 Screen based video see-through displays
For this category of display devices a video camera is used for capturing a scene. Videomixing
augments the capture video with virtual objects. The resulting merged images
are presented to the user by using a regular monitor. This approach is sometimes also
referred to as a “window on the world” setup. It offers a very efficient way of
implementing AR since only a standard PC and off the shelf components are needed.
But this also introduces a number of disadvantages concerning the use of screen based
AR [Bim04a]. Due to the relative small monitor sizes, a small FOV is supported. Also
the resolution of the merged images is a limiting factor. Usually direct interaction with
the real environment and the augmented images is not supported. Figure 2.17 shows a
typical setup, and Figure 2.18 shows an augmented dinosaur’s movement simulation.
Figure 2.17 Typical window on the Figure 2.18 Augmented simulation
world setup
2.2.3.2 Spatial optical see-through displays
This type of display makes use of spatial optical combiners that are physically
positioned in the environment. Those are able to generate images that are aligned within
the environment. Different kind of optical combiners exist that could be taken
advantage of; mirror beam splitters, transparent screens and optical holograms. If a
mirror beam splitter is used a choice exists between a curved and a planar one.
Holograms capture a scene by recording amplitude, wavelength and phase information.
This allows for reconstructing complete optical wavefronts. This results in the captured
scenery having a three-dimensional appearance that can be observed from different
perspectives [Bim04b].
The image of the real world and the overlaid image are visible at the same time. One
property these optical combiners have in common is that they do not follow the
movements of the user but rather support moving around them. Below are shown the
three kinds of optical combiners.
Disadvantages related to the described optical combiners are the following [Bim04a].
Since the optics and the display technology are spatially aligned mobile applications are
not supported. Direct interaction with the real and virtual objects that are located behind
the optics is prevented by the nature of the used optics. The limited size of the screens
13

Development of an Augmented Reality system using ARToolKit and user invisible markers
and optical combiners causes unnatural cropping of virtual objects outside the display
area.
Figure 2.19 Beam splitter Figure 2.20 Transparent screen Figure 2.21 Optical hologram
2.2.3.3 Projection based spatial displays
This approach is not using a plane or surface to project an image on, but instead, the
images are directly projected on the surface of physical objects. Images can be projected
using one single static projector, but for a greater display area also a steerable projector
or even multiple projectors can be applied. Figures 2.22 through 2.24 show the use of a
single static projector for augmentation of a real object with projected images [Ras02].
Figure 2.22 Hardware setup Figure 2.23 Real object Figure 2.24 Augmented object
Some issues related to the use of head-attached displays are resolved by projection
based AR [Bim04a]. The FOV is theoretically unlimited, and a scalable resolution is
supported. Besides the usability factor is significantly improved; ergonomics are paid
more attention to and eye accommodation is relieved.
But also several new problems are introduced. The complexity of consistent geometric
alignment and colour calibration increases with the number of applied projectors.
Conventional projectors can only focus on one single focal plane. When non-planar
surfaces are used for projection images blur is caused. The size, shape and colour of the
physical objects´ surfaces limit the display area. Since front-projection is utilized
interacting users and physical objects cause shadows. Usually projection based AR only
supports one single user.
2.3 Tracking
For an AR system to be able to give the user the idea of seeing the real and virtual
images as fused it is required to correctly overlay virtual objects on the real world. That
is, virtual objects must be accurately registered in all dimensions. The inability to
establish this can result in negative effects ranging from distraction to a completely
unusable AR system. For an AR entertainment system incorrect registration may
14

Development of an Augmented Reality system using ARToolKit and user invisible markers
somewhat lower the experienced fun, but it may have disastrous results for a heart
patient during an operation supported by an AR system. It is not only important to
provide correct registration when the user is in a fixed position, but it also is to be
maintained during movements of the user. For establishing this, the position and
orientation of the user’s head need to be tracked.
This refers to the problem of head tracking. Another problem is related to the tracking
of used tools. This is required for the purpose of interaction within the augmented
scene. The physical instrument’s position and orientation need to be measured for
determining the target and context of interaction. Not for all AR environments this has
an equal importance. In an AR painting application for children the erroneous colouring
of the virtual frog instead of the virtual princess may give some surprise, but one may
think of more impact when something comparable occurs in the AR surgeon room.
Tracking can be divided into two types of tracking systems; outside-in and inside-out
systems. The distinction between those two is based upon the configuration of sensors
and emitters in relation to the tracked objects. Outside-in systems have their sensors
mounted at a fixed location in the scene. The objects to be tracked are equipped with
emitters or landmarks. On the other hand, inside-out systems use a direct way by having
their sensors directly attached to the tracked objects.
Tracking in AR have a set of stringent requirements that need to be met to be really
useful in practice [Pin02]. These are the following:
• High 6 Degree Of Freedom ( DOF) spatial accuracy in position and orientation
• No (very little) jitter, i.e. noise in the output of the tracking system
• High update rates (min. 30 Hz, better several 100Hz)
• No (very little) lag, i.e. delay from measurement to tracker output
• Full mobility of the users (no cables, no restricted volumes of operation)
For AR tracking several approaches have been tried. Each of those has their own
strengths and weaknesses; to combine strengths and/or to compensate for weaknesses
hybrid approaches are used. Now will be shortly discussed five different approaches
that have been proposed for tackling the tracking problem in AR [Bru03]. For each
approach one example is given. It needs to be said that these examples are not unique;
there are available more. To complete the discussion hybrid approaches, outdoor
tracking and collaborative AR are shortly mentioned.
2.3.1 Inertial tracking
Accelerometers and gyroscopes are applied for determining the acceleration for position
determination, and the orientation for position orientation. These can not be measured
directly, but are deduced by using secondary measures. Voltage values are related to the
displacement mass and the rate of change of direction. These provided voltage values
are a source of error; they are influenced by noise, bias errors and quantization error.
Integration is applied once for deriving the desired position, and twice for determining
orientation. This process is subject to drift which introduces the need for periodical
recalibration.
15

Development of an Augmented Reality system using ARToolKit and user invisible markers
Inertial tracking does not suffer from magnetic interference problems and for it to
function no line of sight is needed. Additional advantages are the fact that it senses
orientation in a fast and direct way and that inertial tracking does not put a limit on the
range over which it can be used. However, the method is not accurate for slow position
changes. Also position and orientation are only measured in 3 DOF.
Xsens Motion Technologies have developed an inertial tracker, named MTx, that
provides 3D orientation and kinematic data; 3D acceleration, 3D rate of turn and 3D
earth magnetic field. The tracker has a highly compact design, as shown in Figure 2.25.
2.3.2 Acoustic tracking
Figure 2.25 Mtx inertial tracker
This kind of tracking devices usually applies ultrasonic sound waves for carrying out
their task. This is sound that has a frequency above 20 kHz, which lies above the
frequency range of the human ear. Using only one transmitter/receiver pair provides a
distance measurement from the target to a fixed point. For providing information on a
3D position either one receiver and three transmitters or three receivers and one
transmitter are needed. Three transmitters and three receivers are employed for
estimating position and orientation. The method employed for estimating distances is
converting the time to receive a signal to a distance.
Acoustic tracking offers a lightweight, inexpensive tracking solution that is robust
against distortion by magnetic interference. Besides the advantages some disadvantages
attached to the technology can be identified. Acoustic interference influences the
ultrasonic receivers. Also echoing of waves is a source of distortion. Low accuracy is
established since the speed of sound in air is not constant but varies with the
environmental conditions. The constant need for an available line of sight reduces the
reliability of tracking.
Intersense have developed an acoustic tracking system that uses sonic emitters and discs
for sending out and reflecting ultrasound. A processor is used for timing the return
pulse. The tracking device is encompassed within a solid box, the Intersense IS-900,
shown in Figure 2.26.
Figure 2.26 Intersense IS-900 acoustic tracker
16

Development of an Augmented Reality system using ARToolKit and user invisible markers
2.3.3 Magnetic tracking
Magnetic tracking tries to take advantage of magnetic fields for measuring range and
orientation. Either low frequency AC fields or pulsed DC fields can be used. In the
transmitter a set of three coils wound around a magnetic core is incorporated for
generating magnetic fields. A receiver also has three coils in the same configuration. In
one cycle three measurements of magnetic field strength are taken for each coil. These
nine measurements are used for determining the position and orientation of the receive
coils relative to the source coils.
Magnetic tracking does not suffer from line of sight problems. A large range is
supported by magnetic tracking, while it still provides accurate measurements.
Disadvantages however take some glance off the great potential of this type of tracking.
Electromagnetic interference and field distortion caused by radios and metal surfaces
have their influence on the magnetic fields; magnetic tracking is prone to distortions.
The high accuracy diminishes with increasing distance. High latency also is related to
the use of magnetic tracking.
The Polhemus Fastrak system estimates the position and oientation of a tiny receiver as
it moves through space. One unit contains all the components needed for the tracker to
carry out its function. Up to four receivers are supported simultaneously. Figure 2.27
shows the tracking unit with its connections and one receiver.
2.3.4 Mechanical tracking
Figure 2.27 Polhemus Fastrak magnetic tracker
Mechanical trackers measure joint angles and lengths between joints. One known
position is sufficient to derive all the other absolute positions from relative joint
measurements. Mechanical tracking systems either are ground-based or body-based. In
the first approach one point of the tracker is attached to the ground at a known position.
The second approach usually implements its function in an exoskeleton.
Mechanical tracking is not influenced by magnetic interference and line of sight
problems. It provides low lag, accurate measurements of position and orientation. It
provides a good way for tracking small volumes. But, also the intrusiveness of
mechanical trackers needs to be annotated. Physical range of motion can be limited and
prohibited.
Gypsy4 is a human motion tracking system, that applies 43 motion sensors placed
around 17 human joints. Multiple users can be tracked by the system at the same
17

Development of an Augmented Reality system using ARToolKit and user invisible markers
moment. They need to wear an exoskeleton which contains the motion sensors. Figure
2.28 shows a user wearing such an exoskeleton.
2.3.5 Optical tracking
Figure 2.28 Gypsy4 exoskeleton
For optical tracking light is used to measure angles. Each point in the image plane
provides a ray from that pixel to the centre of projection. As the distance between the
transmitter and receiver increases, the optical energy diminishes with the square of that
distance. As detector can be used video and Charged Coupled Device (CCD) cameras,
or lateral photodiodes. This depends on the type of targets that is used; passive or active.
Passive targets, such as reflective materials or fiducial markers are not powered. On the
other hand, active targets such as LEDs are powered. Optical tracking systems are also
referred to as image-based systems, as recorded images from a target are used for
determining position and orientation.
Optical tracking provides high accuracy at a fast rate, which is not limited by a large
area. Magnetic interference does not have any influence. However, line of sight
problems are present. Intensity and coherence of light sources are a limiting factor for
acquiring good quality images which are the input for optical tracking.
In ARToolKit is implemented an optical tracking system that uses fiducial markers for
calculating the position and orientation of a camera. This is discussed in more detail in
the next chapter.
2.3.6 Hybrid tracking, outdoor tracking and collaborative AR tracking
All of the before described tracking approaches have their own weaknesses and
strengths. To compensate for this and to combine advantages, methods for tracking can
be combined. This is the desired perspective but still some difficulties need to be
overcome to integrate them. Increasing system complexity is a resulting fact from
integration. But it results in a more accurate and more robust way to establish tracking.
The described tracking technologies are shown to work in constrained indoor
environments. Such environments can be completely prepared, are limited in size and
can be controlled. Outdoor environments have opposing properties; they are unprepared,
are principally unlimited in size and do not offer possibilities for modification. Also
18

Development of an Augmented Reality system using ARToolKit and user invisible markers
fewer resources are available; power, computation and sensors are examples of those.
For indoor applications hybrid tracking offers possibilities but this holds even more for
outdoor AR. Also the Global Position System (GPS) could be taken advantage of,
although this is not a very accurate technique. Usually a combination of inertial and
optical tracking is applied.
In collaborative AR users are collaborating within one single environment. In such a
situation each user has its own view on the environment. Also each user separately can
interact with objects in the environment. This aspect adds one dimension to the tracking
problem; that is, head tracking and tool tracking needs to be done for multiple users.
Nevertheless, the before described tracking methods can be applied.
2.4 Registration
The previous section described the tracking problem, composed of both head tracking
and tool tracking. These both are meant for being used for another aspect that is present
in AR. This aspect is presented as the registration problem, and is concerned with the
alignment between virtual and real elements. This is a mostly important and
complicated matter for three reasons. First of all, the human visual system is very
sensitive to small misregistrations, even if they are in the order of few pixels. Second,
for AR applications tighter requirements are put upon on error tolerances that are
allowed. To make things even more complex, one also has to cope with system delay;
the time interval that lies in between tracking and the actual superimposing of the
corresponding images.
Figure 2.29 and Figure 2.30 show the result of an extreme misregistration compared to
the correct registration of a virtual car against the real background environment.
The accuracy of registration of virtual objects is determined by the alignment errors that
result. The possible alignment errors can be classified in two categories [Azu97]:
• Static errors, which are defined as the misalignment the user can perceive when
he is immobile.
• Dynamic errors, which are defined as the misalignment the user can perceive
when he moves.
Dynamic errors are by far the greatest contributors to registration errors perceived by
users. Although static errors have less impact they can not be ignored. There are various
sources of error that can be pointed out. For static errors the most important are optical
distortion, errors in the tracking system, mechanical misalignments and incorrect
viewing parameters. Dynamic errors are caused by unavoidable system latency.
19

Development of an Augmented Reality system using ARToolKit and user invisible markers
Figure 2.29 Misregistration of a Figure 2.30 Correct registration of a
virtual car virtual car
2.4.1 Static errors
Now will be shortly discussed the four sources of error responsible for causing static
errors. Also are mentioned ways to decrease their effect.
Optical distortion is an inevitable property which is exhibited by most camera and lens
systems. Also cameras for recording a real scene and the optics used in display devices
are influenced by this. Usually this distortion is a function of the radial distance from
the optical axis. But also non-linearity may exist in these distortions. The virtual overlay
images are distorted under influence of this, and this causes static registration errors
when they are mapped on top of the undistorted view of the real world.
Optical distortions can be mapped and compensated, since they are usually systematic
errors. For doing this, two options are available; additional optics can be used or the
compensation can be done digitally. Adding optics increase weight, which is often not
desirable. Digital compensation adds a portion to the system delay because it is a
computational expensive method.
Since tracking devices provide the basic input for registration of virtual objects, errors
in the tracking system are the most serious type of static registration errors. These
distortions are not easy to measure, let alone eliminate. Often they exhibit a nonsystematic
behaviour and can not fully be characterized. All of the before described
ways of tracking provide an accuracy which is not adequate enough. Ongoing
developments in the area of tracking methods need to increase the ability to track more
accurately.
Mechanical misalignments are discrepancies between the actual physical properties of a
real system and the model or specification of the hardware components used for
building it. These discrepancies result in subtle changes in position and orientation of
projected virtual images. It is difficult to compensate for this, although some alignment
errors can be calibrated. The best solution would be to have a closer resemblance
between the actual physical properties and its corresponding model or specification.
Improvements in alignment techniques for constructing hardware are expected to
decrease these discrepancies.
Incorrect viewing parameters can be seen as a case of alignment errors where
calibration techniques can be applied. Viewing parameters specify how to convert
between reported head or camera locations and viewing matrices used for generating the
20

Development of an Augmented Reality system using ARToolKit and user invisible markers
virtual images. Systematic static errors in registering virtual objects result from these
parameters being incorrect. However, there exist several methods for adjusting them.
The viewing parameters can be estimated by making minor manual adjustments
according to a non-systematic fashion. This does not give satisfying results. Another
approach is to directly measure the viewing parameters by using a variety of measuring
tools and sensors. Up until now, this has not lead to great success in establishing correct
viewing parameters. Yet another approach is using view-based tasks for setting up
geometric constraints, which are used for determining the viewing parameters. It
heavenly depends on the user accurately performing the specified task and the quality of
the used tracker.
2.4.2 Dynamic errors
Dynamic errors are caused by the existence of system delays, or lags. These delays are
related to the fact that each and every component in an AR system requires a certain
amount of time for carrying out its function. These all add up to the end-to-end system
delay; the time difference between the moment that the tracking system measures the
position and orientation of the viewpoint to the moment when the generated images
corresponding to that position and orientation appear to the user. Dynamic errors only
occur when the user is moving. Several methods to reduce the impact of dynamic
registration errors have been tried; reducing system lag, reducing apparent lag, matching
temporal streams and predicting future locations. These are shortly explained now.
Reducing, or ideally completely eliminating, system lag would be most direct and
effective way to avoid dynamic errors. The current state of the technology is not capable
to eliminate system delays. It is also not expected that this will ever be the case. Then,
at least can be tried to minimize them to the smallest amount possible. This is to be
accomplished by the ever ongoing technological development.
Reducing the apparent lag can be done using image deflection and image warping
techniques. Those only can be applied to systems that depend on head orientation.
Image deflection is a technique that incorporates more recent orientation measurements
into the late stages of the rendering cycle. This results in the virtual images being
displayed with less registration error. Also image warping might improve the
registration by making small adjustments in orientation and translation.
In video see-through AR a delay is introduced in the user’s view on the real world. It
offers possibilities to match the temporal streams of the real and the virtual images. This
is done by adding delay to the video stream to synchronize it with the generation of the
virtual images. The process of matching temporal streams is done dynamically.
Predicting future locations is the last method that could be used to reduce dynamic
errors. The predicted locations, instead of the measured locations, are used as reference
for the virtual images that are overlaid. For short system delays this is shown to reduce
registration errors by an order of magnitude. The prediction is required to be done in
real time. More work is needed to develop better performance prediction and models
that more closely match actual head motion.
21

Development of an Augmented Reality system using ARToolKit and user invisible markers
2.5 Calibration
Tracking is used for providing the input needed for the correct registration of virtual
objects with respect to the real world. But before these two connected processes can be
carried out, another issue which is related to both needs to be handled. That is,
parameters of used display components in the AR system need to be determined. The
complete set of procedures for estimating these parameters is called calibration.
This is done for establishing the viewing projection of the used camera. From that
information, correct transformations for projecting virtual objects on the real world are
established. These transformations are meant to mimic the intrinsic and extrinsic
parameters of the virtual camera. It is necessary to have the parameters of the real
camera and the virtual camera to coincide for projecting both real and virtual objects in
a similar way.
In most cases direct interaction of the user is required for performing calibration. This
precision of calibration is clearly user-dependent. Also calibration is a time-consuming
task and needs to be repeated in advance each user session. To avoid this there have
been developments aiming at calibration-free AR. This does not require the user to do
manual calibration. Instead an affine mapping is constructed based upon the positions of
tracked fiducials. Autocalibration is another method for reducing calibration
requirements. Then redundant sensor information is used for automatically measuring
and compensating for changing calibration parameters.
2.5.1 Manual calibration
Manual calibration has been the first approach for tackling the calibration problem.
Because of its maturity a range of calibration algorithms have been developed. Some of
those use special equipment, but also more convenient methods exist. What they have in
common is the division in three steps. First are obtained the 3D coordinates of
calibration points in the world coordinate system. The corresponding 2D points in the
image plane are determined. From these data the transformation matrix is constructed.
This basic model does not take into account other factors, such as radial distortion
caused by optical elements. However, methods that also compensate for that also exist.
There are differences between calibration methods for video see-through and optical
see-through HMDs. Calibration of a video see-through HMD can be done by using
image processing techniques to determine transformation matrices from relations
between real points and their projected counterparts. That procedure will not be
discussed here, but is explained in the following chapter on ARTooKit.
For optical see-through HMDs this can not be used since a video stream is not available.
Users will have to match real points in space against its virtually projected point. In
[Azu94] is described an algorithm for manual calibration of optical see-through HMDs.
The real wooden frame used for the calibration is shown in Figure 2.31. The virtual
objects to be calibrated against this real object are three mutually orthogonal lines,
forming a coordinate system. Figure 2.32 shows the alignment of the virtual axis with
the real box. Four parameters are measured; location of the frame, apparent centre of
the virtual image, transformation between tracker space and eye space, and FOV. This is
22

Development of an Augmented Reality system using ARToolKit and user invisible markers
used for calculating a view matrix which specifies the relation between the real box and
the virtual coordinate system.
Figure 2.31 The wooden calibration box Figure 2.32 Calibration of a
virtual axis with the calibration
box
2.5.2 Calibration-free AR
Calibration-free AR, as proposed in [Kuk98], is based upon the observation that the
projection of all the points in a set can be expressed as a linear combination of just the
projections of four 3D points taken from that set. This results in an affine virtual object
representation relative to the reference frame defined by the four points in space. No
metric information about the objects in the camera’s view or the calibration parameters
of the camera is needed. Also dynamically changing camera parameters are
compensated for, by constantly updating the viewing matrix. Two images from different
camera positions are taken, in which four predefined points are tracked. This suffices to
construct the affine mapping of virtual objects. These points can be preselected fiducial
markers, or uniform-intensity regions in the real scene.
2.5.3 Autocalibration
This third manner of calibration makes use of redundant information for the calibration
process. In [Hof00] is presented an approach to calibrate an electronic compass in an
outdoor situation by using additional rate gyroscopes. Since the electronic field of the
earth is influenced by distortions, which also vary with the geographic position, it is not
sufficient to calibrate the electronic compass for a single location. For solving this, a
real time procedure is used for constantly recalibrating the compass. This is done by
taking into account measurements from the rate gyroscopes to adjust the estimates given
by the compass. It was shown that autocalibration converge to solutions obtained by
manual calibration using a magnetic turntable.
2.6 Interaction
The foregoing described the three basic aspects necessary for creating AR. But, to give
it real potential, the user must have the opportunity to interact with the AR environment.
In [Bow04] four interaction styles for AR, each offering different interaction levels are
mentioned. Those will be shortly discussed now, in order of increasing interaction. It
needs to be pointed out that interaction styles for AR in principle are related to the
display devices that are being used. The changing state of technology is expected to
expand applicability of interaction styles to other display devices as well.
23

Development of an Augmented Reality system using ARToolKit and user invisible markers
2.6.1 3D Data browsing
The primate idea for the application of AR was to use it for superimposing virtual
objects on the real world. The overlaid information can be three-dimensional, but can
also be textual, in voice form or video annotation. This allows a mechanic to have an
augmented view showing repair instructions, or to guide users through an unknown
environment showing virtual direction pointers. The interaction between the user and
the AR environment is limited to navigation within the overlaid information space.
There exist no means for manipulating, modifying or interaction with both the virtual
and real world. Although 3D data browsing can be supported by several display devices,
many of the AR applications that are guided by this interaction style use an HMD in
combination with 6 DOF optical or magnetic tracking.
2.6.2 3D AR interfaces
To offer the user possibilities for interaction within AR environments 6 DOF input
devices can be used. These devices are able to support natural and familiar interaction
metaphors. To have knowledge on the target and context of interaction it is necessary to
track their position and orientation. This allows seamless spatial interaction, that is, a
user can interact with virtual objects anywhere in the physical environment by using
special-purpose input devices. But for interaction with physical objects real hands are
used. These different input modalities lead to a violation of the natural workflow.
Another shortcoming of this interaction style is the lack of sufficient tactile feedback,
which is a stringent requirement for many real world applications.
2.6.3 Augmented surfaces
This interaction style is related to the before describe spatial display technique. Virtual
3D objects are only registered to selected work surfaces that allow user to interact with
them through familiar and traditional tools, such as a pen. But also specifically designed
physical handles could be used. The interaction with physical objects is known as
tangible interaction. One desirable property is that one single input modality is used for
accessing both the virtual and the real world. Compared to the 3D AR interfaces
interaction is significantly eased, but it introduces a spatial reduction in interaction. Full
interaction in three dimensions is nearly impossible, and is limited to the 2D augmented
surface.
2.6.4 Tangible AR
Tangible AR tries to combine the advantages of 3D AR interfaces and augmented
surfaces; it offers a way of undisturbed spatial interaction using only one input
modality. Usually a video see-through HMD is used for implementing this interaction
style, combined with optical registration. Virtual objects are registered to markers,
physical objects that contain a known pattern. Interaction with the virtual objects is
done through manipulating the physical, tangible interface elements. Also a paddle, a
paper object with an attached marker could be used. This provides a transparent
interface for two-handed 3D interaction with both the virtual and real objects in the
environment. Also this interaction can take place anywhere in the physical environment.
24

Development of an Augmented Reality system using ARToolKit and user invisible markers
2.7 AR applications
Now that has been presented all the building blocks needed for creating AR, the
moment has arrived to take a closer look at application areas where AR has already
been applied. There are several areas imaginable for AR, ranging from the most serious
and demanding areas to the more entertaining and less requiring ones. In [Val98] ,
[Azu94] and [Azu97] are identified several areas; the aggregation of those results in the
following list:
• Medical
• Military
• Education
• Engineering design
• Manufacturing, maintenance and repair
• Entertainment
As stated in the introduction of this chapter, mostly HMDs are used for giving the user
an augmented view on the real world. This also results from the carried out inventory of
known AR applications. For the tracking subsystem usually optical tracking or a hybrid
tracking approach is utilized. Tracking using GPS only is important for outdoor AR.
The way interaction is implemented, completely depends on the posed application
requirements. Surgeons would need an interaction mechanism comparable to the way
medical instruments are used. But for soldiers in the battle field a non-obstructing
manner of interaction is required.
All this may result in different hardware setups for each mentioned area. Following are
described AR systems that are taken from the six identified areas. These systems are
either considered to be representative for their corresponding area, or are most
innovative and generally known within the AR community. Hereby it is obviously
unnecessary to note that this small selection can not give a full intersection of each area.
2.7.1 Medical
In the medical area there can be distinguished four types of applications for AR; image
guided surgery, surgery planning, surgical training and psychiatry. Here will shortly be
discussed which possibilities exist for AR. For illustration, also in each category, a
known AR system is presented. Thereby the focus will be on the before discussed
topics; display technology, tracking, calibration, registration and interaction.
2.7.1.1. Image guided surgery
Minimal invasive surgery and interventions promise advantages for the patient, such as
reduced trauma, faster recovery and better cosmetic result. This comes at a cost of a
restricted view of the region of intervention for the surgeon. Augmented reality can be
used to minimise this or even overcome this lack of visual perception. Medical data can
be overlaid on a patient for guiding a minimal invasive operation. Many medical areas
lend themselves for this concept; brain, heart, liver, spine and knee surgery are just a
small sample taken from the full range of possibilities. Some of those areas have more
practical demands because of moving and deforming parts.
25

Development of an Augmented Reality system using ARToolKit and user invisible markers
AR systems in image guided surgery roughly take the same approach. That is, first twodimensional
data of the patient is acquired. These data are transformed into a threedimensional
model. Secondly, there needs to be a way to register the real patient.
Tracking of the patient, instruments and visualization displays also need to be taken
care of. Finally, the created three-dimensional model is overlaid on the patient to
provide the surgeon with an augmented view. Each AR system implements these steps
in a different way. Now will be shortly discussed one such system, resulting from the
MEDARPA project [Sch04].
The main component of the system is a transparent display attached to a movable swivel
arm. The technology consists of a 17” TFT screen allowing a resolution of 1024x1024,
at a resolution of 75Hz. A working area of about two square meters is supported. The
transparent screen is used for overlaying acquired CT or any other 3D imaging modality
on the body of the patient. The surgeon can interact with the patient in the normal way
using medical instruments. To overcome the lack of stereoscopic view, colour coded
feedback is used for guiding the surgeon. Tracking is supported by a hybrid tracking
approach consisting of electromagnetic and optical tracking. The position and
orientation of the patient, medical instruments, the display and the surgeon’s viewpoint
are continuously tracked. For registering the patient a registering device is used based
upon spherical markers which are attached to the patient’s body. Figures 2.33 through
2.35 show the system setup, its interaction mechanism and registration procedure.
Figure 2.33 MEDARA Hardware Figure 2.34 Interaction Figure 2.35 Registration of the
setup through a medical instrument patient
2.7.1.2. Surgery planning
Not only offers AR possibilities for supporting the surgeon during an operation by
image guided surgery, but it also can be used in the stages before an actual operation.
This activity is known as surgery planning, which allows for predetermining optimal
decisions, and thus for staging of operations. This relieves the surgeon during the
operation, by not having to consider the next step or action to be carried out. As is the
case with image guided surgery, AR for surgery planning, is also not limited to a small
range of the medical area. It can be extended to any imaginable medical problem, given
that it can be visualized in three dimensions. This is done through building up a 3D
model from 2D CT, MR or ultrasound data. Now will be shortly discussed one such AR
surgery planning system, taken from the area of livery surgery [Sor04].
26

Development of an Augmented Reality system using ARToolKit and user invisible markers
For the visualization of the augmented images an optical see-through HMD is used,
which gives stereoscopic view. For determining the position and orientation of the head
of the user the HMD is tracked. The used input devices, a pencil and a transparent
plexiglass panel, are also constantly being tracked. An infrared camera attached to the
ceiling is tracking attached infrared markers. Interaction is established through the use
of the pencil that supports several input events. Figure 2.36 shows the outlook of the AR
system. Visual inspection of 3D liver models can be done from different angles, and
rendering parameters can be dynamically changed. Segmentation refinement is also
supported to edit and refine the 3D models. To support the resection planning process,
volume calculations of remaining liver and tumour are provided. After a resection is
planned a resection plan may be elaborated based upon information from the
visualization. This process of interaction is shown in Figure 2.37.
Figure 2.36 Liver planning system Figure 2.37 Resection planning through interaction
outlook
2.7.1.3. Surgical training
AR offers excellent possibilities to be used in surgical training. Surgeons can be taught
operation techniques in a very efficient and cost-effective way. There is no need for
costly tissue or organs to be physically present; AR can replace their absence. Besides,
numerous repetitions can be done over and over again. Also collaboration between
several persons is supported, to train several persons concurrently.
2.7.1.4. Psychiatry
There exist many examples of successful VR treatment of phobias; examples are
acrophobia, agorophobia and claustrophobia. The obtained results are satisfying, but the
patient is fully immersed in the environment. Here lie great possibilities for AR,
because it does not fully immerse the patient, but only virtually represents the fear. In
most of the areas in which VR has been shown to be successful also AR can be tried.
In [Jua04] is presented the first AR system for treating fear of small animals, in this case
spiders and cockroaches. The system is based upon the vision library ARToolKit, which
provides support for the optical tracking of fiducial markers. This toolkit also
incorporates an algorithm for camera calibration. The visualization of the augmented
environment is done by using a video see-through HMD. The user can interact with the
virtual cockroaches by using real instruments as a fly swatter and cockroach killer.
Tracking of those instruments is done by tracking attached fiducial marker. The
appearance of virtual cockroaches is based upon the recognition of similar fiducial
27

Development of an Augmented Reality system using ARToolKit and user invisible markers
markers that are placed in the environment. Figure 2.38 shows the view as seen through
the system.
2.7.2 Military
.
Figure 2.38 Virtual cockroaches for phobia treatment
The military is an area where a huge amount of technology is incorporated. This will
make battlefield situations growingly complex. Also, future battlefields are expected to
be in urban areas. Despite this overwhelming presence of technology the soldier of the
future is expected have a lack of situational awareness. AR can help to improve this
aspect significantly by providing ways for supplying real-time information overlaid on
the view of the world. The difficulties that need to be faced are factors concerning
mobility, usability and availability.
That AR really is expected to have full potential, speaks from the fact that the military is
actively participating in research. One project in particular is the BARS project [Jul00],
sponsored by the Office of Naval Research. Future projects concerning the same topic
are probable to take a somewhat same approach, concerning system setup. For making
available information to users, a database system containing all the important
information is compiled in advance. Communication is done through a wireless
protocol. To avoid an overload of information being displayed to the user information
filtering is used. The displaying technology used is a calibrated optical see-through
HMD. Tracking is done by using a hybrid approach using both GPS and an inertial
tracker. The other necessary hardware is carried on the back. Figure 2.40 and Figure
2.41 show the hardware setup and a view augmented with a virtual map.
Figure 2.39 Mobile hardware setup Figure2.40 Augmented view with an overlaid
virtual map
28

Development of an Augmented Reality system using ARToolKit and user invisible markers
2.7.3 Education
AR might have the ability to change traditional education methods. As said on the
subject of surgical training, it offers an effective teaching method. Besides, it can
visualize concepts which may lead to an improvement in student’s comprehension.
Another import aspect is that interactivity is added to the learning process. This
especially is important concerning education methods for children.
From the literature there are known a number of examples of using AR in education. It
has been applied to the teaching of geometry, spatial relationships between planets, and
molecule structures. What these examples have in common is the visualization of an
abstract concept. An AR system as a three-dimensional geometry construction tool is
described in [Kau03]. The focus lies on the collaborative aspect between students and
teachers. The system has been built and tested in several hardware settings, but here
only is treated one such setup. Users are wearing an HMD, and the used interface
elements are a pen and a pad. Tracking of both the HMD and the interaction devices is
done by optical tracking of retro-reflective markers. Figure 2.41 shows two users in a
collaborative attempt to construct a geometry proof.
2.7.4 Engineering design
Figure 2.41 Collaboration in geometry construction
Nowadays engineering design is usually supported by computer-aided design. Although
this offers great advantages over traditional engineering, it still suffers from
visualization problems and lack of collaboration support. AR could fill in this gap by
offering a way to visualize CAD models that are not only tangible, but also can be
interacted with. Several designers or potential clients can view such models
concurrently from different angles. Usually augmented surfaces in combination with an
HMD are exploited for the visualization. In [Pen04] is reported a representative AR
system. Figure 2.42 shows the collaboration of two designers working on a virtual city
plan.
29

Development of an Augmented Reality system using ARToolKit and user invisible markers
Figure 2.42 Collaborative augmented design
2.7.5 Manufacturing, maintenance and repair
In these three related areas AR basically can offer comparable functionality by
overlaying information on the real world for guiding the underlying processes. AR is
used to replace and provide in real time the information commonly found in instruction
manuals. This can be annotations of parts of a machine, instructions for assembly or
disassembly, and visualization of hidden inside views. The display technology expected
to be used most often is an HMD. One illustrative project in manufacturing, which
augments the task of door lock assembly, is discussed in [Rei98].
An optical see-through HMD is used, in combination with a small camera for optical
tracking. For the tracking small patterns of cardboard were attached to the door of a car.
Calibration of the HMD is done manually. For providing overlays data from a CAD
system are used. Figure 2.43 shows the used visualization HMD, and Figure 2.44 shows
an augmented view with screwing instructions.
Figure 2.43 HMD, with attached camera Figure 2.44 Overlaid repair instructions
2.7.6 Entertainment
Entertainment is an area where AR already has been accepted and extensively used.
Most areas have been inspired by the television industry; advertising, special effects and
television broadcasts. Thereby the focus lies on the visual aspect; advertisements are
made more appealing by augmentations, special effects are created for various means,
and weather broadcasts basically show an augmented view to the viewer. Another area
is the area of gaming which is considered to be the most spectacular and challenging
one. AR can revolutionary change consisting traditional concepts of games; board
games, strategic games, puzzle games, and even outdoor gaming are suitable for
applying AR.
30

Development of an Augmented Reality system using ARToolKit and user invisible markers
One traditional and appealing form of entertainment is plays, shows and movies shown
in theatres and cinemas. There exist varying degrees of interactivity and augmentation
between those forms of entertainment. AR is able to dramatically increase those two
aspects concerning entertainment. A typical example of an AR entertainment system is
discussed in [Koy03]. It can be used for displaying real time 3D video to multiple
viewers. The type of display that is utilized for presenting the video is a regular
computer monitor. Camera calibration is done by using a mobile land-mark bar and
laser surveying instruments. Multiple cameras are used for capturing a large-scale
space; in the presented showcase a football stadium. An image processing algorithm
extracts textures from the video footage. These real images are combined with a virtual
stadium background. Figure 2.46 shows two such images taken from an augmented
football match scene.
Figure 2.45 An augmented football match scene
31

Development of an Augmented Reality system using ARToolKit and user invisible markers
3. ARToolKit and the AR development process
ARToolKit is a software library written in the C programming language for creating AR
applications [Kat00] [Kat99]. It is distributed free for non-comercial or research
applications. It has been primarily developed by Dr. Hirokazu Kato of Osake
University, Japan, and is supported by the HIT Lab at the Univeristy of Washington and
the HIT Lab NZ at the University of Canterbury, New Zealand. Distributions for SGI
Irix, Linux, MacOS and Windows OS are available, and are distributed with complete
source code.
ARToolKit can be used for the easy development of a wide range of AR applications.
As discussed earlier, one of the key difficulties in developing AR applications is the
problem of tracking the user´s viewpoint. ARToolKit uses computer vision algorithms
to solve this problem by calculating the real camera position and orientation relative to
physical markers in real time. Also the other two elements of AR, registration and
calibration, are contained within ARToolKit. Registration is done by overlaying virtual
objects on the recognized physical markers; either created in VRML or OpenGL.
Calibration is provided by ARToolKit in the form of an easy camera calibration
algoritm.
For developing and running AR applications with ARToolKit the minumum hardware
requirements are a computer with a camera connected to it. Also can be used HMDs and
handheld displays. Both the use of video see-through and optical see-through HMDs are
supported. The computer should at least have a 500Mhz processor and a graphics
accelerator card. For the Windows platform USB cameras, FireWire cameras and
composite video cameras connected to a frame grabber can be used, as long as they
have DirectShow drivers.
3.1 Tracking and registration
Tracking is performed as a form of optical tracking; video frames are being grabbed and
searched for fiducial square markers. Such real world markers are pre-trained pattern
templates. The position and orientation of these markers is used for overlaying virtual
objects onto video frames. This gives the user a real time augmented view on the real
world. For being able to recognize the markers several steps are executed. First the live
video image is turned into a binary image based upon a preset lighting treshold value.
The resulting image is searched for squares regions. For every found square the pattern
inside it is matched against pattern templates. If a match occurs a marker has been
found. The next step is to calculate the position of the real camera relative to the
physical marker. These real world coordinates are used to set the position of the virtual
camera coordinates. This assures that the virtual objects drawn are precisely overlaid.
Figures 3.1 through 3.3 show the three steps involved in overlaying virtual objects.
Figure 3.1 Live video Figure 3.2 Binary image Figure 3.3 Virtual object
overlaid on live video
32

Development of an Augmented Reality system using ARToolKit and user invisible markers
For this process to function properly first a calibration procedure is necessary. There are
differences between optical see-through and video see-through calibration. Both
procedures will be described now.
3.2 Optical see-through calibration
For achieving appropriate stereoscopic view in an optical see-through configuration it is
necessary to measure the position of the eyes relative to the HMD. This is done through
a simple self-calibration method. Every time a user starts an AR application this should
be done. The calibration pattern that is used is shown in Figure 3.4. It has a 80mm
square form and must be printed out and glued on a piece of cardboard. This calibration
pattern is used for aligning a virtual white cross hair shown in the HMD. This is done
for five calibration positions, and for each there are two measurements. First the ten
measurements are taken for one eye, and after that for the other eye. This process is
quite time-consuming, but produces good results.
Figure 3.4 Calibration pattern Figure 3.5 A user performing the Figure 3.6 The view as seen
calibration process through the optical see-through
HMD
3.3 Video see-through calibration
In ARToolKit is specified a file with standard camera parameters which should be
sufficient for a wide range of cameras. But specific camera parameters can be obtained
by a relatively simple camera calibration technique. This is used for warping the video
image to remove camera distortions. For the calibration two patterns are used; one
containing a equally spaced 6x4 dot pattern, and one made up from a grid of lines. The
dot pattern is used for measuring the centre point of the camera and lens distortion. First
this pattern should be calibrated. The camera focal length and other camera parameters
are calibrated with the lines pattern.
When viewed through the lens the dot pattern appears to be unequally spaced because of
lens distortions. The spacings between the dots are used to calculate the camera
distortion. The user has to draw rectangles over all the dots in a frozen video image,
starting from the top-left to bottom-right. A red cross will be drawn in the centre of the
dot enclosed by the rectangle. The process of marking the calibration dots is shown in
Figure 3.7. After all the dots have been found, the positions of the dots are stored and
the image is unfrozen. This process needs to be repeated for 5-10 images taken from
various angles and positions. After that the centre x and y values, and the distortion
factor are calculated. These can be checked by having displayed the grabbed images
33

Development of an Augmented Reality system using ARToolKit and user invisible markers
with red lines drawn through the calibration dots. The red lines should pass through the
centre of each of the dots.
Figure 3.7 A user marking the Figure 3.8 Vertical alignment of the
calibration dots white line; the blue ones have
already been aligned
When the user is satisfied with the calibration, the next step is to calibrate the grid
pattern of seven horizontal lines and nine vertical lines. The algorithm for calibration
needs the earlier obtained centre x and y values and the distortion factor. The user has to
grab a video image with the calibration patten perpendicular to the camera, with all the
grid lines visible and the grid as large as possible. Then a white line will appear that
needs to be aligned with all the horizontal lines. After it has been aligned it will change
its color into blue. This is shown in Figure 3.8. The order to be used for aligning is from
top to bottom. After all horizontal lines have been aligned the process needs to be
repeated for the vertical lines, in the order from left to right.
Then, after this alignment has been done for one image, the calibration pattern should be
moved 100mm away from the camera, and the process repeated. This is done five times,
in total moving the calibration pattern over a distance of 500mm. After that the camera
parameters are calculated and stored in a file.
3.4 Developing AR applications
Now that have been described the way tracking, registration and calibration is supported
by ARToolKit, the global structure of an AR application is discussed now. Creating an
AR application does not only involve writing the application, but also image processing
routines need to be trained on the real world markers.
In writing an ARToolKit application the following steps must be taken:
1. Initialize the video path and read in the marker pattern files and camera parameters.
2. Grab a video input frame.
3. Detect the markers and recognized patterns in the video input frame.
4. Calculate the camera transformation relative to the detected patterns.
5. Draw the virtual objects on the detected patterns.
6. Close the video path down.
ARToolKit provides functions for carrying out these steps. In Figure 3.9 the functions
corresponding to the application steps are shown.
34

Development of an Augmented Reality system using ARToolKit and user invisible markers
Figure 3.9 Application steps and their corresponding ARToolKit functions
Steps 2 through 5 have been discussed earlier, and are repeated continously, while steps
1 and 6 are only performed on initialization and shutdown of the application. The
application may also need to respond to occuring events. A program consists of a main
routine that carries out the actions in step 1, by using the init function. Within the main
routine the argMainLoop function is responsable for calling the functions that
correspond to the application steps 2 through 5. Closing the application will make the
cleanup function stop the video processing and close down the video path.
A video image is captured and displayed on the screen, either a warped image corrected
for camera distortions or an unwarped image. arDetectMarker is used to search the
image for markers, which already have been read in by the init function. The function
delivers the number of found markers, and information concerning markers in the
marker_info structure. This contains the coordinate information and recognition
confidence values and object id numbers for each of the markers. For finding the
transformation between the markers and the camera arGetTransMat function is used.
Finally, virtual objects are drawn by the draw function, based upon the association of
virtual objects and marker patterns read in by init.
3.5 Training marker patterns
To allow ARToolKit to carry out its main functionality of overlaying virtual objects on
fiducial markers, it needs to be taught the different marker patterns to recognize. These
are specified in a text file, by stating the name of the marker, its corresponding pattern
and its width. Patterns to be recognized can be created by an ARToolKit subprogram
that makes template files. For doing so one first needs to print out an empty pattern file,
just containing a black square with a white square in the middle. Then a pattern needs to
be attached to the white square, either a black and white or color one. The pattern
recognition program is used to capture this pattern, preferably under similar lighting
conditions as when the AR application will be running. When the pattern is found this is
indicated by an appearing red and green square around it. The camera should be rotated
until the red corner of the highlighted square is in the top left hand corner. Figure 3.10
shows an identified marker pattern. Then the pattern is stored in a bitmap file and can be
used in AR applications.
Figure 3.10 Training marker patterns
35

Development of an Augmented Reality system using ARToolKit and user invisible markers
4. Hardware components for AR with user invisible markers
The two foregoing chapters provided the necessary background knowledge on the
subject matter. An overview of AR and applications of AR was given. Consequently the
AR development process was discussed. During this extensive discussion of related
aspects only the first element of the problem definition was used. However, in this
chapter the second, third and fourth element of the problem definition will be utilized to
perform a search for hardware components. To recapture, here these are stated again for
reference.
• Find suitable hardware components for the implementation of detecting of
user invisible markers, that is, a suitable invisible medium, a camera to detect
that medium and a connection between the camera and a computer. The
camera should work together with ARToolKit.
• Find a suitable stereoscopic video see-through HMD that incorporates both
capture and visualization. No requirements are stated on intrinsic properties
of the HMD.
• Find a suitable optical see-through HMD. Also for this HMD no requirements
are stated on intrinsic properties. Moreover the stereoscopic requirement is
relaxed.
To get more insight into the problem definition and the way the components are related,
below a prospective system overview is presented. Hereby needs to be noted that a
video see-through approach is presented; also could have been chosen for the optical
see-through method. Both a video see-through HMD with 2 colour cameras and an IR
camera view a scene concurrently. The cameras from the HMD are not able to detect the
invisible marker, whereas the IR camera can. ARToolKit uses the images from the IR
camera to determine the position and orientation of markers. Consequently this
information is utilized for overlaying virtual objects onto the images captured by the
two colour cameras from the HMD. The resulting images are finally shown on the
micro displays of the HMD. This constitutes all for creating a system that complies with
the requirements stated above. Figure 4.1 gives an overview of the system components
needed.
Figure 4.1 Overview of system components
36

Development of an Augmented Reality system using ARToolKit and user invisible markers
To simplify the search for hardware components the problem definition can be split up
into a number of elements. The elements that are encompassed within the problem
definition are invisible marker tracking, stereoscopic video see- through HMD, optical
see-through HMD and ARToolKit. The latter element, ARToolKit is a software
component and was discussed in detail before. The other three elements clearly need to
be supported by hardware components. The second element, a stereoscopic HMD, is to
be expected to be found with the most effort. The difficulty may be presented in the fact
that the HMD will need to have the ability to support both capture and visualization at
the same time. The optical see-through HMD is expected to be found with less effort.
The first element itself can be split up into a small number of sub elements. It was
shown that invisible marker tracking can be accomplished by a configuration composed
of three hardware components [Par04]. It needs to be said that this article was the first
attempt ever to implement invisible marker tracking. To our knowledge, all other
reported cases involving AR based upon ARToolKit make use of visible markers. First
of all, a suitable medium for creating the markers is required. It is fairly trivial that the
medium is to be invisible to the human eye to allow for invisible marker tracking.
Moreover, the medium will need to have certain properties to be able to be detected by
some device other than the human eye. Such a camera is the second hardware
component needed. Finally, to be able to use it together with ARToolKit, the third
hardware component needs to establish the connection between a computer and this
camera.
Now that is known which hardware components are needed, the remainder of this
chapter will, in turn, be devoted to each of those. That approach will give an overview
of the hardware components that possibly could be used. First the three hardware
components for invisible marker tracking will be discussed. The order in which this is
done is the following; invisible marker tracker tracking medium, cameras and finally
connection devices. After that will be treated aspects concerning the HMD system. This
order of representation is chosen because eventually the hardware components for
invisible marker tracking need to be incorporated into the HMD system.
4.1 Invisible marker tracking medium
As described before a suitable medium for invisible marker tracking may not be visible
to the human eye, but should be able to be detected by some other device than the
human eye. Literature study on this topic did not result in much. It appears that in the
AR community all research relies on the usage of fiducial markers, and that invisible
marker tracking has not been paid much attention to up until now.
Somewhat different but related work incorporates imperceptible patterns into images
using a non-intrusive way that enables simultaneous display and acquisition [Cot04].
This is done by modifying off-the-shelf digital projectors in such a way that the patterns
are only visible during a short interval. This non-intrusive approach only minimally
degrades the displayed images. This degration is not noticed because of the ability of
the human eye to recover color constanty. This results in that to the human eye the
embedded patterns are not visible. However, cameras synchronized with the digital
projectors are able to detect them. This basic idea behind this approach provides the
framework for invisible marker tracking.
37

Development of an Augmented Reality system using ARToolKit and user invisible markers
In the literature, at least to our knowledge, only one example exists that implements
invisible marker tracking. It makes use of ink that possesses the two required properties
to be a suitable medium for invisible marker tracking [Par04]. The experimental setup
consists of 2 Sony camcorders with IR functionality. One is equipped with an IR filter
which allows it to capture light in the infrared range. The second camcorder has a colour
correction filter and is used for capturing the real scene. A half-mirror is used for having
the optical centres of the two cameras coincide. Markers are drawn in fluorescent ink
that is invisible in the visible range but can be detected in the invisible range. Both
camcorders were used to record a scene. An overview of the system is presented in
Figure 4.2. This system delivers two videos of the same scene; one that only contains
the visible light in the scene and one that also captures light in the infrared range. Figure
4.3 shows two of such images taken from captured video. Images from the second video
were used by ARToolKit for determining the position and orientation of the invisible
markers. Consequently virtual objects were overlaid on the first video. In this way
Augmented Reality is realized using invisible markers.
Figure 4.2 The invisible marker tracking system
Figure 4.3 Visible light image and infrared image
38

Development of an Augmented Reality system using ARToolKit and user invisible markers
4.1.1 Ink Pen
Now will be described in more detail the properties of the before mentioned ink. The
ink is invisible to ultraviolet light and the human eye. Ultraviolet light is below 400nm.
The human eye can see light between 400nm and 750nm. The ink emits 840nm
frequency light and has a 793nm absorption frequency, which lies in the IR range. There
exist three classifications for IR light, near-IR, mid-IR and far-IR. This classification is
based on ascending light frequencies. The ink fits into the near-IR spectrum, 750nm to
3000nm. The light spectrum is graphically displayed in Figure 4.4.
Figure 4.4 The light spectrum
The ink itself is delivered in the form of an ink pen, the so called IR1PenSM. The tip of
the pen is green fluorescent to the human eye. The manufacturer of the ink notifies that
the ink is influenced by ultraviolet light. Estimated is that the ink will last for about a
week, but keeping it out of ultraviolet light will increase this period. The manufacturer
mentions a price of $18,43. This is without handling costs, administration costs and
shipping costs. The pen is displayed in Figure 4.5.
Figure 4.5 IR1PenSM
Anything written on white paper can not be seen by the human eye. But, when viewed
with a modified camera with IR functionality the ink becomes visible. An external light
source is not required. This is being shown in Figure 4.6 and Figure 4.7.
Figure 4.6 Image as seen by the human eye Figure 4.7 Image captured by IR camera
4.1.2 IR Powder
Not only the manufacturer delivers ink that is visible in the infrared range, but it also
has at its disposal powder with similar properties, the so called IRDC2. This IR
fluorescent powder is visible at both 880nm and 1050nm. Therefore it needs to be
stimulated by an external light source, that produces either blue or red light, of
39

Development of an Augmented Reality system using ARToolKit and user invisible markers
frequencies 450nm respectively 630nm. The powder is said to have very high long term
stability, although there exists no accurate definition of period. Just as the ink is visible
green fluorescent, the same holds for IR fluorescent powder. The price for this powder
is reasonably higher than that for the ink. A quantity of 10 grammas has a price of
$150,40. Also this is without handling costs, administration costs and shipping costs.
The higher price, together with the need for an external light source, which also needs to
be obtained, makes it less cost attractive to apply. Moreover there is no specific
information available on application of the powder, as is provided with the ink.
Although the powder may be used as an invisible marker, the external light source will
be visible. This may interfere with application requirements.
4.2 Optical filters
Now that is described the invisible marker tracking medium, the visualization aspect is
explained next. For the visualization it is necessary to have a camera that can detect the
ink described above. Normal filters let pass through visible and IR light. For the ink to
be detected a camera should be infrared sensitive, that is, the standard IR blocking filter
should have been removed. This can be done manually but also the company at
http://maxmax.com offers a service to adapt cameras. Prices for modification range
from $150 to $350. This is without shipping, insurance and administration costs. For
several Sony cameras this is known to work. For other cameras it is not sure, and it can
result in a damaged camera. Also, the size of the filter may not fit with other cameras.
For the mentioned Sony cameras the filter are known to fit, but the weight is the
limiting factor.
4.2.1 IR filters
After the IR blocking filter has been removed, a special optical filter should replace the
removed filter. This is necessary in both cases; for viewing visible light and to view
infrared light. It appears that the manufacturer of the ink delivers two kinds of special
optical filters. Those either filter out visible light and block IR light, or filter out IR light
and block visible light.
The ones that filter out the IR light come in different length waves. The available optical
filters have different cut off values; these are 630nm, 665nm, 715nm, 780nm, 830nm,
850nm and 1000nm. Not only are several cut off values available, but also different
filter sizes; 25mm, 30mm, 37mm, 58mm, 72mm and 82mm. Moreover, there are
several possibilities to choose for the thickness of the filters; 1mm, 2mm and 3mm. The
filters are constructed from precision made and ground polished German glass. They
appear completely black to the human eye. Such a filter is shown in Figure 4.8. In
Figure 4.9 the transmittance characteristics of several filters are displayed.
40

Development of an Augmented Reality system using ARToolKit and user invisible markers
Figure 4.8 An IR filter Figure 4.9 Transmittance graphs of several IR filters
For detecting the invisible ink such a filter is needed. The one with cut off value of
715nm is mentioned by the manufacturer of the special ink.. For detecting the IR
powder one should use either use a cut off value of 830nm or 850nm, depending on the
frequency of the used external light source; 450nm or 630nm.
4.2.2 Colour correction filter
The filters that block the IR light come in two kinds, the so called X-Nite CC1 and X-
Nite CC2. They are called colour correction filters. They give the opposite effect to the
filters mentioned before. The CC1 filter is the standard filter to be used. For some more
recent cameras the CC2 filter is needed. The CC2 lets pass through more visible light
which causes it to give a higher sensitivity compared to the CC1. The CC2 should be
used when the removed IR blocking filter is aqua-coloured. The colour correction filters
will not be used, but for reasons of completeness they are mentioned anyway.
4.3 IR Cameras
A camera with the IR blocking filter removed is able to detect infrared light when the
removed filter is replaced with an IR pass filter. LDP NET has several Sony camcorders
available that have this property, but for the application as stated in the problem
formulation those are not suitable, because of weight limitations. But also similar smallsize
and light-weight cameras are available. These are known as so called lipstick
cameras. These will be described next.
The cameras have a permanently installed IR filter and are of type Black White. The
camera is put into a small lipstick sized tube of 2.5 inches long, with a diameter of
0.8125 inches. The sensitivity of the camera is 0.1 Lux and it delivers >380 TV lines. It
turns out that there is no defined relation between TV lines and resolution. The
manufacturer of the camera can not provide that information either. Comparable
cameras with the same number of TV lines have resolutions like 512x492 or 512x582.
Therefore it is expected this camera will have a similar resolution, but that is not certain.
41

Development of an Augmented Reality system using ARToolKit and user invisible markers
The camera has a frame rate of 29.97 fps ≈ 30 fps. The image sensor is 1/3 CCD with
290,000 CCIR pixels. The output of the camera is standard composite video, 75ohm,
1Vp-p. This is the standard for almost all consumer video equipment. It is also known
under the name RS-170 NTSC. This signal combines three pieces of information in one
signal (http://www.datapro.net/videodoc.html#Page04). These are colour, brightness
and synchronization data. In Figure 5.8 the camera is displayed to show its size
compared to a coin. The weight of the camera is 50 grams. Together with the camera are
delivered a mounting bracket, mounting screws and integral power and video
connectors. In Figure 5.9 are all the components together with the camera are shown.
There are several options to choose for the IR filter; 715, 780, 830, 850 or 1000nm. All
cameras with different options have the same price of $99. As with before mentioned
prices, also this does not include handling costs, administration costs and shipping costs.
Figure 4.10 Bullet camera Figure 4.11 The complete component set
4.3.1 Power supply
The camera needs a 9-12 VDC power supply. Transformer power supplies can result in
voltage surges that will damage the camera. Therefore it is advised to make use of a
regulated power supply. This takes as input an input voltage of 100 to 240 Volts AC
(alternating current), 50 to 60 Hz. In Spain the standard voltage used is of type 230V
AC, 50Hz (http://www.school-for-champions.com/ science/ac_world_volt_freq.htm).
This regulated power supply will output a stable, regulated 12VDC. Figure 4.12 shows
such a power supply. It has a price of $15, again extra costs are not included.
Figure 4.12 Power supply
42

Development of an Augmented Reality system using ARToolKit and user invisible markers
4.3.2 Connection to computer
As stated in the problem formulation the final solution needs to have the ability to be
used with ARToolKit. Therefore it is clearly needed to have the camera connected to a
computer running ARToolKit. It is said that on the Windows platform any camera can
be used, required that it has DirectShow® drivers. There are many examples of cameras
that are suitable for ARToolKit, which also have been seen working. These include
USB cameras and cameras connected through an IEEE1394 FireWire port. The
ARToolKit manual also mentions a third possibility to establish ARToolKit working
together with a camera.
4.3.2.1 USB
To connect the IR bullet camera through USB it is necessary to make a conversion from
the video out signal from the camera to USB. Therefore a device is needed that accepts
the video out signal from the camera and outputs a USB signal. This USB signal can
comply with either the USB1.1 or USB2.0 standard. This will depend on the features of
the USB port of the computer. Here will be presented converting devices for both
USB1.1 and USB2.0. For both types 2 different devices are presented.
USB 1.1 VideoLink
The USB 1.1 VideoLink connects a video camera to any computer with a USB 1.1 port.
For this model there are no specifications about the frame rate available. Only for the
model described further on, the USB 2.0 VideoLink, it is specified (640 x 480 pixels at
up to 30 frames/sec). It is expected to deliver the same. The device is DirectShow®
compliant. It accepts as input signals either S-Video or composite video. It will output a
USB 1.1 signal. The price to be paid is $229, at http://www.zarbeco.com. The device is
shown in Figure 4.13. In Figure 4.14 the computer requirements are shown. Although
there will be no problems to expect with today’s computers they are shown to give a
complete overview. These are taken from the USB 2.0 VideoLink, because precise
requirements are not available. They are expected to be similar or perhaps even the
same, but at least not higher.
Requirements
• Windows 2000/XP
• Free USB1.1 port
• DirectX 8.1 or greater
• Pentium III 1.0 GHz or faster
• 128 MB of Free Memory
• Video card with 16MB of Video
RAM, Minimum 800x600 resolution
at 16-bit
Figure 4.13 USB1.1 VideoLink Figure 4.14 Requirements
43

Development of an Augmented Reality system using ARToolKit and user invisible markers
GrabBeeII - USB video grabber
The GrabBee II video grabber converts a composite video in signal to a USB1.1 signal.
It delivers a frame rate of 30fps at screen resolution of 320x240. The price is €99 at
http://www.breukhoven.nl. This is without shipping costs. The device and its
requirements are displayed in Figure 4.15 and Figure 4.16.
Requirements
• USB equipped computer
• Pentium 233MMX
• 32MB of RAM
• Windows 98/2000/ME/XP
Figure 4.15 GrabBee II Figure 4.16 Requirements
USB 2.0 VideoLink
This device complies with the description of the USB 1.1 VideoLink that was discussed
before. It is delivered by the same company. The only difference is that this device
outputs a USB 2.0 signal. The resolution that delivers this device is known to be 640 x
480 pixels at up to 30 frames/sec. The computer requirements are shown in Figure 5.8.
The price to be paid is $279, at http://www.zarbeco.com.
USB 2.0 Video Grabber
This device converts a video composite in signal to a USB 2.0 signal. It delivers a frame
rate of up to 30fps at a resolution of 640x480. It is only for use with Windows 2000/XP
and is compliant with DirectShow. The price at http://www.lindy.com is €59, when the
buyer is eligible for VAT reduction. Therefore a VAT registration number is needed.
The device is shown in Figure 4.17.
4.3.2.2 IEEE1394 Firewire
Figure 4.17 USB 2.0 Video Grabber
To connect the IR bullet camera through IEEE1394 Firewire it is necessary to make a
conversion from the video out signal from the camera to Firewire signal. Therefore a
device is needed that accepts the video out signal from the camera and outputs a
FireWire signal. For now, only one such device will be presented.
44

Development of an Augmented Reality system using ARToolKit and user invisible markers
PixeLINK PL-A544 FireWire Video Converter
This FireWire converter accepts as input signal either RS-170 PAL or RS-170 NTSC.
The device outputs a 6pin FireWire signal. It has drivers that are DirectShow®
compliant. It supports a maximum frame rate of 30fps. Also three different video
standards are supported; 640x480, 320x240 and 160x120. There are four packages that
can be bought. It suffices to have the Starter’s kit. It includes a FireWire cable, a
desktop adapter card and the PL-A544 itself. This package carries a price of €535 at
http://bfoptilas.com. The product itself is displayed in Figure 4.18, together with its
computer requirements in Figure 4.19. As was the case with before mentioned system
requirements these also will not give rise to problems.
Figure 4.18 PL-A544 Figure 4.19 Requirements
4.3.2.3 Frame grabber
Requirements
• Pentium III microprocessor, 450Mhz
• 128 Mbytes RAM
• DirectX 8.1 or greater
• Pentium III 1.0 GHz or faster
• 128 MB of Free Memory Video card
with at least 4MB Video RAM,
minimum 800x600 resolution at 24-bit
or 32-bit True Color
• A FireWire cable
• An OHCI FireWire connection
In theory it should be possible to connect a camera through a frame grabber to a
computer to work with ARToolKit. Therefore it is needed that the frame grabber is
DirectShow® compatible. One needs to be careful that no frames are lost during
grabbing. Now will be described one such frame grabber, Matrox Meteor II,
http://www.matrox.com/imaging/products/meteor2/home.cfm.
The frame grabber needs a PCI bus slot. It captures from different video sources, among
which RS-170. Acquired images can be transferred in real time to either system
memory or display memory. It is unnecessary to mention system requirements, because
the frame grabber has been used in practice before, which shows that these requirements
are met. The Matrox Meteor II frame grabber is displayed in Figure 4.20.
Figure 4.20 Matrox Meteor II frame grabber
45

Development of an Augmented Reality system using ARToolKit and user invisible markers
Hauppage WinTV-PVR
There has been an example of an AR application for museum artefact visualization that
used ARToolKit together with a Hauppage WinTV-PVR video capture card [8]. This
shows that there exists a possibility to have a frame grabber and ARToolKit work
together. From the source it is not completely clear which type of frame grabber was
used. There are several available. Here will be described the oldest model, WinTV-
PVR350 because it will have lower price, but offers the same basic functionality.
The frame grabber is delivered as a PCI card. Captured video can be stored on hard disk
or sent to a monitor in real time. Input to the frame grabber is either NTSC or PAL.
Using NTSC display rate is 30fps, Full D1: 720x480, MPEG1: 352x240. In Figure 4.21
and Figure 4.22 are displayed the frame grabber itself and its system requirements.
Again these are not expected to give problems. The frame grabber is available at
http://www.mini-itx.com. The total price is €215,67, including tax and shipping costs.
Requirements
• Pentium III 730MHz or faster
• Windows XP/2000
• Free PCI slot
Figure 4.21 WinTV-PVR350 Figure 4.22 Requirements
4.4 Firewire board camera
The hardware components described before are all meant to provide a solution in
cooperation with the described IR bullet cameras, which have a permanent IR filter
installed. There also exists a second possibility that implicitly make use of the described
IR filters. This manner will be more extensive and there exists more uncertainty about
successful results. What needs to be done is that a camera manually is taken apart, the
IR blocking filter is removed and that the IR filter is installed. But several problems
may arise; the camera may be damaged, the IR filter may not fit or the images may not
be of good quality.
The process of removal the IR blocking filter can be tried with any camera, but again
the weight of the camera should not be too high. Therefore first has been tried to find
suitable cameras. These cameras are of type board level camera. Two of such cameras
are described now.
The model that is described first is the Dragonfly, delivered by Point Grey Research,
http://www.ptgrey.com. The camera is delivered with a 1/3” Sony CCD image sensor; it
has dimensions of 2.5”x2.0 inches, or 63.5x50.8mm. Different options are available for
the resolution, either 640x480 or 1024x768. Also Colour or Black and White can be
chosen for. The 640x480 version can deliver a frame rate up to 30fps. The camera has
an IEEE 1394 interface. Together with the camera are delivered a 6pin IEEE 1394 cable
and an IEEE 1394 PCI interface card. This constitutes all for having the camera
46

Development of an Augmented Reality system using ARToolKit and user invisible markers
connected to the computer. In Figure 4.23 and 4.24 are shown the camera itself and its
system requirements.
Requirements
• Intel Pentium II or better
• Windows 2000 or XP
Figure 4.23 Dragonfly Figure 4.24 Requirements Figure 4.25 FireFly2
The second camera, the FireFly2, is delivered by the same company. It has properties
similar to the camera described before. It is also delivered with a 6pin IEEE 1394 cable
and an IEEE 1394 PCI interface card. Differences are the size of the image sensor, the
resolution, the type and the size of the camera. This camera has a 1/4” Sony CCD image
sensor with dimensions of 40x40mm. The only resolution and type available is a
640x480 colour image sensor. The maximum frame rate is 30fps. The weight of the
camera including a micro lens is 12grams. Figure 4.25 displays the camera. The system
requirements are the same as in Figure 4.24.
Both the Dragonfly and the FireFly2 cameras are delivered with 3 micro lenses. The
focal lengths of these are 4, 6 and 8mm. The lens holders for both cameras are designed
to accommodate lenses that are designed according to the M12 standard; that is, the
diameter is 12mm and the thread pitch is 0.5mm. The DragonFly camera is delivered
with a C/CS mount lens holder. The FireFly2 does not include such a lens holder, but it
can be obtained. Now the difference between micro lenses, C mount and CS mount
lenses is explained.
4.4.1 Lenses
Micro lenses are designed for use in micro CCD board. They can cover a variety of
CCD formats, including the 1/3” and 1/4” size CCD.
C and CS-mount lenses are both threaded lens mounts found on most industrial CCD
cameras. The difference between C and CS-mount equipment is displayed in Figure
4.26. The flange back distance is the distance between the sensor image plane and the
flange of the camera (the part of the case that buts up against the camera). For CSmount
lenses this is 17.52mm and for C-mount cameras 12.52mm. The focus can be
corrected by placing a 5mm spacer. This only works with a CS-mount camera. Then
both types of lenses can be used, CS-mount and C-mount lenses.
47

Development of an Augmented Reality system using ARToolKit and user invisible markers
Figure 4.26 The difference between C-mount
and CS-mount lenses
So, with a CS-mount camera there are three possibilities to choose for the lens; a micro
lens, a CS-mount lens or a C-mount lens. To have a camera to be IR sensitive first the
internal IR blocking filter needs to be removed. Contact with the company that delivers
the cameras has shown that the company is willing and able to remove the internal IR
blocking filter. After this has been done the camera needs a suitable IR cut off filter to
make it able to detect light in the infrared range.
The cameras are delivered with 3 types of lenses; 4mm, 6mm and 8mm. Properties of
the lenses are displayed in Figure 4.27. Any IR filter that has similar properties should
be able to fit the cameras.
Figure 4.27 Lens properties of DragonFly and FireFly2 cameras
It would be very handy if the IR filters described before would fit the Point Grey
Research cameras. Then no other search for suitable filters would have been needed.
But this is not the case; the diameter sizes of the cameras and the IR filters do not meet.
That means that more information on IR filters needs to be gathered. It seems that there
is no standard information that all manufacturers provide.
There are two possibilities for the IR filter. One round filter that fits the lens of the
camera could be used. This type of filter will be hardest to find because of the diversity
of available information. The other possibility is an IR filter that has a size bigger than
the diameter of the lens. This filter then could be put in front of the camera’s lens. This
will be a square sized filter, because usually those are made from sheet material.
48

Development of an Augmented Reality system using ARToolKit and user invisible markers
4.4.2 Square IR filter
First the second suggestion will be discussed. There has been an example in which
invisible bar codes are recognized by a system that is composed of a Canon RE552
CCD camera in combination with a square IR filter [Mak97]. The bar codes are near
infrared reflective, like the invisible ink discussed earlier. Inside the CCD the infrared
filter was replaced by transparent glass to make it IR sensitive. This is the same as
removing the internal IR blocking filter of the discussed Point Grey cameras. In front of
the camera a visible light cut filter was mounted that is square sized. This will only let
pass the IR light. The two pictures in Figure 4.28 and Figure 4.29 show the results of
this modification to the camera. The left figure shows the image with only visible light,
the right one shows the near IR light.
Figure 4.28 Visible light image Figure 4.29 Near IR light image
Now will be discussed the properties of the IR filter that was found. The filter is a so
called RT-830 IR filter manufactured by HOYA Company. The filter has the property
to let pass the IR light above a frequency of about 700nm. The transmittance graph is
shown in Figure 4.30. For the values of the light frequencies we are interested in the
values are specified in the table in Figure 4.31.
Figure 4.30 Transmittance graph of the RT-830 IR filter
Figure 4.31 Corresponding transmittance values
The table shows the values T and ι. The first parameter represents the transmittance and
the second stands for the internal transmittance. The discussed ink emits light of
frequency of 792nm. The maximum transmittance is 92.9% at 800nnm. The internal
transmittance of the filter around 792nm is approximately 90%. This could be used for
49

Development of an Augmented Reality system using ARToolKit and user invisible markers
detecting the markers written in ink. The thickness of the filter is 2.5mm. The square
size of the filter is 25mm x 25mm, and it appears black. This is enough to cover the lens
of the camera. The IR filter is available at http://edmundoptics.com. The filter has a
price of $35.50. This is without shipping costs.
Now also a way to attach the filter on a fixed position in front of the lens needs to be
provided. The requirement to this is that the filter can not move while the camera is
being moved. It should always hold its position relative to the lens of the camera. There
are filter holders, but those normally are only suitable for a certain type of camera. Such
a filter holder is displayed in Figure 4.32. Also usually such a filter holder will increase
the weight of the solution with a reasonable amount. In our case this is the limiting
factor. Besides, at least to our knowledge there are no such filter holders for the
DragonFly or Firefly2 cameras.
Then the problem arises how to attach the filter in a fixed position, so that it covers the
complete lens, and it can not move from its position. This position has to be perfectly
parallel to the CCD plane. It also needs to maintain this parallel position at any time.
Figure 4.32 Filter holder
It can be tried to attach the filter in some other way to the camera. Then this solution
needs to be invented by making use of some material like tape and glue. This will be a
cheaper solution than having a custom made round IR filter. But also it needs to be
assured that this solution will be robust enough, and most important that no light enters
the lens of the camera without passing through the IR filter. This will be done by
placing the filter as close to the camera as possible.
4.4.3 Round IR filter
The manufacturer of the DragonFly and FireFly2 cameras mentions not only what kind
of micro lenses they use in their cameras, but also information about the manufacturer
of these lenses is given. That is Boowon Optical, http://boowon.co.kr, a company that
produces lenses for CCTV cameras and CCD cameras.
It was found out that the just mentioned company has the ability to custom make the
required IR filter. There has been contact with the company. What they need is the size
of the filter, together with the thickness of it. And to complete the request they need to
know the specific requirements for the IR filter. That is, a specification of the
wavelength intervals, together with a percentage of transmittance of light for those
intervals. Engineers from the company came up with such a specification after the first
50

Development of an Augmented Reality system using ARToolKit and user invisible markers
proposal was found out to lack the right properties. This transmittance graph is shown in
Figure 4.33.
Figure 4.33 Proposed transmittance graph
This would be a better solution than the one explained before. First of all, the total
weight will be significantly lower. Also, there should not be any thoughts about how to
mount the filter in a fixed position. Simply the filter is attached to the camera, like a
micro lens, or C/CS mount lens would be. It will be at the right position, without having
the problem of finding a way to put it in parallel with the CCD plane and avoiding light
to pass.
4.5 AR HMD system
Now that have been described the components for invisible marker tracking the results
of the search for suitable HMDs are laid out now. The way the results are presented will
only focus on the main aspects of the found hardware. Only an overview is given to
keep things clear and to save the reader from skipping through secondary information.
First will be described three video see-through HMD systems, followed by another type
of HMD, an optical see-through HMD. Hereby needs to be noted that one should be
careful with the stated data on FOV of the HMD; these can be measured horizontally,
vertically or even diagonally. When the way the FOV is measured is known, this is
mentioned.
4.5.1 Video see-through
This type of HMDs uses two cameras to capture a scene. The images from those
cameras are displayed on the two microdisplays of the HMD. To create Augmented
Reality these images are overlaid with virtual objects.
ARVision3D manufactures a full stereo HMD which is available in two kinds; either as
an ARVision3D HMD or as an ARVision3D Goggles model. Both make use of two
colour cameras to provide the stereoscopic vision ability. No synchronization of the
cameras is needed. The view is presented to the user through two SVGA micro displays
with 480,000 pixels, equal to 800x600 resolution. Refresh rate of the HMD is 120Hz
flicker free. The FOV that the cameras deliver is approximately 40° diagonal (4:3, 32°
horizontal, 24° vertical). The weight of the device is 120 grams. Together with HMD or
51

Development of an Augmented Reality system using ARToolKit and user invisible markers
Goggles is delivered a power unit that interfaces with a computer. Price is €7300, at
http://inition.co.uk/indexf.htm. In Figure 4.34 and Figure 4.35 are shown both models;
HMD and Goggles. Figure 4.36 displays the power unit.
Figure 4.34 HMD model Figure 4.35 Goggles model Figure 4.36 Power Unit
The first described HMD and Goggles are a complete all-in-one solution. Now will be
described a device that, together with an HMD also provides an all-in-one solution.
The VideoVision has two colour NTSC/PAL cameras which deliver a video-out signal.
These video signals connected to an HMD give it stereoscopic possibilities.
Requirement is that the used HMD needs to have the ability to handle stereo input. The
cameras have a resolution of 640x480 pixels. Each camera has a FOV of 78 degrees.
Refresh rate is measured at 60Hz. The dimensions of the device are 185mm x 48mm x
47mm. Total weight of the device is 230 grams. The VideoVision needs to be used as
an add-on to an HMD. The device is shown in Figure 4.37. The manufacturer advices
to use it together with the nVisor SX HMD which is available at
http://www.nvisinc.com. This HMD, depicted in Figure 4.38 carries a price of $23,900.
It needs to be accentuated that of course other HMDs can also be used. The
VideoVision costs $8,900, including a rendering computer with a NVIDIA
GeForceFX6800 graphics card to connect the HMD.
Figure 4.37 VideoVision add-on Figure 4.38 nVisor SX HMD
The foregoing described a somewhat more elaborative solution. The HMD solution
described now is also a bit more complicated. A Dutch company, http://cybermind.nl,
sell HMDs and other type of VR equipment. One of the HMDs they have is the
Hi-Res800PC, an HMD normally used for VR applications. Contact with the company
has shown that they have the possibility to adapt the HMD in such a way that it can be
used for AR. For obtaining this, two cameras need to be integrated into the HMD for
capturing a scene. The images from the camera are processed by two video cards, so
that a frame sequential stream of images is created. These images are fed back into the
two micro displays which will create a stereoscopic view of a scene.
The bare HiRes-800PC HMD, displayed in Figure 4.39, carries a price of €3275.
52

Development of an Augmented Reality system using ARToolKit and user invisible markers
Figure 4.39 HiRes-800PC HMD
The company has a number of solutions they can provide, with increasing degree of
completeness. First of all they can just include two cameras into the HMD for having it
stereoscopic enabled. Then the camera for detecting the invisible medium has to be
added by ourselves. But also they have the possibility to incorporate this camera. If an
even more complete solution is desired next to these two solutions, that possibility
exists. They can deliver a stereoscopic HMD with the camera for detecting the invisible
medium included, together with a computer equipped with two video cards. This would
be a complete all-in-one system that can be used for AR applications with invisible
marker tracking.
4.5.2 Optical see-through
The before described HMDs are video see-through HMDs. Now will be described
another type known as optical see-through. This type of HMDs does not use cameras for
acquiring the images, but are transparent to allow the user to see the real world through
it. Then virtual objects are rendered over these images. This type of HMD could also be
used for creating augmented views.
The LitEye-500 is an optical see-through HMD that has an option to open and close the
display of the HMD. Both are shown in Figures 4.40 and 4.41. The resolution of the
HMD enables 800x600 with a FOV of 28 degrees. Video inputs of the HMD are PAL,
NTSC and RS-170. The LitEye-500 is available at http://liteye.com, for a price of
€2950, excl. VAT. For creating stereoscopic AR a pair needs to be obtained, since the
LitEye-500 only covers one eye.
Figure 4.40 Open HMD Figure 4.41 Closed HMD
53

Development of an Augmented Reality system using ARToolKit and user invisible markers
Figure 4.42 nVision DataVisor HiRes see-through HMD
nVision offers an HMD with an additional see-through option. This DataVisor HiRes
HMD has full colour capability. The FOV delivered by the HMD is 60 degrees. RGB
inputs are accepted ranging from 640x480 to 1280x1024. The HMD and the seethrough
option are not delivered as a complete solution, but need to be purchased apart
from each other. The HMD carries a price of €17725 at http://www.inition.co.uk. For
the optical see-through option an additional amount of €2720 is to be paid. The HMD
with see-through option is shown in Figure 4.42.
4.6 Summary of results
A suitable medium is always needed. Either the Ink Pen or the IR powder can be used.
Type
Light source
Filter needed
Medium
Ink Pen IR Powder
X-Nite 715nm
Blue, 450nm Red, 630nm
X-Nite 830 X-Nite 850
Medium Duration Price Quantity Where?
Ink Pen ≈week $18,43 1 http://maxmax.com
IR Powder Very long high
term stability?
$150,40 10 grams http://maxmax.com
The first solution proposed uses the described IR bullet cameras. Depending on the
choice of medium a different IR filter installed in the IR bullet camera is needed.
54

Development of an Augmented Reality system using ARToolKit and user invisible markers
Type
Connection
1.1 2.0
USB1.1
Videolink GrabBeeII
IR Bullet
Camera
715,780,830,
850,1000nm IR
filter
Regulated Power
Supply
Camera
IR bullet camera
+ power supply
+ filter
(715/830/850)
USB IEEE 1394 FireWire Frame grabber
USB2.0
Videolink
USB2.0
Video
Grabber
PixeLINK PL-A544
Matrox
Meteor II
(already
available)
Price Extra costs Where?
$99 handling costs
administration costs
shipping costs
$15 handling costs
administration costs
shipping costs
Hauppage
WinTV-PVR
http://maxmax.com
http://maxmax.com
Connect
through USB1.1
Price Extra costs Where?
USB 1.1 $229 $25 Postal service www.zarbeco.com
VideoLink
government duties
GrabBeeII
Connect
through USB2.0
€99 Shipping costs www.breukhoven.nl
USB 2.0 $279 $25 Postal service www.zarbeco.com
VideoLink
government duties
USB 2.0 Video $59 none, if VAT
www.lindy.com
Grabber
registration available
Connect
through
FireWire
PixeLINK PL-
A544
Price Extra costs Where?
€535 shipping costs http://bfoptilas.nl
55

Development of an Augmented Reality system using ARToolKit and user invisible markers
Connect
through frame
grabber
Price Extra costs Where?
Matrox Meteor II - none, frame grabber MedICLab
Hauppage
WinTV-PVR350
already available
€215,67 none, tax and shipping
costs included
http://www.mini-itx.com
The second solution proposed uses the described Point Grey Research board cameras,
model DragonFly or FireFly2. The internal IR blocking filter will be removed by the
manufacturer to make it IR sensitive. An IR filter has to only let pass IR light.
Type
Filter
Square
RT-830
Hoya
Camera
Dragonfly FireFly2
Round
Custom
made
Square
RT-830
Hoya
Round
Custom
made
Board camera Price Extra costs Where?
DragonFly $795 removing IR filter
duties
shipping costs
http://ptgrey.com
FireFly2 $299 + $30 CS-mount removing IR filter http://ptgrey.com
adapter
duties
shipping costs
IR filter Price Extra costs Where?
RT-830 Hoya $35,50 shipping costs
attachment material
http://edmundoptics.com
Custom made Depends on the filter
characteristics
shipping costs http://boowon.co.kr
For the HMD there are few options available, presented below.
56

Development of an Augmented Reality system using ARToolKit and user invisible markers
Type
Fully integrated
video see-
through HMD
ARVision3D
HMD/Goggles
ARVision3D VideoVision
add-on
Extra
hardware HMD
HMD
Video see-through Optical see-through
Custom
made
HiRes800PC
Depends on
chosen
configuration
LitEye-500
Price Extra costs Where?
nVision
DataVisor
HiRes
€7300 shipping costs http://inition.co.uk
HMD add-on Price Extra costs Where?
stereoscopic $8,900, rendering shipping costs
http://www.worldviz.com
add-on
computer included HMD
Custom made
video seethrough
HMD
stereoscopic
HiRes-800PC
stereoscopic
HiRes-800PC +
camera for
detecting
invisible medium
stereoscopic
HiRes-800PC +
camera for
detecting
invisible medium
+ computer
Price Extra costs Where?
Bare HMD price : €3275 2 cameras
custom adaptation
Bare HMD price : €3275 2 cameras
IR camera
custom adaptation
Bare HMD price : €3275
2 cameras
IR camera
custom adaptation
computer
2 video cards
http://www.cybermind.nl
http://www.cybermind.nl
http://www.cybermind.nl
Optical seethrough
HMD
Price Extra costs Where?
LitEye-500 €2950 + VAT shipping costs http://www.liteye.com
nVision
DataVisor
HiRes
€17725 for HMD
+ €2720 for optical see-
through option
shipping costs
57
http://inition.co.uk

Development of an Augmented Reality system using ARToolKit and user invisible markers
5. Proposal for system components
Now that have been described the components that could be used, it is necessary to
narrow down the view onto a selection of components that eventually could be selected.
This will be done by evaluating the found hardware components from the previous
chapter. This results in a proposal for the five mentioned components from the problem
description. This proposal constitutes the answer to the last part of the problem
definition, which here is stated again for reference.
• Propose the hardware components to create an AR system that uses invisible
marker tracking in conjunction with ARToolKit.
5.1 Invisible medium
For the invisible medium there are just two options to choose from. Either ink or
powder could be used. The requirement to use an external light source with the powder
makes it an unattractive medium. It would introduce a significant distraction for the user
of the AR system. The ink does not cause any such distraction which makes it the better
medium out of the two. It has to be taken for granted the short durability of about a
week. The powder lasts longer but nevertheless also has finite durability.
5.2 IR Camera
The camera for recognizing the ink can be selected from two possibilities. The IR bullet
camera or the board camera could be used. The first type of camera is a solution which
does not require any additional adaptation to the camera. The IR filter is installed in the
camera. The ink and such an IR filter are manufactured by the same company, which
assures that it actually can be used for recognizing the ink. The second type of camera
requires two adaptations; first it needs to be made IR sensitive by removing the internal
blocking filter, after that an IR filter needs to be attached. The first adaptation can be
done by the manufacturer. The second will have to be done by ourselves.
These adaptations needed for the board cameras are more elaborative and a possibly
error-prone. Especially the attachment of the IR filter may deliver problems; it needs to
be attached in a fixed parallel position with respect to the image plane and no light is
allowed to enter without passing through the filter. Moreover the characteristics of the
filter may give rise to problems concerning recognition of the ink.
The IR filter can be a square or a custom made round one. The custom made filter does
not have the difficulty of attaching it in the right position, and is, with respect to this
aspect, favoured over the square filter. The specification of the filter is to be delivered
by ourselves. This is done in conjunction with engineers from the chosen company,
which reduces the risk that the filter eventually will not fit its task.
Because of the custom manufacturing process the costs of the filter are expected to be
high. If it turns out that they constitute an amount too high of the total costs there still
can be chosen for a square filter. In that case, great effort is needed to overcome the
filter attachment problem.
58

Development of an Augmented Reality system using ARToolKit and user invisible markers
Considering the foregoing it is recommended to apply the IR bullet camera for
recognizing the ink. Second choice could be a board camera in conjunction with a
custom made IR filter. The third possibility that exists is to have a board camera with a
square IR filter.
5.3 Connection to computer
If is chosen for the board camera, there is no necessity for deciding on a connection to a
computer. Since the camera has an IEEE-1394 FireWire interface the type of connection
is prescribed. However, the IR bullet camera can have three different types; USB,
FireWire and frame grabber.
USB and FireWire are favoured over a connection through a frame grabber. The first
two are known to function with ARToolKit and are a “plug-and-play” solution, which
makes it possible to connect them to any computer that has an USB or a FireWire port.
This increases the possibilities to use the system on remote locations. When the third
connection type is used these are narrowed because of the need for a frame grabber. For
the purpose of this evaluation only USB2.0 and FireWire are taken into account, and
they are considered equal. The option of a USB1.1 connection is discarded. Because of
the equality, one can decide which to use by a price comparison. Then it shows that
USB2.0 provides a cheaper connection, compared to FireWire.
To summarize, there are three proposals for the hardware components for invisible
marker tracking. These are depicted below, together with the total cost of these
proposals. The numbering of the proposals indicates the favourability of the proposal.
Proposal 1 Price Proposal 2 Price
Proposal 3 Price
Ink Pen $18,43 Ink Pen $18,43 Ink Pen $18,43
IR Bullet $99 FireFly2 board $299 FireFly2 board $299
camera
camera
camera
Power Supply $15 CS-mount
adapter
$30 Square IR filter $35,50
USB 2.0 Video $59 Custom made Depends on filter
Grabber
IR filter characteristics
Total price $191,43 Total price $347,43 + price of
custom made filter
5.4 HMD
Total price $352,93
For the choice of an HMD there are a number of options available, which are split up in
2 categories; video-see through and optical see-through. The video see-through variant
is preferred over the optical see-through solution.
The first video see-through solution describes the fully integrated HMD. At first this
seems to be the best solution because it delivers just exactly what we are looking for.
There is no necessity to make adaptations to the HMD; it can be used as it is delivered.
The price may lower the attraction of this solution a bit, but will not cause major
problems.
The stereoscopic add-on for an HMD on its own does not only have an even higher
price than the fully integrated HMD, but also may suffer from possible compatibility
59

Development of an Augmented Reality system using ARToolKit and user invisible markers
issues. Also there are not many HMDs that could be chosen from. This almost leads to a
situation in which there exists no other option than to use it in combination with the
advised HMD. The accumulated price of the add-on and this HMD are the limiting
factor.
The third described video-see through solution has the most promising perspective out
of the three. The bare HMD on its own carries the lowest price, which is a desirable
fact. Adaptation of the HMD, of course also has its price. But besides this, the fact that
the HMD is custom made gives the most possibilities to have it adapted to our wishes.
The type of cameras to give it stereoscopic view can be chosen by us. Also the camera
for detecting the ink is to be chosen by us. The company offers several custom made
solutions, which have a different degree of completeness.
The solution consisting of the HMD with the added camera for detecting the ink is
favoured over the solution that just adds two cameras to the HMD. One can go one step
further and acquire the complete all-in-one solution offered, but it is expected that this
also can be achieved by us ourselves with little effort.
To summarize, there are three proposals for the video see-through HMD. These are
depicted below, together with the total cost of these proposals. The numbering of the
proposals indicates the favourability of the proposal.
Proposal 1 Price Proposal 2 Price
Custom made video
see-through HMD
Total price
- Fully
integrated
HMD
Depends on chosen
configuration
Proposal 3 Price
€7300 Advised HMD
(nVisor SX HMD)
$23,900
HMD add-on $8,900
Total price €7300 Total price $32,800
The first proposal can consist of several configurations. Also there exists an order of
favourability between those possible configurations. These are depicted below. The
numbering of the proposals indicates the favourability of the proposal
Proposal 1 Price Proposal 2 Price
stereoscopic
HiRes-800PC
camera for
detecting invisible
medium
€3275 stereoscopic
HiRes-800PC + +
depends on camera for
choice for detecting invisible
proposal medium
Computer with 2
video cards
Proposal 3 Price
€3275 stereoscopic
HiRes-800PC
depends on
choice for
proposal
determined by
company
€3275
Total price > €3275 Total price > €3275 Total price €3275
For the optical see-through HMD there are just two possibilities. Out of these two the
first described one, the LitEye-500, is preferred over the second. The high price of the
nVision DataVisor HiRes is debet to this. When using the LitEye-500 one has to take
into account that two of them are needed for creating stereoscopic vision, and that it
may be cumbersome to put them in the right configuration. The two proposals are
60

Development of an Augmented Reality system using ARToolKit and user invisible markers
depicted below, together with the total cost of these proposals. The numbering of the
proposals indicates the favourability of the proposal.
Proposal 1 Price Proposal 2 Price
LitEye-500 €2950 nVision DataVisor HiRes €20445
Total price
€2950 Total price €20445
5.5 Final choice on system components and perspective
Now that has been presented several proposals for the components with their respective
desirability, here the final choice for system components is stated. Hereby the rationale
behind the choice is not laid out; just the results are given.
For the invisible marker tracking components the choice has been made for the first
proposal. This consists of the following:
• IR Ink Pen
• IR bullet camera with 715nm IR filter
• 12 VDC Power supply
• USB 2.0 Video Grabber
The choice for the stereoscopic video see-through HMD is taken from the second
proposal:
• ARVision3D, Goggles model
The choice for the optical see-through HMD is taken from the first proposal:
• Liteye-500, SVGA model
With these chosen system components the next step is to merge them into a complete
system that is capable of supporting AR applications based upon ARToolKit with user
invisible markers. Two directions will be taken to cover the complete possibilities, to
have a greater chance on success and, if successfully implemented, to have a choice on
which approach to use and further develop in the future. The two variants that will be
tried are the follwing:
• Implementing an AR system based upon the selected hardware components for
invisible marker tracking, and the selected stereoscopic video see-through HMD.
• Implementing an AR system based upon the selected hardware components for
invisible marker tracking, and the selected monoscopic optical see-through
HMD. Hereby needs to be noted that this solution will only cover one eye.
61

Development of an Augmented Reality system using ARToolKit and user invisible markers
6. References
[Azu01] Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., MacIntyre, B.,
Recent Advances in Augmented Reality, IEEE Computer Graphics and
Applications, vol. 21, n. 6, Nov/Dec, pp. 34-47, 2001
[Azu94] Azuma, R., Bishop, G., Improving static and dynamic registration in an optical
see-through HMD, Proceedings of SIGGRAPH 94, pp. 197-204, 1994
[Azu97] Azuma, R., A Survey of Augmented Reality, Presence: Teleoperators and
Virtual Environments, vol. 6, n. 4, August, pp. 355 – 385, 1997
[Bell02] Bell, B., Feiner, S., Höllerer, T., Information at a Glance, IEEE Computer
Graphics and Applications, vol. 22, n. 4, Jul/Aug, pp. 6-9, 2002
[Bim04a] Bimber, O., Raskar, R., Modern Approaches to Augmented Reality, 25th
Annual Conference of the European Association for Computer
Graphics, Interacting with Virtual Worlds, Tutorial 8, pp. 1-86, 2004
[Bim04b] Bimber, O., Combining Optical Holograms with Interactive Computer
Graphics, IEEE Computer (cover feature), pp. 85-91, 2004
[Bow04] Bowman, D.A., 3D User Interfaces: Theory and Practice, 2004
[Bru03] Brunner, S., Bimber, O., Mader, S., Report on Tracking Technology, 2003
[Cot04] Cotting, D., Naef, M., Gross M., Fuchs H., Embedding Imperceptible Patterns
into Projected Images for Simultaneous Acquisition and Display, IEEE and
ACM International Symposium on Mixed and Augmented Reality
(ISMAR’04), pp. 100-109, 2004
[Emm02] Emmelkanp, M.P.G., Krijn, M., Hulsbosch, H., de Vries, S., Schuemie, M.J.,
van der Mast, C.A.P.G., Virtual Reality Treatment versus exposure in vivo: A
Comparative Evaluation in Acrophobia, Behaviour Research and Therapy, vol.
40, n. 5, pp. 25-32, 2002
[Hof00] Hoff, B., Azuma, R., Autocalibration of an Electronic Compass in an Outdoor
Augmented Reality System, Proceedings of International Symposium on
Augmented Reality 2000, pp. 159-164, 2000
[Hua04] Hua, H., Brown, L.D., Gao, C., Scape: Supporting Stereoscopic Collaboration
in Augmented and Projective Environments, IEEE Computer Graphics and
Applications, vol. 24, n. 1, Jan./Feb., pp. 66-75, 2004
[Isd00] Isdale, J., Nov/Dec 2000: TechReview Augmented Reality,
http://vr.isdale.com/vrTechReviews/AugmentedReality_Nov2000.html, 2000
[Jua04] Juan, M.C., Botella, C., Alcañiz, M., Baños, M., Carrion, C., Melero, M.,
Lozano, J.A., An Augmented Reality System for treating psychological
disorders: Application to phobia to cockroaches, Proceedings of the Third IEEE
62

Development of an Augmented Reality system using ARToolKit and user invisible markers
and ACM International Symposium on Mixed and Augmented Reality, pp. 256-
257, 2004
[Jul00] Julier, S., Baillot, Y., Brown, D., Rosenblum, L., BARS: Battlefield Augmented
Reality System, NATO Symposium on Information Processing Techniques for
Military Systems, 2000
[Kat99] Kato, H., Billinghurst, M., Marker tracking and HMD calibration for a
video-based augmented reality, Conferencing system, 2nd IEEE and ACM
International Workshop on Augmented Reality (IWAR’99), pp. 85-94, 1999
[Kat00] Kato, H., Billinghurst, M., Poupyrev, I., ARToolKit 2.33 Documentation,
http://www.hitl.washington.edu/artoolkit/tutorials.htm, 2000
[Kau04] Kaufmann, H., Geometry Education with Augmented Reality, Dissertation,
2004
[Kij97] Kijima, R., Ojika, T., Transition between Virtual Environment and Workstation
Environment with Projective Head-Mounted-Display, Proceedings of IEEE
Virtual Reality Annual International Symposium 1997, pp.130-137, 1997
[Koy03] Koyama, T., Kitahara, I., Ohta, Y., Live Mixed-Reality 3D Video in Soccer
Stadium, Proc. of The 2nd IEEE and ACM International Symposium on Mixed
and Augmented Reality (ISMAR2003), pp.178-187, 2003
[Kuk98] Kukulakos, K.N., Vallino, J., Calibration-Free Augmented Reality, IEEE
Transactions on Visualization and Computer Graphics, vol. 4, n. 1, January-
March, pp. 1-20, 1998
[Loo04] Looser, J., Billinghurst, M., Cockburn, A., Through the Looking Glass: The
use of Lenses as an interface tool for Augmented Reality interfaces,
Proceedings of the 2nd international conference on Computer graphics and
interactive techniques in Australasia and Southeast Asia, pp. 204-211, 2004
[Mak97] Makino, H., Morishita, F., Abe, Y., Yamamiya, S., Hasegawa, M, Ishii, I.,
Nakashizuka, M., 3-D Object Recognition and Description: A Method for the
Visually Impaired Using an Invisible Bar Code, System and Computers in
Japan, vol. 29, n. 8, November, pp. 1-8, 1997
[Mil94] Milgram, P., Takemura, H., Utsumi, A., Kishino, F., Augmented Reality: A
Class of Displays on the Reality-Virtuality Continuum, Proceedings of
Telemanipulator and Telepresence Technologies, pp. 282-292, 1994
[Par04] Park, H., Il-Park, J., Invisible Marker Tracking for AR, Proceedings of
ISMAR, pp. 272-273, 2004
[Pen04] Penn, A., Mottram, C., Fatah gen. Schieck, A., Wittkämper, A., Störring, M.,
Romell, O., Strothmann, A., Aish, F., Augmented Reality meeting table: a novel
multi-user interface for architectural design, pp. 213-231, 2004
63

Development of an Augmented Reality system using ARToolKit and user invisible markers
[Pin02] Pinz, A., Brandner, M., Ganster, H., Kusej, A., Lang, P., Ribo, M., Hybrid
Tracking in Augmented Reality, ÖGAI Journal, vol. 21,
n. 1, pp. 17-24, 2000
[Ras02] Raskar, R.., Ziegler, R.., Willwacher, T., Cartoon Dioramas in Motion,
Proceedings of International Symposium on Non-Photorealistic Animation and
Rendering, pp. 7-12, 2000
[Ras03] Raskar, R., van Baar, J., Beardsley, P.A., Willwacher, T., Rao, S., Forlines, C.,
iLamps: Geometrically Aware and Self-Configuring Projectors, ACM
Transactions on Graphics, vol. 22, n. 3, pp. 809-818, 2003
[Rei98] Reiners, D., Stricker, D., Klinker, G., M¨uller, S., Augmented Reality for
construction tasks: Doorlock assembly, Proceedings of First International.
Workshop on Augmented Reality (IWAR), pp. 31-46 , 1998
[Sch00] Schuemie, M.J., Bruynzeel, M., Drost, L., Brinckman, M., de Haan, G.,
Emmelkamp, M.P.G., van der Mast, C.A.P.G., Treatment of Acrophobia in VR
: A Pilot Study, F. Broecks and L. Pauwels (Eds.) Conference
Proceedings Euromedia 2000, pp. 271-275, 2000
[Sch01] Schwann, G., Physically Based Animation in an Augmented Environment,
2001
[Sch04] Schnaider, M., Röddiger, S., Seibert, H., Schwald, B., Implementation and
Evaluation of an Augmented Reality System Supporting Minimal Invasive
Interventions, Federal Ministery of Education and Research: Virtual and
Augmented Reality Status Conference 2004 [CD-ROM], 10 p., 2004
[Sin04] Singh, S., Cheok, A.D., Ng, L.G., Farbiz, F., 3D Augmented Reality Comic
Book and Notes for Children using Mobile Phones, Proceeding of the 2004
conference on Interaction design and children: building a community,
pp. 149-150, 2004
[Sor04] Sorantin, E., Werkgartner, G., Beichel, R., Bornik, A., Reitinger, B., Riccabona,
M., Wegenkittl, R., Fuhrmann, A., The Virtual Liver Surgery Planning System,
European Congress of Radiology: The Matrix, 2004
[Tan03] Tang, A., Owen, C., Biocca, F., Mou, W., Comparative Effectiveness of
Augmented Reality in Object Assembly, CHI 2003, vol. 5, n. 1, January, pp. 73-
80, 2003
[Val98] Vallino, J., Interactive Augmented Reality, Ph.D. Thesis, Department of
Computer Science, University of Rochester, Rochester, NY, April 1998
64

Development of an Augmented Reality system using ARToolKit and user invisible markers
7. Abbreviations
AR - Augmented Reality
AV - Augmented Virtuality
CCD - Charged Coupled Device
DOF - Degree Of Freedom
GPS - Global Position System
HMD - Head Mounted Display
PDA - Personal Digital Assistant
SUD - Subjective Units of Distress
FOV - Field Of View
VR - Virtual Reality
VRC - Virtual-Reality Continuum
65