THIRD ANNUAL SCREENS ISSUE - MediaPost

Fusing data from sensors already in most smartphones with mobile
cameras allows Metaio to align augmented reality information with gravity.
Since the camera knows what is up or down and understands what it
sees, adding digital information gets easier, like orienting windows on the
side of a building. Realtors could then offer in-depth views of available
apartments. Marketers could overlay the building with digital murals.
for the Metaio R&D department:
How do we combine
sensors in a way that enhances
optical recognition and image
tracking? In other words, how
could we make the phone’s
camera smarter? Though the
methods were complex, the
answer was simple. We align
everything according to one of
the most basic principles of our
day-to-day lives: gravity.
Through algorithms and
the fusion of sensors (movement,
orientation) with optical
data from the camera, we were
able to teach the smartphone
how to see with respect to up
and down. Imagine trying to
do anything in your daily routine
without a sense of up and
down — or even better, imagine
if the concept and effects
of gravity were completely
alien to you. You’d be floating
around in near chaos trying to
TO MOBILE
make sense of your environment,
much like the camera on
your phone before we taught it
to understand its surroundings.
We taught it orientation.
With our software, mobile
cameras no longer search for
independent points of contrast
when trying to recognize
images. Aligning the camera
with gravity supplies a constant
framework for processing
images. Take the facade of
a brick building: Every window
on that building has an almost
identical set of contrasting features
(see Fig. 1). With Gravity-
Aligned Feature Descriptors
(GAFD), the camera automatically
orients each window with
respect to up and down. The
inertial sensors work to give
the camera an understanding
of what it sees. With a standardized
orientation, overlaying
the side of buildings with
digital information becomes
very straightforward.
The implications border on
science fiction. Looking at an
office building from the street
level and immediately determining
what windows correspond
to different commercial
real estate listings: selecting
the highlighted window yields
a 360° navigable photograph,
pricing and lease information,
and direct access to the realtor.
An urban scene marred
by graffiti is restored to its
original state for photographs.
Virtual, interactive murals that
span city blocks. Every poster
or billing for a film or concert
yields a corresponding visual
key and information on how to
attend. Furnish your home or
office without tape measures
or blueprints. And that’s just
the beginning.
GAFD works on a smaller
scale as well. Recognition comes
faster; tracking is smoother —
and it sticks. That virtual bunny
rabbit you just discovered on
the cover of a magazine isn’t
going to jitter uncontrollably
(unless it’s supposed to). Virtual
objects and animations will be
more attached to their anchors
— whether they’re a building,
a newspaper, a product package
or a business card — and
they’ll behave more realistically.
Candle flames on virtual birthday
cakes will always point to
the sky, just like in real life, no
matter how you tilt the reference
image. This technology
is so subtle that you may have
even believed that AR already
worked like this.
It didn’t. But it does now.
In the not-so-distant
future, we will be able to
access astounding amounts
of information from our
surroundings, just with our
smartphones. There may
come a day when it suddenly
hits you how intuitive the process
of interacting with your
surroundings has become.
And you’ll pause, marveling
that it all began with a phone’s
camera and a gyroscope.
Welcome to the augmented
world, where the digital is as
natural as gravity.
Spring 2012 MEDIA MAGAZINE 59