Conversations of Presence - Peach | - Presence Research in Action

peachbit.org

Conversations of Presence - Peach | - Presence Research in Action

S t a r l a b

PEACH

2


S t a r l a b

PEACH

3


©STARLAB BARCELONA SL 2009

S t a r l a b

PEACH

4


TABLE OF CONTENTS

PREFACE!

8

DAVID BENYON!

11

OLAF BLANKE!

16

WIJNAND IJSSELSTEIJN!

22

MARÍA SÁNCHEZ VIVES!

27

MARTIN BUSS!

32

MEL SLATER!

36

S t a r l a b

PEACH

5


MATTHEW LOMBARD!

42

RALPH SCHROEDER!

51

PAUL VERSCHURE!

58

GIULIO RUFFINI!

65

PHIL TURNER!

72

Summary Table!

75

S t a r l a b

PEACH

6


S t a r l a b

PEACH

7


PREFACE

What is reality? Is there anything really out there? What are the ingredients that

make a perceived scene in a CAVE feel so real as to make our hearts race and

bodies tremble? Certainly, the human brain is involved (it must be central!) in

creating the illusion of reality. In fact, everything, from quanta, your neighborhood,

the distant stars and down our own physical bodies is a mental construct.

And new science and technologies are seeking to understand this “reality construction”

process and, as a result allow, us to tweak our vital experiences.

The issue of what the field of Presence is remains somewhat fuzzy, as the following

interviews show. Yet, the core interest remains on scientific and technological

aspects involved in creating “real”-feeling experiences and the the impact

of such new interaction technologies on social relationships and networks.

Furthermore, once can argue from physical fundamental principles that reality

is all about information. This explains why interest and research in Presence is

booming today hand in hand with the explosion in information technologies,

and its close relationship with ICT (Information and Communication Technologies).

Community building is not trivial. Presence is truly a scientific and technological

multi-disciplinary field requiring overlapping work in human cognition,

including social aspects, human-machine interaction and machine cognition.

Progress in these three areas is needed, yet together they span a very wide subset

of human knowledge. How do we bring the right people together? This is

certainly a challenge.

Despite the different views on the definition, it appears to be easier to reach a

consensus in terms of a fundamental goal of the field: to develop science to

achieve successful replacement/interaction (i.e., Presence, being there, with

some body, with others–real or not–some time) and develop technology opening

up a wide range of powerful applications. Designing and carrying out a

visionary project with this goal in some concrete form would have the benefit of

concentrating the efforts and building a strong community, while opening up a

large set of powerful applications. As an example, consider telepresence: traveling

and being there with others in a satisfying way without moving. This is no

longer science fiction, and current sustainability issues demand we move forward

as fast as we can. Physical travel consumes too much energy and time.

S t a r l a b

PEACH

8


This book stems from PEACH, a coordination action funded by the Future and

Emerging Technologies program of the European Commission. Peach, among

other things, aimed at collecting information from the Presence research community

(what is Presence, what are the main scientific and technology issues to

address?) and based on these and other inputs, to provide a roadmap for future

research. As part of Peach, Starlab conducted a series of interviews with key

players in the field to try to answer such questions and help define a research

and technology roadmap, as well as disseminating the core questions and values

of this interdisciplinary field.

In this short book, we have collected the ideas and proposals from a relatively

large set of researchers who were interviewed during the Presence 2007 event

which took place in Barcelona. The questions asked touched on various key

aspects, including where the focus of the field should be, current research activities,

future plans, ethical issues, technology bottlenecks, favorite applications

and philosophical and existential implications. The main themes detected in

these interviews are that the core of the field lies in cognitive neuroscience (the

brain stuff of Presence), and that there are two somewhat differentiated subfields.

One focuses on physical/natural/perceptual Presence aspects, relying on

using Immersive technologies and related approaches, the other on social aspects

(interaction, co-Presence, social side-effects).

We give special thanks to the researchers which provided generously their time

and effort for the interviews and ISPR’s Presence 2007 for hosting the interviews.

!

! ! ! ! ! Giulio Ruffini, Editor.

! ! ! ! ! Barcelona, February 18, 2009.

! ! ! ! ! ! ! ! !

S t a r l a b

PEACH

9


S t a r l a b

PEACH

10


DAVID BENYON

Presence is absolutely central to the future of

Human Computer Interaction”

I am a professor of Human computer systems in Napier University , Edinburgh. I

started way back in 1984 my first publication in Human Computer Interaction and I

have being involved in that field ever since. Currently, I’m involved in Companions, a

project that aims to develop a virtual conversational 'Companion', an agent or 'presence'

that stays with the user for long periods of time, developing a relationship and

'knowing' its owners preferences and wishes.

What are you currently working on?

The main project we have ongoing at the moment is called Companions and it

has to do with the development of a new form of interface, a new form of interaction

I should say, particularly with the Internet. The fundamental idea is that

the interaction will be personalized. It will know enough about you that you

won’t have to fight through all the information that is out there: you will get

personalized information coming back. We use the term “companion” because

we are trying to evoke a sort of anthropomorphic response, to get people to

think as if it were your friend although it wouldn’t necessarily have to be an

avatar or a picture.

The other thing we do is participating in Peach, the coordination action for

Presence research, where we lead the outreach and public understanding side

of the work.

What is your definition of Presence?

Presence is being connected: when you feel really connected with a person or a

place, then you are feeling present. If you are at a boring lecture and it is very

easy for you to drift off you wouldn’t be saying you feel present. But if you are

watching a great movie and you feel really identifying with the characters, then

S t a r l a b

PEACH

11


you would be saying you feel present. I like one of John Woodward’s examples:

when you are rock climbing you have to feel really present because your life is

at stake whereas if you are lying on a beach with no danger you don’t really

need to feel present. So Presence has to do with the feelings of being connected

to things beyond where you are just sitting or where you are just physically

located.

Presence is a wide field. Is there a need for focus?

No. I think we need to embrace the whole range of issues that currently come

under Presence research. Some of that would be of traditional Tele-Presence

work but others would be much more modern, sociologically based new approaches

in which we are more concern with connecting people. I think that

sort of a liberal and open approach to the work that people are doing within the

Presence community is probably fine.

Is Presence a science or is it a technology?

Presence is an area of a study, a discipline if you like. It uses appropriate scientific

methods withdrawn from different areas and evolves its own to look at his

own things. I just think Presence is a particular discipline to do with that whole

feeling of being in touch.

Where would you situate the discipline of Presence?

It’s really akin to psychology or sociology, to one of the big social sciences. So

Presence is a social science, if you like.

What brought you to Presence?

We were involved in a project about photorealistic representations of places. It

was very much a technological focused project and we were brought in to help

with the measurements,. They had people who took photos, people who did the

rendering, people with a cave, people with HMD’s... What they didn’t have was

any people to do the studying, the measuring, the understanding of what was

affecting Presence. That brought us in to that particular area.

What keeps you interested?

I think it is absolutely central to the whole future of Human Computer Interaction.

Whether or not it is the design of companions or whether or not it is the

S t a r l a b

PEACH

12


design of Telecare systems for the elderly, we are trying to do things from a distance

and we have to design that properly to make them a great and real experience.

Presence is a lot of fun for researchers, but where would you see the first applications?

We have already seen many. Some of the best live things are about Presence so

Imax cinemas are already taking care at the current time. Or in Disneyworld,

where they bring together the mechanics, the visual, the audio and every all the

other senses to give you a real sensation of being in a completely alien environment.

What would you want to see as an application for yourself?

I am not too worried about what it is just now, but in the future it is going to be

keeping in touch with my children. To be able to maintain good contact; not just

a telephone call but a more intimate level, as if they were actually coming to

visit me. I think that is going to be a big area: systems to keep the elderly in

touch with their friends and family.

What are the ethical issues associated with Presence?

There are huge and many issues. Firstly it is this whole business about creating

things that are unreal and yet making people believe they are real. That is ethically

very dangerous. There are also further issues to do with the movement

between the real and virtual worlds: when we are designing experiences to help

people to learn things and then they transfer that experience to the real world.

There is also a sort of ethics to do with how we treat a virtual representation of

ourselves. If we meet in Second Life: is it OK for me to beat you up? I would say

not, but that is not actually going to hurt you physically. You do identify, you

feel present with yourself in a virtual environment, so we have responsibilities

to behave ethically in those circumstances as well.

What are the technology bottlenecks in Presence today?

All of them. Although the results are very impressive technology is still not

good enough: refresh rates are still too slow, cave images are still pixelated and,

rather than looking really solid, things look very see-through. There is also the

S t a r l a b

PEACH

13


Haptics side of it, that has to be seriously improved to provide a real sense of

touch at a distance. If we want to have real Telemedicine, in which you really

are operating on someone who is a long way away, we need to have all the

senses reacting in real time.

Is there any experiment you would like to do but you haven’t been able to?

No, I think in our area it is OK because we are not really working on that technological

end but on the usage. We can always almost mock out future technologies

to look at what effect they may have, and that is what we do a lot of

times with the Artificial Companions idea: instead of being a real avatar really

talking, we use something like Skype. You have somebody talking and they

answer radio alike, which makes it look as if the avatar is talking. Those things

are quite easy to create to look at the social side of things.

Will Presence research change our conception of reality?

Sure. Definitively, yes. It is going to be all mixed up. We are going to consistently

have experiences which mix the real and virtual, so the whole idea of

distinction between real and virtual is going to gradually vanish as the technology

becomes increasingly a part of the mediated experience.

Has research in Presence changed the way you see the world?

It has changed the way we see the world because we did a lot of work on the

understanding of the sense of place, what it means to be somewhere. The traditional

definition of Presence is “to be there” and we took the “there” very seriously,

investigating what a sense of place was and how it can be affected by the

technology. I think that is probably the biggest effect that had on me.

Could you summarize your work in just one question you are trying to answer?

How to design satisfying, exciting, good and great experiences for people?

S t a r l a b

PEACH

14


S t a r l a b

PEACH

15


OLAF BLANKE

“How does the brain distinguish my body from

your body?”

I am a medical doctor by training and I did my PhD in neurophysiology on eye movement

control. Currently, I’m working as an assistant professor of Cognitive Neuroscience

at the Swiss Federal Institute of Technology in Lausanne, where I lead a multidisciplinary

team of biologists, psychologists, medical doctors, physicists, engineers and

computer scientists. We try to understand how the brain represents the own body: the

functional and neural mechanisms of body perception, corporeal awareness, and self

consciousness.

What are you currently working on?

We are a multidisciplinary team of medical doctors, biologists, psychologists,

physicists, engineers and computer scientists trying to understand how the

brain represents the own body. An example would be this arm. How is this represented

in the brain? I can look at my left hand and this goes to the visual

brain; I can close my eyes and I still feel exactly where my arm is. What puts

these two representations together? Because I have the experience of just one

arm!

Another question is: what helps me distinguish my body from your body? Because

from the brain’s point of view it is not evident. We are looking at this kind

of questions using experimental psychology, using questioners and, most importantly,

doing neural imaging to find out which brain area is involved in solving

this tasks.

What is your definition of Presence?

I’m very new in this field so for the moment I wouldn’t have a definition. I am

most aware of definitions of Presence such as being here or being there. The

being there is a spatial way of formulating where I am localized, where the self,

S t a r l a b

PEACH

16


the I, is localized. Presence and Virtual Reality are ways of studying the mechanisms,

functional, behavioral and brain-wise, to manipulate this sensation.

Presence is a wide field. Is there a need for focus?

It depends, as in other fields. I think it is maybe good to have different approaches,

different false eye, and try to see how they come together at a later

point.

Is Presence a science or is it a technology?

It is a scientific field much dependent on technology. There is not a gold standard

yet that everybody will use for the next ten years. There are always new

techniques coming in and replacing older ones. And while somebody has just a

technological advance over other groups we will still have to talk a lot about

technology.

Where would you situate the discipline of Presence?

Computer science and engineering have been the driving forces and then there

are fields of science, such as psychology, experimental physiology, neuroscience

and neurology, who can make great use of it. And there are great opportunities

for arts, too. I think Presence will always remain a very interactive multidisciplinary

field.

What brought you to Presence?

Basically reading. Since we are looking at body representation and self, what is

a very basic form of self? Where am I in space? Presence links by the phenomenon.

We are trying to describe it in brain states that account for such experiences

and Presence research has developed tools that allow to manipulate, to

systematically study these aspects.

What keeps you interested?

I think it is a very hard problem from the scientific point of view to really understand

what Presence is, to really nail down the few mechanisms that would

explain it all. I don’t think we will come up with something like that in Presence,

or maybe we will, but it is a rather new field and I think there is a lot to be

developed. There is a relationship of these findings, if they come, with respect

to other fields that are adjacent: it may be relevant for philosophy, physiology,

S t a r l a b

PEACH

17


neuro-imaging. At the end it interests everybody! We all want to find out who

we are, what we are, what makes this moment special, open questions that cognitive

neuroscience has started looking at over the last five to ten years. So it is

an interesting research field for the cognitive neuroscience and I think it is the

right moment to fuse interests.

Presence is a lot of fun for researchers, but where would you see the first applications?

Education might be a way. In the Geneva University Hospital Virtual Reality

and related applications are used in order to plan an operation before you do it.

Young neurosurgeons can already relive much closer and in three dimensions

what the main surgeon is actually doing. Then, Teleoperating systems, by making

you present at higher levels at web sites or in certain three dimensional

cues. In Architecture, I could envision lots of applications, not just for the user

who wants to build a dream house, but also for the architect planning it. It must

be fantastic, for an architect to see the house at that time you drew it! Walk

through it and find certain mistakes, make this better or see that one idea works

less well. And for brain science, which is an interest I have, having three dimensional

animations and getting motor cues while you’re looking at the brain,

might be an interesting way to render more evidence spatial aspects

What would you want to see as an application for yourself?

I would like to see Virtual Reality environment and Presence related questions

as well established techniques in the cognitive neuroscience. That there are certain

well established paradigms that we can agree on and basic equipment that

many labs can afford. Also, systems should get more user friendly, so you don’t

need heavy loads of engineering power and people just to do a very simple

experiment.

What are the ethical issues associated with Presence?

I think one has to see how it evolves and look at it closely. What is important

when there are ethical issues is to have a good team who evaluates the ethical

risks. I can see obvious ones: is it good for a four year old kid to spend 20 hours

in a high level Virtual Reality computer game? Probably not but I think this has

to be seen, and we really have to see what the effects are. There might be certain

abilities that are just strengthen very much. It might be very good to have these

S t a r l a b

PEACH

18


environments, and one shouldn’t just see it as a bad computer game. It is important

to let ethics evolve with technologies. Not too late, not too early, not too

restrictive, but in parallel.

What are the technology bottlenecks in Presence today?

It is a very egocentric kind of view but: the head mounted displays. To make

them lighter, easier to set up, with bigger fields of view and more affordable. If

head mounted displays and tracking systems would work in a lower budget

that would be of a great advance. Otherwise, full body tracking is still complicated.

What probably needs to be done also is to reduce this amazing amount of

information that comes from our body and just control and track the pieces of

information that the user most strongly relies on.

Is there any experiment you would like to do but you haven’t been able to?

In the shorter term I would like to explore how far can we take the feeling of

self-hood away from our bodies and into an avatar? What is really needed to

take it to a higher level? Another would be to use Virtual Reality to generate

out-of-body experiences.

In a longer term, I would like to find out how these predictable states that you

can induce by Virtual Reality are related to brain activity. We are still far away

from this, but it would be a real dream to understand which areas and which

activity patterns relate to certain forms of well defined Presence.

Will Presence research change our conception of reality?

I think it will not profoundly change what everybody thinks about me or you

unless you spend your whole life in such an animation. Science has found a lot

of explanations for illusions: we still find the rainbow magnificent although we

just know it is the same color of light that is just refracted in different ways. So

we will still love our experiences, our self-hood and what makes as human and

I don’t think Virtual Reality will modify it in a very systematic way. After all,

we live out here without Virtual Reality equipment.

Has research in Presence changed the way you see the world?

It has not really been my research in Presence but to work with a number of

conditions where this basic centre of awareness that we all have, where we lo-

S t a r l a b

PEACH

19


calize the self, can be profoundly disturbed. Seeing that for others I have assumed

that the same mechanisms are present for me, and that simple manipulation

suffices to modify this representation. This is very intriguing and works as

an explanation for the general population but when you apply it to yourself

there are always some limits that you prefer to just leave unexplained.

Could you summarize your work in just one question you are trying to answer?

How does the brain generates my visual spatial perspective of seeing this scene

from here? Why is it body-centric? How is this generated, and by which brain

areas?

S t a r l a b

PEACH

20


S t a r l a b

PEACH

21


WIJNAND IJSSELSTEIJN

“People will have Presence experiences without

actually being aware”

I work as an assistant professor at the Human Technology Interaction Group of the

Eindhoven University of Technology. My specialty is in Psychology of Media. My current

research includes embodiment, bodily identification, and especially how non biological

props and tools can become part of our body.

What are you currently working on?

Relating to Presence I’m working on embodiment, bodily identification, specially

on how non biological props and tools can become part of our body. Another

project that I am currently involved in is more on side of Telerobotics, and

has to do with minimally invasive surgery and assisting surgeons to give them

haptic feedback. We have another couple of projects that are more related to

social Presence. We are working for instance on the PASION project, where we

are looking at how to enrich current communication media with contextual

data, such that people have better mental model of the other person and more

emotional connectiviness and group identification.

What is your definition of Presence?

I’m quite inclined to just trying limit it a bit. I would say Presence is the sense of

being there in a mediated environment. It is actually not just a subjective experience:

it should also be conceptualized as a phenomenon that can be measurable

at multiple level of responses. For my PhD work, I’ve looked at for instance

behavioral indicators of Presence, such as how people responded posturaly to

certain mediated stimuli to see it that correlates with subjective indicators of

Presence. So, short-end definition? I am still sort of sticking with that. I am

aware of definitions like the one by Lombard: “Perceptual illusion of non -mediation”,

I think it is very elegant to describe the large body of literature it’s out

there, but I do feel that there is a bit of scope creep, if you like. A bit of a prob-

S t a r l a b

PEACH

22


lem with delimiting exactly what we mean and what we are measuring in particular

experiments. Whatever your definition is, you should be very explicit

about it so that everybody knows the kind of Presence you are working on,

instead of just calling everything Presence.

Presence is a wide field. Is there a need for focus?

Yes, there is a great need for focusing in terms of definition but I don’t think

there is such a need in terms of the number of disciplines that are occupying

themselves. Presence is a very rich phenomenon that will attract scientists from

a number of fields and, as long as we agree on a common vocabulary and set of

measures, a number of different disciplines can reflect on this phenomenon

from their own unique viewpoint and take different things from the Presence

research field. A engineer who wants to optimize a sense of Tele-Presence for a

Tele-robotics system is different from someone who is into entertainment systems

for games for instance. Presence experience is important for both fields but

in very different ways, so they can contribute and take in very different ways.

Is Presence a science or is it a technology?

I would say it is a science. As a research field it includes both technological sciences

and humanities. But there is also a very important technological component

to engendering Presence.

Where would you situate the discipline of Presence?

I would see it as a multidisciplinary undertaking which is going to be positioned

somewhere in telecommunication, computer science, psychology, neuroscience,

and the social sciences as well

What brought you to Presence?

Ah! That’s an interesting question! I applied for a PhD scholarship at the University

where I got the job, They were looking at stereoscopic displays and they

wanted me to evaluate them in terms of their perceptual experience. And one

things that we found was that a term like image quality, which would be used

typically for evaluating displays, was not sufficient to describe the complete

experience that people were having if they were confronted with a volume in

space. That brought us to think of the concept of Presence as one potential

evaluation metric in these more immersive displays. Later I gave a presentation

S t a r l a b

PEACH

23


in the US that was picked up by some people from the Commission and they

asked me to write a vision document which ended up to be the basis for the first

call of the Presence project, funded by the European Commission.

What keeps you interested?

I don’t think we have cracked the main issues yet. I have a sense, an intuition,

that Presence is very important for understanding some of the brain processes

behind consciousness and it might be a path into that area. Also, we are working

on the bodily self identification and the media environments allow us to

play with all kinds of experiences in ways that unmediated environment do not

allow, so I see Presence technology as a tool to research the mind. And as a psychologist

I am really curious about how the mind and the brain work! Another

particularly appealing aspect of this area is that you stay in touch with the latest

developments in media, where there are a lot of interesting experiences to be

had.

Presence is a lot of fun for researchers, but where would you see the first applications?

I think there are already applications, mainly in Tele-robotics, but also in Telecollaboration

and Tele-conferencing. Companies like Hewlet- Packard and

Cisco systems have all launched last year Tele-Presence systems that would

help business people deleting them a bit of the travel and also enabling teams to

work more effectively together. For them, for this industry, it is about real dollar.

The prove of the pudding is in the eating so we can be very philosophical

about the topic on the research side but they have to sell those systems. I think

this field is definitely a growth market for our research area.

What would you want to see as an application for yourself?

It is interesting to see that Presence researchers have to still come together

physically, in the same space, in the same city, and have to bear all the hassle of

travel. It would be nice to be able to give a good Tele- Presence experience for

multiple parties such that you could actually join to a conference like this without

feeling you are missing out a major interaction. I would be personally interested

in having that system if it works.

What are the ethical issues associated with Presence?

S t a r l a b

PEACH

24


There are some, I guess, and we are starting to discuss them. On the other hand,

I feel that the Presence field in itself it’s not inherently unethical. It is no secret

that a lot of Presence technologies have been funded by military institutions to

give better training grounds to create better soldiers. This is not a particular

application that appeals to me, but there is a need for that as well and there is

good money in it. So I guess each Presence researcher must weight the the importance

that he or she gives to staying away from certain topic areas.

What are the technology bottlenecks in Presence today?

I think full body interfaces are interesting and although there is some full body

tracking available, these are very intrusive and expensive systems. In essence, a

lot of the systems that I would like are already around in some shape or form,

but they are very costly and intrusive. You are introducing all kinds of intrusion

on to the body, and the freedom of movement of the person, which hampers the

fluency of interaction with mediated environment. Another active field of research

is trying to find reliable measures for experience. This is very important

for Presence, because in the end it is about creating these experiences and seeing

what technological factors weight into that equation of optimizing them. We

are still in early days in terms of finding those. We have a number of questionnaires

out there and there are a number of objective correlates, but there are

very variable results found when relating to those measurement methodologies,

I’m talking now about GSR or Gilman-Gilman scheme response, or also the

research that I did on postural responses. Definitively I wouldn’t recommend

that as a measure of Presence at the moment because it is such a lot of work and

it is very noisy. So yes, I would like to have a better grasp of trying to quantify

the experience.

Is there any experiment you would like to do but you haven’t been able to?

Well, there are always a lot of them but I think I can’t complain. As a laboratory,

we are fairly well outfitted and we have good people so I’m quite happy doing

what we are doing. But I wouldn’t mind to have more of full body interface

interaction devices and doing experiments similar to what we are doing with

the “rubber hand” illusion and the virtual “rubber hand” illusion, and extend it

to full body experience. Also to be able to implement that into gaming scenarios.

So there is for us a niche in this area that we still want to invest in.

Will Presence research change our conception of reality?

S t a r l a b

PEACH

25


II think the availability of high-end media technology is doing that. And as

technological progress is speeding up and people are getting more and more

immersive in very realistic experiences, there will be a point at which people

will not be aware of the fact that they are not looking for through a window but

at a display. And that is actually a point where I am very interested in. There is

an interesting paradox there: if this media gets completely optimized, people

will have Presence experiences without actually being aware. So, to them, it

will not be a particularly interesting experience unless they are made aware that

it is a medium that they are looking at. At the moment media can’t do that

quite convincingly, so people are well aware that they are interacting with a

medium... and still they respond to it in many ways as we expected them to

respond to reality! It is important to distinguish different levels of processing:

the cognitive level is not being fooled by the current media representations but

at the psychophysical level we are actually responding to media as if there are

real. In the evolutionary history of mankind the last hundred years is just a

small blip, and our biological system has not adjusted to that kind of simulation

versus reality distinction yet.

Has research in Presence changed the way you see the world?

It has tuned me to media developments and keep me apprised of what are the

criteria for reality. What kind of experience should we produce for people such

that they will accept it as a more real experience? Trying to answer that question

from a scientific point of view has changed my perception of what media can

do.

Could you summarize your work in just one question you are trying to answer?

How do people know the difference between a mediated environment and a

real environment, and how can we know where to look for the right cues to

make the experiences more realistic?

S t a r l a b

PEACH

26


MARÍA SÁNCHEZ VIVES

“How does the brain generate unity in perception

to deliver a single conscious experience?”

I am Doctor of Medicine and Surgery (PhD) and I study Presence from a neuroscientific

point of view. I work as a professor in Physiology and researcher at the Institute of

Neuroscience in Alicante, where I lead a group focused on Systems Neuroscience research.

My interests include the perceptual roots of Presence in virtual environments

and the exploitation of virtual reality for the understanding of brain function (Mechanisms

of spatial processing, Body perception and 3D visualization of neurons).

What are you currently working on?

My main line of work is systems neuroscience at a very low level, cellular and

systems level. Six or seven years ago I started working on Presence research

and today we still work on it from a neuroscience angle using VR, EEG and

physiological measures. We are also studding different aspects of Presence,

mostly multi - sensory correlations and body representation, but always from a

very neuroscientific point of view.

What is your definition of Presence?

As a phenomenon, we use a definition that is very useful for designing experiments,

which is to measure physiological, behavioral and cognitive responses to

a virtual environment to find out if people feel in this virtual environment as if

they were in reality. On the other side, Presence is a subjective experience and

therefore it is also useful to have a description of the subjective experience of

the subject but this can not be measured and, in order to design experiments, it

is better to have some kind of objective measurements.

Presence is a wide field. Is there a need for focus?

S t a r l a b

PEACH

27


By its nature, it has to be very multidisciplinary and it is very diverse and there

are going to come scientists with different points of view. But at this moment, as

it is a very young field, it should have a definition as restricted as possible, and

by restricted I mean that it should refer to virtual environments only. Once it is

well characterized and we have some kind of physiological or neural signature

for it, maybe it can be expanded to other aspects like Presence trough mobile

phones, or through reading books.

Is Presence a science or is it a technology?

It is a science and it has to be achieved through technology.

Where would you situate the discipline of Presence?

I always see a very strong link between neuroscience and Presence, because

they are very interrelated: if we understand how the brain works then we can

design optimal virtual environments that induce Presence; and, at the same

time, these environments are a very good tool to understand how the brain

works and how perception works.

What brought you to Presence?

In June of the year 2000, I attended a meeting organized by the European Union

in order to define how the Presence research should be addressed. After that we

thought it would be a very good field for a neuroscientist and we applied for a

grant.

What keeps you interested?

Well, there are many aspects. One of them is that using virtual environments is

a very good way to study how our body image is generated. We have been

working during the last couple of years on a virtual arm and on how can we

incorporate this virtual arm into our own body by exploiting a property of the

brain which is the interpretation of multiple sensory correlations. And we are

working now in how to be connected to a whole body, not just an arm!

Another aspect I find very interesting is that Presence in a virtual environment

is very close to what consciousness is in the real world. Presence occurs on a

more restricted environment so it is more accessible experimentally. For instance,

it is easier to look for the neural correlates of Presence, and the neural

S t a r l a b

PEACH

28


correlates of Presence could be very close to the neural correlates of consciousness.!

Presence is a lot of fun for researchers, but where would you see the first applications?

There are many people using it for all kinds of things, like communication or

entertainment but I am particularly interested in the applications that involve

motor rehabilitation, brain training, brain plasticity or recovering of damaged

brain circuits. Also, it can be used in psychological therapies like phobias or

traumatic stress disorders, and for other therapies as well. For example, for

treating burned patients that go through painful procedures. By being very present

in a virtual environment they decrease their attention to their bodies and

therefore they suffer less pain.

What would you want to see as an application for yourself?

I would like to use it for entertainment. It will be fun to be in a more immersive

3D Second Life!

What are the ethical issues associated with Presence?

We follow all the ethical terms and restrictions but I am not particularly concerned

about anyone.

What are the technology bottlenecks in Presence today?

I am very interested in brain computer interfaces, and how to be able to operate

in a virtual world through thinking. To be able to navigate in a closer way as we

do in the real world; that is, to move our body by thought and therefore move

our virtual body. Brain computer interfaces would be also useful for people that

have tetraplegia or any other physical limitation to movement. The main limitation

we have at this moment is the lower spacial resolution of brain generated

signals: one can get very few signals at the same time, so it is difficult to have

very fine movements.

Is there any experiment you would like to do but you haven’t been able to?

We have kind of crazy ideas but there are always time limitations. Right now I

am mostly interested in exploring the body representation by using Virtual Re-

S t a r l a b

PEACH

29


ality so I would be happy to devote more time to any aspect related to that, and

be able to move a virtual body by thinking.

Will Presence research change our conception of reality?

Yes. I am sure that in the following years Virtual Reality will get more integrated

in our daily lives and we will experience it in the streets, so we could

easily confuse what is real and what is not real. In fact, I think, it is already

happening: people that spend eight hours a day running an activity in Second

Life may be not sure if it was real or virtual when they think about it the following

day. So if this environments become 3D and even more immersive then they

will create memories. And even if at the moment you don't confuse what is real

and what is not, when you remember it one week after you may be so.

Has research in Presence changed the way you see the world?

Yes. As neuroscientist I am used to think in more detailed brain mechanisms,

like how a sensory system works, for example, and to think about how a circuit

can underly an action or a perception and so on. Now, because of my research

on Presence and the use of these other technologies my view is much broader

and I see also the brain function from a broader point of view.

Could you summarize your work in just one question you are trying to answer?!

How does the brain generate unity in perception, integrating sensory inputs

and previous experiences to deliver a single conscious experience?

S t a r l a b

PEACH

30


S t a r l a b

PEACH

31


MARTIN BUSS

“If i was a grad student again I would develop

haptic musical instruments”

I am head in chair of the Institute of Automatic Control Engineering (LSR) - Technische

Universität Muenchen. I am mainly interested in control, human machine interaction,

Haptics, telePresence, tele-manipulation and control theory. My current work includes

applications of the control theory such as telepresence, haptic interaction in virtual environments

and medical applications like rehabilitation engineering.

What are you currently working on?

On the basis is control theory. On the top of that we have a variety of interesting

applications, one of them being telePresence, others Haptics, multi model interaction...

Also medical applications, like rehabilitation engineering, where we are

interested in neuro-prosthesis to help people walk again.

What is your definition of Presence?

As we all know, Presence is the sense of being there, and this “there”, is a more

complicated “there”. It can be a virtual environment, but it can also be a real

environment, which is the case in TelePresence applications,where you have a

remote environment which is not accessible. So, its different environments

where we can be present in.

Presence is a wide field. Is there a need for focus?

It is a wide field ranging from Neuroscience to Psychology, to Computer Science

Engineering, Mathematics and Moore system theory. I absolutely agree

that it is a wide area but there is focus: Presence in itself is the sense of being

there and this is the focus among all activities in this area.

Is Presence a science or is it a technology?

S t a r l a b

PEACH

32


It is both. This sense of being there, this sense of Presence, needs both fundamental

science and technology.

Where would you situate the discipline of Presence?

It lives from many other sciences and disciplines, so you can not really place it

into one faculty better that in the other.

What brought you to Presence?

I was a PHD student at the University of Tokyo and I started to work in the area

of manipulation skill in robotic hands. We were interested in acquiring manipulation

skill from human hands to transfer it to multi-fingered robotic

hands, but this was a very difficult area because the technology was not readily

available and mechatronics was just in the beginning. This made me move in

the area of Teleoperation, where you take away the autonomy of the robot and

have it remote controlled, so the difficult tasks and the task decision making are

done by a human who uses an interface to control the robot. Going in this direction

I got interested in the human system interface part and eventually in Haptics.

What keeps you interested?

It is still exciting as we see a lot of technology coming up: computer graphics,

more computing power, more computing resources for more realistic 3D experiences

and, as well as, war mechatronics. In Haptics, mechatronics technology

is becoming really powerful.

Presence is a lot of fun for researchers, but where would you see the first applications?

There are already a lot of applications, in particular in the medical area, where

you look at virtual realities with haptic feedback for virtual pathology. One of

the biggest problems in medical education I hear from my medical colleagues is

that you have huge classes of students but very low number of bodies to study.

People in United States or Switzerland, for example, are pushing very hard in

this area where Virtual Reality meets medicine. I believe we will see in the very

near future a lot of medical applications, where Presence, multimodal Presence

in virtual environments and digital modeling of human bodies and organs will

S t a r l a b

PEACH

33


ecome one of the key enablers. Other than that I would say that a number of

applications in psychotherapy are fairly successful as far as I know.

What would you want to see as an application for yourself?

I would like to see more arts and music coming to the field of Haptics. I am

getting interested right now with a colleague and friend in the United States

who is trying to have a robot playing drums in a band. It is actually that the

robot looks at you as you being a band member, and the band member looks

back, and we do music together, me being the robot, you being the real human

musician.

What are the ethical issues associated with Presence?

Well, certainly if you do psychophysical and psychological evaluation studies

with a lot of human subjects it is very important to keep the ethical standards

very high. There are different rules in different countries but you usually have

to keep the data confidential, just to give an example. Other than that, of course,

there are ethical issues if you go to really brutal computer games. They are being

discussed, so we may have a problem already in this society. I guess you

just have to be careful, basically.

What are the technology bottlenecks in Presence today?

I would say the computing core is still short. It is coming along but it is still

maybe five to ten years away from what we really need. But the technological

needs are not really that important: maybe it is more important to change the

ways we can program the systems. Right now, the engineering and programing

effort goes into software writing, which is very expensive and very tedious.

Maybe we need other ways to assemble and program these systems.

Is there any experiment you would like to do but you haven’t been able to?

As I said, haptic musical instruments, I would like to do that if i was a grad

student again.

Will Presence research change our conception of reality?

We have already seen that. We are already having problems of people getting

involved in spaces like Second Life and loosing touch to the real world. You can

S t a r l a b

PEACH

34


particularly see that in Japan, where they call these people “otaki”, which

means specialist.

Has research in Presence changed the way you see the world?

No.

Could you summarize your work in just one question you are trying to answer?

What's next?

S t a r l a b

PEACH

35


MEL SLATER

“If you respond as if it were real, it is Presence

I am the founder of the Virtual Environments and Computer Graphics group in the

Department of Computer Science, University College London and currently ICREA

professor at the Universitat de Barcelona. I started in the early 1980’s studying, researching

and teaching in computer graphics and that led me on to 3D computer graphics

and interaction by the late 1980’s. Then, in 1990 or so, I got involved in Virtual

Reality research and since then that’s mainly what I have been doing. My major research

interest is in helping to find out what makes virtual reality work for people.

What are you currently working on?

My main research revolves around an integrated project that I lead called

PRESENCCIA. This project is funded by the European Union and studies various

aspects of Presence, ranging from the Neuroscience point of view and trying

to understand how Presence works, and through to the computer science

point of view, which is more technical work in engineering. So in my lab we do

research from both parts: we carry out technical work to build virtual environments

and also we evaluate them through experiments.

What is your definition of Presence?

My definition of Presence as a phenomenon is that when you are going into an

artificially created simulation and you take it as real; not that you believe that it

is real but you respond as if it is real. The way I approach Presence is to look at

what people do, what they say, what they tell me they think, and relate that to

how they would have responded had the events themselves been real.

Presence as a field is two aspects. The scientific side is understanding why this

happens or why it doesn’t happen, what i the condition under which this phenomena

occurs. The engineering side is using that scientific knowledge to engi-

S t a r l a b

PEACH

36


neer Virtual Reality and mixed reality to maximize the chance that it would

happen.

Presence is a wide field. Is there a need for focus?

Absolutely. When this field started within Virtual Reality it was inherited from

another field, called teleoperator systems. In a teleoperator system you basically

place yourself in a system where you see through the eyes of a robot and you

manipulate the limbs of the robot which is in some remote place. The scientists

who studied this and engineered it talked about the idea of being there, which

is being in that remote place where the robot is, rather than being in the lab o

wherever it was that you were.

This same idea was inherited by Virtual Reality, where you know you are standing

in a real physical lab, in a real place, and you are interacting inside the virtual

world instead of a real physically remote world. So it is conceptually the

same thing. When this field started in the 1990’s everyone had a very clear idea

of what they meant by Presence, which was this transfer of the sense of place to

the virtual. Since that time, and specially in the last few years, the term has become

very diluted, to mean anything like the sense of involvement within the

environment; the sense of engagement; extending it from Virtual Reality to even

some people saying the sense of Presence when looking at a mobile phone display...

I think all these things are very interesting to study but when you have

such a diverse definition in such a diverse realm of application you can’t make

any scientific progress. So my believe is to narrow the field to considering Presence

in relation to digitally generated virtual environments and using a definition

like the one I said which is how people respond, which you can actually get

at scientifically, rather than this kind of vague sense of being there which you

can’t really get at scientifically. In fact, I am almost at the point of stopping using

the word Presence and using another term which I coined two or three

years ago called Pretense. If you remember being a child and being really involved

in some game that you were playing with other children, that is an example

of pretending. It is also an example of Presence but it is just not in a digital

environment although probably many of these same things come to play.

Is Presence a science or is it a technology?

S t a r l a b

PEACH

37


It is both. There is the scientific side which is to understand, why it happens,

and then there is the technological side, which is based on the understanding,

trying to reproduce circumstances when it has a hard chance of occurring.

Where would you situate the discipline of Presence?

It is a sub-discipline of Psychology and Neuroscience and it is also related to the

study of consciousness. On the technical side I wouldn’t ‘say it is a part of Virtual

Reality technical research but it gives Virtual Reality technical research an

orientation, which means that the goal of such research is to produce Presence.

This is why in my lab, whenever we develop a new technique in computer

graphics, we evaluate its impact on Presence. So, it gives us a focus for our research.

It’s both scientific and technical.

What brought you to Presence?

It came from some very interesting personal experience I had when I first encountered

Virtual Reality at a SIGRAPH conference in the United States in 1990.

I was invited to put on a head-mounted display by the only company who were

producing them at that time. I was very disappointed first of all just to see some

fat pixels but then I moved my head and suddenly I was in a different place. It

was a very profound experience that ever since I have been trying to understand:

how does this work? How can we make it better? What technology is

needed to produce it? What is the scientific explanation? I was working in the

area of 3D computer graphics and I shifted very slightly away from simply concentrating

on algorithms and users interfaces to saying “OK it is computer

graphics algorithms, but the only user interface I am really interested in is the

immersive one, the Virtual Reality one”.

What keeps you interested?

There are several reasons. One is that I still haven’t recovered from that first

experience, although it was seventeen years ago. Another is that I have always

been interested in things like dreams so it kind of relates to my personal interests.

Also it is very exciting when I construct a Virtual Reality environment and,

even though I know how it works, it still affects me. And it is also really exciting

to see the responses of other people. For example, right now we are running

an experiment where I think it is probably the first time in the world you’re

going through a virtual environment that includes a mirror and shadows of

S t a r l a b

PEACH

38


yourself. This is in a cave system, so you are not wearing a head- mounted display

but you are in a room with four walls where there are back projected from

projectors and a stereo 3D. And you are looking to the scene, and in the scene

there is a mirror where you see a reflection of your own virtual character: as

you move, it moves, as you turn your head, it turns its head and so on. But not

only that: as soon as you have a mirror in a virtual environment you see what’s

going on behind you! What happens in this experiment is that a virtual person

suddenly appears behind you. To me it is really exciting to see everyone suddenly

turning around to look who’s coming in!

Presence is a lot of fun for researchers, but where would you see the first applications?

It’s not always such fun. It is fun to do the programming and the experiments

but then you end up with a lot of data that you then have to spend days or

weeks analyzing. And sometimes that is not so much fun. But… what kind of

applications? Well, given the definition I work from, which is that people respond

at it as if it were real, then any aspect where you want to use Virtual Reality

for training something which they are later going to do in real life. For

example: something that it is better to train in Virtual Reality because doing it in

real life it’s too dangerous, or impractical or unethical. Another example that it

doesn’t seem like training but in a way it is is psychotherapy: it can be used by

psychotherapists in helping people to overcome their problems. As an example

of that, we work with people in the Institute of Psychiatry in London, who

study paranoia. And what we have observed is that when people interact with

virtual characters they exhibit the same kind of tendency as people do in everyday

life. So you look at someone, at that precise moment by accident they just

happen to look away, and some people would make a story in their head:

“whenever I look at that person they look away so they don’t like me.

What would you want to see as an application for yourself?

That is easy. I spend a lot of time going to meetings and traveling. I would

much rather send my virtual character to go to the meetings instead, so I would

just stay home relaxing or reading books on Presence...Or writing books on

Presence.

What are the ethical issues associated with Presence?

S t a r l a b

PEACH

39


Yes, there are ethical issues, as in many realms. If you can create an environment

in which people respond as if it is real, you can use that to cure a phobia

or you can use that to give somebody a phobia. So I think it is very important to

have a proper ethical control when doing experiments with people.

What are the technology bottlenecks in Presence today?

Presence requires multi-modal sensory generation from a computer. The visual

side is pretty OK: we can do quite realistic looking virtual environment; the

sound side is OK: it is possible to generate sound coming from any direction,.

But when it comes to Haptics no matter how much money you spend on that,

there is still no real solution which is practical today: if I walk through a virtual

environment and my elbow happens to brush against a virtual object, I’m not

going to feel anything! I think this is the biggest technical bottleneck that there

is today.

Is there any experiment you would like to do but you haven’t been able to?

Not really, because every experiment I wanted to do so far I have done or I am

doing. What is difficult though is the amount of time it takes to do it. So for

example the one we are doing now, which is on the virtual fire, is taking ten

personal years and a cost of 250.000 Euros. Getting all this different systems to

work together takes a huge amount of time. Also, there are no standards, so if

we create an application here in Barcelona and we want to go and run it on

some machine in Germany we will need six months of more work .

Will Presence research change our conception of reality?

No but it may bring home to people how much our own reality is fake or inside

our head. For example, we were doing our first experiments on paranoia with

ordinary people in which they went in to the virtual environment, which was a

library, and a few virtual characters looked at them. The stories they came out

telling us were incredible! Were like people were talking about them behind

their back, people were looking at them in a funny way, people were friendly or

unfriendly... But none of this was really going on and we knew it because we

programmed it. So it made me realize most of what we think is going on in everyday

reality is coming from inside of our heads. Of course, you get cues from

the real world but so much of what we think is going on is an invention from

our brain based on our own past history and the way we selectively look

S t a r l a b

PEACH

40


around the world, the way we selectively hear things and the way we interpret

things. So Presence won’t change reality but it might make us understand there

is not a reality. There is reality but what we perceive is largely coming from

inside ourselves.

Has research in Presence changed the way you see the world?

Not really because I started from a pretty crazy point of view anyway. I have

always been interested in dreams and out of body experiences, which are now

coming back into science even in the realm of Virtual Reality, so I am very

happy that some of my interests and my research are somehow surprisingly

coming together.

Could you summarize your work in just one question you are trying to answer?

Why do people respond in VR as if it were real when they are fully aware it is

not?

S t a r l a b

PEACH

41


MATTHEW LOMBARD

Presence is the illusion of non-mediation”

I am the president of the International Society for Presence Research and an associate

professor in the School of Communications and Theatre at Temple University in Philadelphia.

What are you currently working on?

One project is trying to take all the different definitions that people have put

forward for Presence and putting them into a scheme, a typology, so that there’s

a unification, a windowing down of some of the duplicating efforts that have

happened defining Presence. We have a long term project that we are doing

related to portrayals of telePresence in popular culture: movies, television

shows, series, novels, comic strips, video games... All kinds of places where it is

not that the media presentation evokes Presence in the user or viewer but it

actually portrays people having Presence experiences. We want to know what

kind of things are portrayed as an indication of what are people’s concerns

about how the technology might be used. What are people thinking now about

how Presence will be used? What do they fear about? What do they want to

happen as a result of the new technologies that are being developed?

The ISPR is sort of a project in itself because there is this whole effort to bring

the community together. One thing we have done recently is a systematic

method for identifying the telePresence literature across all kinds of different

fields and databases. It is a way of bringing the community together. Because, if

you are interested in a Presence question, instead of having to look in all the

places that you might think of looking, you can go to one place and say: “Does

anyone done this kind of thing?” It includes everything, from the traditional

telePresence studies to things that I wasn’t even aware of. For instance, where

people use the phrase “sense of Presence” but they’re not talking about technology

based experience, they’re talking about when you wake up suddenly

feeling the sense of a Presence of somebody else on the room.

S t a r l a b

PEACH

42


What is your definition of Presence?

I guess in 1997 Theresa Ditton and I wrote the piece “At the heart of it all: the

concept of Presence”. In that, we looked at a lot of seemingly unrelated phenomena

that seemed to have something that tied them together which was this

concept of Presence. When we looked at that thread that was in common across

all the phenomenon and the literatures we labeled it “the illusion of nonmediation”,

and I largely still subscribe to that definition. I understand that

people use “a sense of being there in a virtual environment or in a mediated

environment” to get people to understand Presence, but I think that is too narrow.

There are all kinds of phenomenon where people, either purposely or not,

experience technology in a way that is not accurate. That they misconstrue the

way technology is playing a role in their experience in even things as relatively

simple as the fact that I’m wearing glasses. Assuming the technology is good

enough you come to the conclusion very quickly that you habituate to it and

you don’t think: oh, I’m wearing glasses! Most of the time it is invisible to me,

so the technology is transparent. In the last couple of years, some very big players

of the videoconferencing industry have started promoting the idea of telePresence

as a high-end meeting tool where you can have people in different cities

interacting as if they were in the same room. And with high speed communications,

very high definition monitors and sound that follows you start to forget

about the fact that you are using a very sophisticated technology. And it doesn’t

have to be a high tech experience, it can be just watching television. It is the case

of talk show hosts, who look straight at the camera and act as if they’re talking

to you in your living room even saying things like: “We will be right back!” as if

they were leaving and then coming back.

Presence is a wide field. Is there a need for focus?

The danger of having this broad definition is that then you have everybody

says this is Presence or that is Presence or the other thing is. I have been very

frustrated recently, as we worked on this project to identify the literature and

the definitions. People aren’t explicit enough about their definitions and they

often invent new terms and new definitions that are very closely alined with

other things that have already been done. Of course, there is nothing wrong

with that and there are no rules against it but I would like to see more focus. We

don’t want to go tell people they can’t use that word but maybe we can make

S t a r l a b

PEACH

43


them more aware of what has been done and then sort of clear the underbrush

in the field so we can identify the core things.

Is Presence a science or is it a technology?

A science or a technology? Oh, that’s a good question. The people who do this

hi-end videoconferencing systems describe it as a technology: the tools that we

create, the products that we create and want to sell. But, I am definitely of the

opinion that telePresence is not a technology. It is a property of a person not a

thing. I experience telePresence, it is a property of me as the user. There might

be technologies that are very likely to evoke this sense of illusion of non mediation

but it’s not guaranteed; it all happens in your head. So I would definitely

not say it is a technology. I don’t know what it would mean to say that it is a

science. What does it mean to say telePresence is a science? If you are asking if

the field is scientific versus we don’t have the rigor of a scientific explanation

and understanding of the telePresence experience, then I would say we are on

the path toward that level of understanding but we don’t have it now. So I

wouldn’t say it’s a science or a technology.

Where would you situate the discipline of Presence?

The field is a very interdisciplinary or multidisciplinary or cross-disciplinary

(some people care about the distinctions between those terms) but obviously it

includes people who are interested in Presence phenomenon from all kinds of

different perspectives. If you just look at the academic side of the study of telePresence

I would say that it is a field that is not yet scientific but it is on its way.

We are asking better, deeper questions and not just talking about basic debates

about definition and measurement. We are becoming a more mature field, more

rigorous and more able to answer the questions that people are asking about

Presence.

What brought you to Presence?

When I was doing my Phd at Stanford in the late 80’s and early 90’s my adviser

was Byron Revees. This was before he and Cliff Nass wrote “The media equation”,

the book that sort of brought together both of their areas of research:

Cliff’s responses to computers and Byron’s “Jungle TV”, the idea that we are

hard wired through evolution so that if you are watching a television screen

and a tiger seems to run toward the screen, even though you are well ac-

S t a r l a b

PEACH

44


quainted with the fact that’s just a TV screen and that there’s no wild animal

coming, at some level your body is having some sort of reaction that reflects the

fact that you treat it as real.

As he developed those ideas and we did some initial studies I became more and

more interested. Then the Presence journal started on 1992, and as I struggled

for something to call these responses. First I called them “natural responses”

and then “direct responses” because you’re responding not to the symbol of an

objet on the TV screen, but the object itself. But none of those terms seemed

quite right and then, when I started learning about the Presence literature, that

sort of gelled, and that is how I got into applying that literature to what I was

interested in and doing more research. Then Frank Biocca asked me to write a

short piece about Presence for the journal of Computer Media Communication

and that is what turned into the very long piece that Theresa and I wrote. It was

originally supposed to be short, but as we started looking at the literatures that

be both knew from communication and from other fields, specially psychology,

we realized that there was something even more broad and interesting going on

than just the specific things that we had talked about and so it became a larger

effort.

What keeps you interested?

Why am I still interested? I will tell you. I post a news article every weekday

that illustrates some phenomenon from the news that reflects this idea. It may

be about hospital care, where they have robots going down the hallways of

hospitals, or remote telesurgery, or art projects that involve giving people some

sort of unusual experience that fools them about the nature of technology... I

have posted close to 2000 of these things over the last eight years and I keep

seeing these things and they keep interesting me. I am more and more convinced

that it is a central concept that also has great potential for good or bad,

which makes it interesting.

Presence is a lot of fun for researchers, but where would you see the first applications?

There have been a lot of them if you considered a larger concept than just the

high-end technology that people are still keeping in the scientific labs and

haven’t reached the public. We have had applications since cave paintings because

they were carved on rocks chosen so that they could give depth to the

S t a r l a b

PEACH

45


animals that were appropriate to the way the wanted the viewer to perceive the

animal.

That is as far as back as you can go but even in the 20th century, you can find an

example on that thing about relationships that people feel with television personalities

that is called parasocial relationships or parasocial interaction. That

was originally proposed in 1956 by Horton and Wohl and it rings true today.

People were having Presence experiences with that very low technology. Simulation

rides are maybe the most sophisticated out there but they are set up to

reproduce a specific experience for each person who goes through it, they’re not

as interactive. Video games have now leaped into the realm of the Nintendo

stuff, where you can actually physically move and interact with the images. It is

there already! It is just a question of where is it going.

What would you want to see as an application for yourself?

Cisco recently said they were going to make the high definition conferencing

system they have available via some kind of set up box that you would have at

home and eventually even incorporated into your regular entertainment box so

you could use your high definition TV at home to have a conversation with

people at remote sites. I recently bough a 50 inch plasma high definition one

and I find fascinating how, if you get in the right position and you sort of will

yourself into it, you can feel like they were talking to you in your room. So I can

imagine how compelling it would be to have that system and what a benefit

that would be to people. You could talk to your relatives that live in another city

or just feel like you are in the same room and spend time with them. Even with

just the technology that we have now, if you can make it cheap enough, you

could have lunch with your parents wherever they are.

What are the ethical issues associated with Presence?

There are ethical issues. Take the definition of Presence as “an illusion of nonmediation”.

Some people say, “well that is a bad thing, because you are being

fooled”. Most of the times, you want to be fooled. The ethical problems raise

when you are being fooled without being aware of it. Just one example. Cokie

Roberts, of ABC news, was supposedly doing a live stand up in front of the

White House and then it was revealed she wasn’t actually there but they used a

blue screen. That is no big deal, but imagine if you extend that. You wouldn’t

know what to believe, what is authentic and what is not, and pretty soon, what

S t a r l a b

PEACH

46


is real and what is not to the extend that you don’t have control over that

knowledge. I would say that is the core of the telePresence ethics issues.

Then there are all kinds of things that are just controversial. Imagine you take

the next step and add Haptics, so you are able to interact with an avatar. Then

you could have sex with the virtual character. It is the same thing that they have

even now with child pornography, that is created with a graphic representation,

a virtual child. The courts have to decide: is that pornography? Is that illegal? If

the main issue that people have with child pornography is that the child is being

abused, which certainly would be a reasonable concern, there is no child

being abused here. But if you are concerned with the effect on the consumer of

child pornography, it doesn’t matter wether it is virtual or not. It gets really

dicey and this is another reason I am so interested in the phenomenon and in

the study of it.

What are the technology bottlenecks in Presence today?

We are evolving relatively quickly in the ability to make more convincing reproductions

of reality but there are limits in terms of memory storage and

bandwidth. Second Life, for instance, is a great prototype in a lot of ways but I

have read a lot of complaints from people who say they can’t keep up with the

number of users who are adding to the system. This is just the same problem as

with PC’s: you always have to buy a new one because you don’t have enough

ram or you don’t have enough processing speed.

Those things are always sort of, natural bottlenecks and that hurts the ability of

people to study them because we could do better studies if we had the technologies

that we envision for the future. So, a lot of those limits, are constantly

being like Moore’s law about how fast the computer technology evolves. Every

time we come up against a limit like “oh, we have made the circuits as small as

we could, so now what do we do?” somebody comes up with a new way of

doing it. They shift the paradigm and find a new way to solve the problem. So, I

am fairly convinced that the progress will continue. People just have to keep

pushing and being patient with that break being at the edge. The interesting

thing about Presence experience is that they don’t have to be as perfect reproducing

reality as you would expect. We respond very well to primitive cues

because we are wired that way and because we are willing. Once you accept the

environment as a place, as a thing that you can deal with, then you are engaged

S t a r l a b

PEACH

47


y it. So I am not convinced that the technology has to evolve to continue to

evoke interesting kinds of Presence.

Is there any experiment you would like to do but you haven’t been able to?

There are probably an unlimited number of experiments that I would like to do,

not just one particular thing. The general dissatisfaction goes back to your last

question, which is, by the time you get the technology in your lab so that you

can do the study that you envisioned, there is a new version out that is better.

So, to an extend, I have been frustrated by that problem that we have at MIND

Lab at Temple. It has been like pulling teeth to get the process going so that we

would have the latest technology to do what we would like to. I don’t know if

the readers of this will be familiar with the MIND Lab consortium of labs

around the world but the promise of that model for how to do research tis that

some places have the technology and some other places have certain kinds of

expertise to the extend that they can collaborate. So, even that we may not have

a cave at Temple we can design a study with people who do. This model would

really advance Presence research specifically and just research in general.

Will Presence research change our conception of reality?

I think there is already a huge problem with the term “reality” and what it

means. Often, when you are writing about Presence things, you have to put real

in quotes because people may mean real as “in the existing world” and they

may mean real as “it’s not mediated”. So there is “mediated” and “non mediated”

versus “real” and “unreal”. So, real or unreal in what respect? You have to

define your terms to even have the discussion! If real is corresponding to something

that exists in the physical world that we think we know, then it is very

easy to get confused because we are not experiencing the world directly, we are

experiencing it thorough our senses and so everybody experiences it a little bit

different. Not to mention the fact that all your viewer experience and background

leads you to experience and interpret something in a specific way. We

don’t even agree on what’s real... and then we add these sophisticated technologies

that are designed to confuse you. I think yes, we have a problem.

Has research in Presence changed the way you see the world?

I think it has changed more my view of my world as it is evolving, because I

have been thinking more about the technologies and the way they are develop-

S t a r l a b

PEACH

48


ing but I don’t think I see the world substantially differently. Even when I was

not focusing specifically on Presence, my background is in communication so I

am interested in thinking about the perceptions that people have of each other

and how they get confused. This project about telePresence portrayals, combining

my interest in those novels, tv shows and movies that portray this kind of

experiences and my interest in the experience have definitely made me more

concerned about how the world will look in 10, 20, 30 years ... There are great

possibilities but also worries, because knowing how human nature as I do,

there is great potential for good but also great potential for bad.

Could you summarize your work in just one question you are trying to answer?

What is the role of media (defined as technology) in altering, enhancing or lessening

human experience and human communication? !

S t a r l a b

PEACH

49


S t a r l a b

PEACH

50


RALPH SCHROEDER

“Do avatars behave differently from how we are

relating here face to face?”

çI am a research fellow at the Oxford Internet Institute and I have been working on

Virtual Reality since the early 1990’s. Before that I did an undergraduate degree in

Philosophy and History and a PhD in Sociology at the London School of Economics. I

have written extensively about virtual reality and I am now synthesizing this research

in a book that will link how people interact in virtual environments to other types of

computer-mediated communication and new media.

What are you currently working on?

At the moment I’m mainly working on eResearch or eScience: how scientists

collaborate at a distance. I am also still working on multi-user virtual environments

and social virtual environments, and how people interact with each other

in those.

What is your definition of Presence?

Presence that phenomenon of having the sensation of being in another place or

space, being able to interact with it. That means that Co-Presence is being at

another place and being able to interact with the other person. I have a rather

strict definition because I think that the experience of being there must be a sensory

one. So it must involve one of the five senses, primarily vision and auditory

interaction.

Presence is a wide field. Is there a need for focus?

There is a need for a focus, maybe in technology development and maybe in

study. But my particular area has been multi-user environments, social environments,

and there there is very little work so far, so it is not even possible to

say what the focus is at the moment.

S t a r l a b

PEACH

51


Is Presence a science or is it a technology?

The multi-user environments and the systems that you use to have a sense of

being in another place, those are technology. The study of how we experience

that and how we interact within the environments, that is mostly a science. In

my particular area, a social science.

Where would you situate the discipline of Presence?

I think it is not possible to say for Presence as such. You have the technology on

one side and then there is another side which is how we understand, how people

interact. And that, in terms of science, composes of a broad range of disciplines

such as psychology and sociology.

What brought you to Presence?

I read a book by Harold Reingold which came out roughly in 1991, and which

was the first kind of a journalist account of Virtual Reality. At that stage, Virtual

Reality was like a big new thing, there was a lot of hype about it. I was living in

London at the time and I went to a talk that he gave in a bookshop and I went

out to him afterwards and I said: “Is there anyone studying this yet from a social

perspective?” And he said: “No, not as far as I know”. And so I said: “This

is interesting. I will do that”.

What keeps you interested?

The first few years I studied Virtual Reality it was still mainly single user systems.

Since then, I’ve moved more and more into multi-user systems, an absolutely

fascinating area not so many people is studying. It is something that you

can study endlessly: How do avatars interact? How do they behave in relation

to one another? Do they behave differently from how we are relating here face

to face? How do they interact in small groups, in large groups?

The other thing relates to what I think is one of the main problems in this area.

Video-conferencing has been around since the 1960’s, and yet even nowadays

not so many people use it at all. Why is this? In about the year 2000 we showed

that if you have two cave type systems (I was working in Sweden and another

group of Mel Slater was working in London) and you connect these two cave

systems and they try to do a Rubik’s cube puzzle together, they could do that

just as well as they could do it just on a table passing physical cubes to each

S t a r l a b

PEACH

52


other. I think that is a very important finding because it essentially means it is

not always as good to do things face to face as it is maybe in some kinds of Virtual

Reality systems. We are still trying to figure out: what can you do in virtual

environments or in video- conferencing systems that you can do just as well as

face to face? The implications of that are very important over the long term.

Imagine if we don’t have to fly to different places anymore, if we don’t have to

work together in the same place. This is a very important question and it is very

curious that it hasn’t been focused on and studied more.

Presence is a lot of fun for researchers, but where would you see the first applications?

One of the things that we have talked about is video- conferencing and distributed

work. Another area is large scale social spaces online like Second Life.

Many people have heard of Second Life and it has thousands of simultaneous

users so, if you think it is a virtual environment, then it is the most widely used.

The interesting thing about this is that although Second Life has caught a lot of

attention recently, these systems have been there since the mid 1990’s! So the

question is: why did this only recently take off with Second life? You might

have some answers like more people have broad band internet, more people

have been getting interested into these systems, the graphics cards and the

sound have got better... But still we have a very limited understanding of why

people use these things, how they use them, and what kind of possibilities do

they have. A very simple question that is almost nothing written about it yet is:

what is the difference between how people interact with each other depending

whether they have text or voice? So there are lots of possibilities for research.

What would you want to see as an application for yourself?

I would quite like to see a very nice fully functioning video-conferencing or a

virtual environment system for being able to work together. I would also like to

see online social spaces like Second Life or massively multi-player games. In

those types of environments, you have a lot of very nice possibilities for experimenting

with how people can interact with each other. What can they really

do with each other apart from chasing after monsters or collecting points? One

of the nice things that you can do in this large on scale online virtual spaces, is

that you can build together: you can build houses and cities, you can experi-

S t a r l a b

PEACH

53


ment with different ways of forming societies creating the rules by which you

can live.... All of that is very exciting and has a lot of different potential.

What are the ethical issues associated with Presence?

This is a very large topic, so we could sit here for a few hours! There are a lot of

issues for single- user systems, but I am not so interested in those. I think the

issues become much more complicated and much more interesting when you

have multi-user systems. There you have a range of questions that really revolve

around how people interact with each other and how they see each other.

For example, if I am participating in a multi-user environment and I have a particular

avatar with certain types of clothes that I might have made myself, do I

have ownership of that representation? Or do the people who have design and

own the system have control over it? Or imagine that I have been spending a lot

of time in a multi-ser virtual environment where I have build up some property

and, all of a sudden, the people who own the system have designed another

system which follows on from it. Can I take my personal appearance with me?

Should I have the right to do that? Do I take my property with me? These sound

like very abstract questions but you can relate them to a whole set of technologies

that have been developed nowadays. For example social networking sites

as Facebook or Myspace,. If I have created a certain profile, a certain appearance,

a certain set of relationships with people in those kinds of technologies,

what rights do I have? How should I present myself over the various mediated

systems in which I am represented? Should I have a consistent identity? Should

people be able to use what I portrait myself with and modify it? These are the

kind of questions that are becoming important as we spend more time presenting

ourselves to other online entities in a particular way, as we have an online

persona. What is our relationship with each other in the light of how we present

ourselves?

What are the technology bottlenecks in Presence today?

It is interesting because, from a technological point of view, we can do most of

the things that we want to do. We can have very realistic virtual environments;

we can present ourselves in the environment around us in a very realistic way

both in a computer generated virtual environment or in a video-conferencing

environment with 3D technologies. The main bottlenecks are actually not the

technological ones, but the social ones, about how we use them. How can we

S t a r l a b

PEACH

54


most effectively use these systems? What kind of virtual environments are

really suitable for different purposes under different circumstances? When is it

best to use just a desktop system as opposed to an immersive system? What are

the conditions for using each of those different types of systems under different

circumstances? It is not the technology, is how we use it that poses the most

important questions to my mind.

Is there any experiment you would like to do but you haven’t been able to?

I have been able to do many of the things that I wanted to do because, in my

previous job, we had a cave at our place of work. The main problem is that you

don’t have enough people power to do the experiments. There are a million

different studies that I would like to do, especially in multi-user worlds where

you vary the size of the group. As you increase the number of persons, how

does that influence the way they can really effectively interact? My sense is that

you can’t really interact with too many people online but that has not really

been studied so far.

Will Presence research change our conception of reality?

I don’t think so. What is happening is that more and more of us are spending

more and more of our everyday lives in mediated communication: on the mobile

phone, looking at the screen, perhaps in an online space... We tend to see

Virtual Reality as being this new thing that gives us a new view of reality but

we are being mediated more and more throughout the course of the day. I think

we need to understand that these are all forms in which we communicate with

one another. And they are quite different but there are also some similarities

between them. So, for example, I would suggest that even using things like mobile

phones and even things like writing e-mails or getting information from the

web can be compared in some way to using virtual environments. So we need

to understand much better how all these different mediated environments work

and not worry so much about these larger questions about reality.

Has your research in Presence changed the way you see the world?

I would say not so much how I see myself, but I certainly think that I have been

doing a lot more thinking about how we spend our days in online spaces. It is

interesting if, for example, I watch my kids: they are thirteen and fifteen now

and they have a very different understanding of the online world from what I

S t a r l a b

PEACH

55


do. They are fascinated in a different way, but they are also not so curious about

other parts of virtual environments that I am curious about, so I think that is

quite interesting.

Could you summarize your work in just one question you are trying to answer?

How do people interact with each other in multi-user virtual environments?

S t a r l a b

PEACH

56


S t a r l a b

PEACH

57


PAUL VERSCHURE

“What allows a physical structure like the brain to

construct a reality? That is amazing, a miracle!”

I am the director of Universitat Pompeu Fabra’s Institute for Audiovisual Studies. I am

an ICREA research professor in Barcelona, where I have moved my research group -the

Laboratory for Synthetic Perceptive in Motive and Cognitive Systems- back from Zurich.

My scientific aim is to find a unified theory of mind, brain and body through the

use of synthetic methods and to apply such a theory to the development of novel cognitive

technologies.

What are you currently working on?

Our work in divides itself along three dimensions. On the one hand there is a

strong focus on how does the brain work and how does it construct reality,

where we mainly use computational neuroscience methods to understand how

the physiology and the anatomy of the brain gives rise to psychological phenomena.

So there we have a bunch of people mainly engaged with computer

simulations of different anatomical structures with their physiological properties.

The second dimension translates this work towards the real word. The brain

does not exist in isolation but it is part of your body and your body interacts

with the world, so we have to give our models the same capabilities. For that,

we give these computational models an artificial body, artificial sensors and

effectors, such as that means that you interface them to a robot. So now we have

an artificial behaving system. But the sense of robots can be very broad: you

could think about your standard robot that looks like a humanoid robot or a

wheeled robot but also these whole room here could be considered as a robot.

That is one of the links towards Presence research. We are saying: look, the

technology we develop to induce Presence is very much overlapping to large

S t a r l a b

PEACH

58


extend in terms of their control capabilities with how we look at other robotic

systems.

Then the third dimension translates these technologies back to the brain. We

say: OK, here we are trying to understand the brain, we do this by building

artificial brains, but long we have testing our theories to see if we can use it to

repair the brain. So there we use different technologies for rehabilitation and

neuroprostheses, to see how can we fix a brain that is broken. For instance we

are working right now in the clinic Esperanza here in the Barcelona with stroke

patients. We try to see how can we use interactive media as a training environment

to speed up their recovery. This brings us then back again to the brain,

and now in a way of using our technology to test our theories about how brains

work. !

What is your definition of Presence?

Presence is for me a field of psychology. I define it: Presence is a state of being

that we just have, and that as such, is not open for direct measurement, so we

don’t have to care about it. However, what we can think about is that the world

reveals itself through physical sources: I see you because light reflects off your

skin and your clothing and that impinges in my retina. We can now try to replace

these sources, because we have technological capabilities: I can replace

you by an avatar, so now the physical source is replaced by a virtual source that

is approximating the sense of stimulation that gives rise to my understanding

that I’m seeing you. To me, the whole game replaying is to understand: to what

extend and what conditions can we replace physical sources by virtual sources

to still induce the same perceptual, cognitive and emotional experience that we

have under real world conditions. And that is very much a psychological question.

Because in Psychology the whole question would be: what are the key

features that will induce a certain percept? To recognize you, I could have just a

black and white sketch with an outline, and it still would work. That means that

is just some extraction over the information from all this set of physical sources

that now turns you as a physical source into a percept. And that is something

we can investigate very explicitly. And whether we do that in the context of

what we now call Presence or in the context of experimental Psychology, for me

that is all the same.

Presence is a wide field. Is there a need for focus?

S t a r l a b

PEACH

59


Absolutely. I think the field of Presence research is right now in a very protoscientific

stage. It is even before proto- scientific stage, because a proto- scientific

stage would be, for instance, when Biology was started by people like Linnaeus

in the 16th century. They built taxonomies, they described that these are

all possible plants and animals that exist in the world, so they tried to make a

systematic overview at least. Even that, we haven’t started to do yet in this

field! I think the big problem is that people have not paid sufficient attention to

the history of other disciplines. People are repeating very much the history of

Psychology when posing this question ofPresence is the sense of being there”.

That is like 19th century Psychology, where people were very much driven by

introspection, like “what does it mean to perceive?”. But it is a very naïve view

and it is not a very scientific because you are not measuring. I think we have to

stop to consider this field as something special isolated in the corner. We have a

lot to learn from other disciplines and we have a lot we can bring to bear on that

issue. And, in addition, we should not be naïve about the proximity of this

whole Presence debate, to the whole questions around consciousness and

awareness. It is a huge debate, very philosophical, and there is not much we can

do in that area right now from a scientific perspective. So I think we should be

very clear about our limitations, we should be willing to learn of the history

from other fields and really phrase the question out in more scientific terms. It is

an absolute requirement if we want to survive in long term. That is really what

we are doing in our own environment and also in, for instance, the PRESENC-

CIA project of which I am a member.

Is Presence a science or is it a technology?

It constitutes both. In the one hand, it is a new psychological paradigm we can

explore. I can work with complex avatars and I can have them interacting with

human so I can perform new kinds of psychological experiments in a very controlled

way. That makes it part of a science. As such, a can pose deeper questions

about how experience is constructed and manipulated. On the other hand

I need technology to do this.. The big challenge is how do we bring these different

components together.

Where would you situate the discipline of Presence?

It is a confluence of Psychology, Neuroscience and different technological domains.

But I am very much developing this in my own institute, in my own

S t a r l a b

PEACH

60


esearch group, as a cognitive neuroscience exercise. As just new paradigms to

understand how humans work.

What brought you to Presence?

We did a large interactive exhibition in Switzerland in 2002, where we turned a

hundred sixty square meters space into a robot, just like an artificial organism

that visitors could encounter. We built a cave environment in which you could

have dozens of people interacting simultaneously in a virtual environment.

Then Doron Friedman saw the relevance of this to future developments in Presence

research and we got linked up in his PRESENCCIA project. We are now

working very closely with that consortium to see how can we explore this kind

of technology and concepts to build a persistent mixed reality community. That

is, a community where physical people, remotely present people, and virtual

characters can co-exist.

What keeps you interested?

There are still many problems to solve. This evening we show a mixed reality

performance called re(PER)curso which opens new doors in expression because

there we are trying to understand how can we use mixed reality technologies to

enhance human expression and creation. That is almost an open ended domain.

On the other hand, I really see that we have just opened the door into a whole

new set of psychological experimentation where we hardly scratched the surface.

There is so much to do and also it helps us to directly access a deeper understanding

of the brain: as I said, we are using virtual and mixed reality technology

to try to repair the brain. That means it is not only from this fundamental

perspective that you can use the approach to understand how the brain

works, but you can also see how this can have an impact in other people’s quality

of live. There is a huge promise for this kind of technology so every day is

still very exciting.

Presence is a lot of fun for researchers, but where would you see the first applications?

For me the first applications are in the area of neuro-rehabilitation, training

quality of life, that is where we are very much bringing this technology towards.

Then we are working with a number of industrial partners on applications

in media, interactive content, synthetic content. I think there are important

S t a r l a b

PEACH

61


applications in that area but they are are a little bit more driven by commercial

interest, so in that sense, they are a bit less exciting and not so romantic.

What would you want to see as an application for yourself?

Well, I would like to see an environment like this room that knows me and tries

to please me, entertain me, challenge me with whatever stimulation that can

generate. So, I could walk in the room and it tries to cheer me up. Says: look,

why don’t’ we have an exciting movie? But it is not going to go on a database...

that is boring: it will generate a movie de novo on the spot, depending on my

current state, my reactions. Whatever I say it will just improvise. That is a technology

we are to develop and we already have this in the domain of music

composition. It is an interactive computer based composition system called Ro-

Boser that is capable of generating from scratch, in real time, complete musical

compositions that can be completely driven by an interpretation of my current

state. RoBoser is believable because we won a price with it in the Swiss competition

for production movies, where the jury did not notice it was computer

music. The prices went to some guy from the production company who was

responsible for sound, but the music came directly from the machine! What it is

important about this is that is not only about having a belief about the future.

Everything is easy for the person who does not have to do it! The big challenge

is to do it: put your hands down, get dirty and get stuff done.

What are the ethical issues associated with Presence?

I don’t see any special ethical issues that we don’t know about already in other

fields. The lawyers and the philosophers can have fun with this and will keep

them off the streets for a long time but I don’t think it is very interesting. It is

not a grounded domain.

What are the technology bottlenecks in Presence today?

Integration is our biggest problem. To deliver Presence we want to replace all

the physical sources of stimulation with virtual sources so this must be of different

modalities, of different fidelities ,different coupling to the user. In the end

you have a huge integration problem: how do I get all my input and output

devices together in a coordinated way that I can create a coherent stimulation

for the user? There are no technical solutions we can just get off the shelves to

do that! We take a neuromorphic approach where we use a system called IQR

S t a r l a b

PEACH

62


we have developing over the last twelve years, which allows us to build large

scale real time neuronal control systems. We are using our limited understanding

of how brains process and control real world stimulation and output as a

starting point to try to understand this integration issue. I think this is really

where the biggest challenges are.

Second challenges are the level of learning, adaptation, perception reality is

constructed. We learn about reality and also our controlled structures have to

have these capabilities. I can have a very realistic high fidelity humanoid avatar

but, as soon it starts to behave, it fails and then the dream is gone and you

really feel cheated. So how do we give these kinds of systems control systems

that still make them believable? That is a huge problem, too.

Is there any experiment you would like to do but you haven’t been able to?

I would like to perform an experiment that proves that our Virtual Reality system

has a positive impact on recovery of stroke patients. It is an experiment we

are doing right now but it is a hard work: we looked almost at hundred patients

that we have to follow over a half a year or more.

Then I would like to be able to reliably use all aspects of the mixed reality space

that we have to define interactive paradigms for psychological experimentation.

To say: look, I want an avatar. I want the avatar to start a negotiation game with

a human user; I want the avatar to violate the expectation of this user at some

point in time... It would be very difficult to control the logic of this, so it is

something we really have to try to develop and that is still a little bit in the future.

Will Presence research change our conception of reality?

No, because if Presence research makes sense we should follow this conception,

we shouldn’t violate it. However, it can deepen in the understanding of how we

construct reality. But I don’t see it as such revolutionary, because Presence research

as such is not a political program that says: “and this is the radical new

way in which we think brains construct reality!” That view is not there. We use

Presence technology to test our own hypothesis on this.

Has research in Presence changed the way you see the world?

S t a r l a b

PEACH

63


No. But it doesn’t have to! Presence research has enriched our activities because

it has shown to us new ways to perform experiments. We want to measure, to

quantify, and Presence research has opened new doors to do that. An excellent

example of this is the famous Milgram experiment we replicated with Mel

Slater. It is an absolutely new way to look at it which is really wonderful. And

this is just beginning!

Could you summarize your work in just one question you are trying to answer?

What allows a physical structure like the brain to construct a reality? That is

amazing, a miracle!!

S t a r l a b

PEACH

64


GIULIO RUFFINI

“Reality is not what we believe it is”

I am the coordinator of Peach, the coordination action of the European Commission

focusing on Presence research and supporting Presence research, as well as the director

of Starlab, a company transforming science into technologies with a profound and positive

impact on society. Besides my work in PEACH community, I am currently involved

in various Space and Neuroscience research projects, including the recently

completed Sensation (advanced sensor development for attention, stress vigilance, and

sleep/wakefulness monitoring) and HIVE (Hyper Interaction Viability Experiments).

The latter targets the development of technology for non-invasive brain stimulation,

with applications in research, therapy and direct computer to brain communication

(which I call hyper-interaction).

What are you currently working on?

The most relevant part of my work in Presence is supporting the community in

defining where they want to go and providing the means to talk to each other.

Presence is an interdisciplinary field, and that gives it a lot of richness but it is

also a source of problems because there are many fields that don’t connect to

each other or don’t talk to each other sufficiently well. I also support the European

Commission and the community in the Road Map, defining the visions

where we do want to go and ensuring that they can have a great impact on society.

At Starlab we do also some research on neuroscience, mostly about physiological

recording of data on the go and extracting meaningful information about it,

but it is a little bit tangential to Presence at the moment. We are also entering

the field of brain stimulation, a very exciting field which is the complementary

to sensing and needed for bi-directional interaction.

What is your definition of Presence?

S t a r l a b

PEACH

65


It is very important to distinguish between what Presence is as a phenomenon

and as a field of research. As a phenomenon, Presence restarts with something

that physiologists call “qualia”, which is the experience of being there, being

somewhere, and which is a very profound aspect of human experience. Most

people who are alive are present in their mind, so I think everybody can relate

to that. Here all aspects of Presence enter: the environment, the own body and

the interrelationship between them.

As a field of research Presence is much more restricted, because you have to

define it in terms of something that you can scientifically quantify: you have to

set a measurable success criteria of Presence. That criteria would be to show the

same kind of behavior and physiological responses to a virtual environment as

you would do in a real situation. This is more restrictive than the “qualia” but

it’s something that we can work with.

Presence is a wide field. Is there a need for focus?

Within the coordination action of Peach we have observed that there is quite a

dispersion in what the field is about. This has to do with the term “Presence”,

which is something that everybody can give its own interpretation because it is

closely related to the human experience of existence. There are many forces

pulling in different directions associated to the multidisciplinarity involved, so

it will take long for us to come to some consensus in the definition of Presence.

Our approach is trying to define a clear goal instead: to develop technologies

and scientific understanding to truly achieve successful replacement, to be able

to put somebody in a virtual environment and making him feel like he is there,

perhaps having a different body. The strategy is to try to gather researchers

around a grand challenge project, something like an Apollo or Manhattan project

for the field, which is challenging, long term and it is at the core of the field,

and which by its own complication will collect all the community around pushing

in the same direction.

Is Presence a science or is it a technology?

Presence is first and foremost a science. It has to be seen as a science. But Presence

has also a technological side that is moving faster than the science side: it

is a fact that we can achieve Presence in virtual environments today although

we don’t really understand how that is happening.

S t a r l a b

PEACH

66


Where would you situate the discipline of Presence?

Presence is part of trying to understand how the human brain constructs models

of reality itself and in that sense it falls squarely within cognitive science.

What brought you to Presence?

There are three things that I can say about this. One is an experience I had in

Epcot Center. There was some kind of attraction in which you got into a cart

with a wide screen with a wide field of view, and you had a chair which, unbeknownst

to you, would tilt back while you were observing the cockpit of an

aircraft taking off. In that moment I experienced very clearly the feeling of being

in that airplane in a very powerful way.

The second aspect is that being a physicist I am very interested in how physical

sciences have evolved pulling us further and further away from our experience

of the physical environment. We have principles like the relativity theory, in

which the concept of simultaneity and time and space are totally changed from

what we have experienced; there is quantum mechanics, in which you substitute

an objective state for a physical environment around you with a probability

amplitude function... All this points in to the direction: reality is not what we

believe, it is something else.

This brings me to the last point which is information theory. Physics theories

are based in extracting information from the environment, and that is exactly

what our senses do. I think the concept of Kolmogorov or algorithmic complexity,

which defines the limit of information compressibility, is closely related to

how brains model reality and how physicists construct theories.

What keeps you interested?

First, today my task as Peach coordinator: to try to make the best of this opportunity

to promote this field. I believe Presence will become more and more important

as time goes by as people realize how important is to understand this

manipulation of the subjective feeling of being somewhere, with somebody,

with some body. From the conceptual point of view, I’m very interested in this

aspect of model making and the construction of models of reality in itself that

cognitive systems do through the extraction of information. From a physical

and philosophical point of view, I’m interested because it relates very much to

what reality is, or, rather, what is not.

S t a r l a b

PEACH

67


Presence is a lot of fun for researchers, but where would you see the first applications?

There are applications already today having to do with training, engineers being

able to navigate through an aircraft and seeing things in mixed reality.

There are also applications in the medical training and day to day activities.

There is a very important aspect in telePresence, as I think Presence may provide

very important means to reduce the impact of travel and have a very positive

effect which regards to climate change. There is, of course, entertainment:

games, movies, etc. And space travel may also benefit from Presence research:

when you have somebody traveling in a little tin can for six months going to

Mars, they may benefit pretty much, from Presence technologies to spend some

hours in Hawaii instead.

What would you want to see as an application for yourself?

It’s a boring answer but for my day to day work, having efficient means for

telePresence and for interacting with people in virtual environments would

save time, fuel and carbon emissions, so it would be a very desirable thing to

do.

What are the ethical issues associated with Presence?

One could turn the question around: what are the ethical issues of not doing

Presence? Going back to the climate change issue, I think it is a moral obligation

to develop Presence technologies today. Of course, there are ethical issues that

one can ponder about Presence, and the fact that you can be in many places,

assume any forms or have multiple selves. It is a very complex field also from

the legal point of view. If you are doing something in a virtual environment that

breaks the law somehow in the real world, how is that to be handled? These are

all complex issues but my personal worry is more on the moral obligations that

we may have to do Presence research. And I would not like Presence research to

be used for military use, but of course like any other technology, that will happen.

It is already happening.

What are the technology bottlenecks in Presence today?

There are many. Technologies like head mounted displays have to evolve to a

degree of being ready useable by may people with a low cost. That is a very

important challenge and once this challenge is remounted it will open wide

S t a r l a b

PEACH

68


markets in a lot of applications. There are also very strong challenges regarding

Haptics, a key and very challenging field.

But the greatest bottleneck right now that is perhaps not being sufficiently addressed

may be the machine intelligence side of things. One thing is to have a

subject with the right interfaces, which is a technological challenge itself: the

right effectors, the right audio, the right video... But the other side of the coin is

that you must have some machine cognition, some artificial intelligence, that is

capable of understanding the state of the subject, presenting the right stimuli

and reacting in the right way to make a credible landscape, or a credible social

interactive situation. This is a huge bottleneck today.

Is there any experiment you would like to do but you haven’t been able to?

I do have some ideas, but they are not as concrete as being experimental set ups

yet. I believe that complexity plays a role so, going back to this modeling issue,

I think that if you can do experiments in which you provide stimuli to a subject,

and these stimuli are consistent with a simple model of what is going on, then

you will have a much higher success in achieving the Presence experience. Take

for example the Pinocchio nose experiment, in which somebody is touching

your nose while your finger is touching a nose further away from you. If you

can realize experimental set ups in which there are simpler explanations of

what is happening that are available to the subject, the Presence experience will

become bigger. There’s some experimental work to to be done there.

Will Presence research change our conception of reality?

I think so. This is more of an intuition than a clear concept but I think that although

we already are in the digital age we have not entered quite as much as

we will. And as soon as we start interacting with virtual and real universe in

very similar ways we will realize that reality is not what we think it is.

Has research in Presence changed the way you see the world?

Not beyond what I just said. I think we are in this transition phase in which

human kind is mastering the bit and this will have profound impacts in what

we think reality is in a very internalized way.

Could you summarize your work in just one question you are trying to answer?

S t a r l a b

PEACH

69


What are the principles that would allow us to understand how cognitive systems

such as the brain -or artificial cognitive systems one day- model reality

and self?

S t a r l a b

PEACH

70


S t a r l a b

PEACH

71


PHIL TURNER

Presence does not need focus. It needs breadth.”

I am a senior lecturer in Napier University, Scotland, where I teach Human Computer

Interaction. My background is a mixture of psychology aspects and a bit of computing. I

am currently working in the theoretical basis of Presence from the perspective of the

intentional arch.

What are you currently working on?

I have been looking at the issue of listening, which is sort of active hearing, and

the interpretation of that; Also on the theoretical bases of Presence from the

perspective of the intentional arch.

What is your definition of Presence?

I don’t have one. “Present where?” is a more interesting question for me. I think

it is a more concrete question we should be asking on that, so I don’t really have

an answer for that question.

Presence is a wide field. Is there a need for focus?

No. There is a need for breadth. I have listened to six papers this morning and,

with one exception, they were quite narrowed. There were hypothesis testing,

taking one factor, examining where that factor has an effect on… I came with

thinking “well, these were very clever but I am not really none the wiser”. I

think it is a naturally rich, crossed and even multidisciplinary field, and to treat

it as technology meets psychology it’s just too narrow.

Is Presence a science or is it a technology?

When we say science there are two answers for that. I mean what sort of science?

It is a social science? Or is it a natural science?

Where would you situate the discipline of Presence?

S t a r l a b

PEACH

72


It really depends on whether we are treating it as a natural science or social

science. It has a place on social sciences because we teach Social Presence as

part of a Computer Support Recovery Working program, which is a conjunction

of social psychology, sociology and technology. So it really it is one of those

things which overlaps with so many different domains and it depends where

you are coming from. I see lots of people coming from Computer Science and

increasingly from the biological approaches; I guess it is a conjunction of all this

things.

What brought you to Presence?

I was playing Doom. Remember Doom? 3D, shooting the monsters… I was

looking for a topic to do my PhD and I found it interesting that people were so

involved in a computer game that that they would swerve when the monsters

shoot them.

What keeps you interested?

I thinks there’s a couple of interesting gaps in Presence related with the qualitative

and the conceptual, which are two areas I’m interested in. It is a very rich

area to work in because we can bring a wide variety of the disciplines, and I just

enjoy working with breadth rather than depth.

Presence is a lot of fun for researchers, but where would you see the first applications?

This is a problem. I read in a magazine that Virtual Reality would never work

because it requires the businessman to get into a suit and putt on a helmet. “Yes,

fair point”, I thought. I can’t see the major applications for Virtual Reality strict

Presence research until we solve these problems

What would you want to see as an application for yourself?

For myself… I have got a long term interest in space exploration, and when we

finally get to Mars I want to put on a Virtual Reality suit and walk on Mars

along with the astronauts.

What are the ethical issues associated with Presence?

S t a r l a b

PEACH

73


I have done some work with memory, and the idea of seamlessly manipulating

people’s memories, people’s histories, people’s past is absolutely terrifying, I

think there’s some real issues there.

What are the technology bottlenecks in Presence today?

“The illusion of non mediation”, that’s the biggest technological problem. You

will always have to wear some technology that you are going to be aware of.

Until that is invisible you will always going to be brought back to earth.

Is there any experiment you would like to do but you haven’t been able to?

I am not really an experimentalist. So, no, thank you.

Will Presence research change our conception of reality?

I had some interesting conversation with someone about whether we could

induce LSD like experiences with people, and that is quite interesting to me. So

we are sort of reinventing the 1960’s all over again. We are doing psychophysics

in Presence research; we are doing very basic visual perception stuff… So, why

not? Let’s try and do LSD using electronic media instead!

Has research in Presence changed the way you see the world?

Not remotely.

Could you summarize your work in just one question you are trying to answer?

How do we use philosophy to allow us to think very clearly about these very

everyday ideas, like being in a room or being in a certain place?

S t a r l a b

PEACH

74


Summary Table

Who Definition Focus? Research

David

Benyon

Olaf

Blanke

Presence is

being connected,

with a

person or a

place

Where is my

body?

No. Include Artificial companions.

classic telepresence

and great experiences?

Social side-effects. How can we design for

social

Yes, eventually Why is my experience body-centric? How

is this generated, and by which brain areas?

How far can we take the feeling of selfhood

away from our bodies and into an

avatar?

Another is to use VR to generate out-ofbody

experiences.

More long term would be to relate these

induced states to neuro-imaging. Which are

are the areas related to the Presence qualia?

Wijnand

IJsselsteij

n

María

Victoria

Sánchez

Vives

Presence is the Yes, there is a

sense of “being great need for

there” in a mediated

envinitions.

clarity of defironment

Presence is Yes, on VR

subjective, but related work.

acting as in Multidisciplinary,

though.

reality indicates

presence

Experiments with full-body interfaces to

extend our “rubber hand” experimental

scenarios.

How do people know the difference between

a mediated environment and a real

environment, and how can we know where

to look for the right cues to make the experiences

more realistic?

Multi-sensory correlations and body representations

from neuroscience point of view.

Use VR in neuroscience research

Body representation is a very good to use

VR to study how our body image is generated.

Full-body representation and manipulation

is the next step.

Looking for the neural correlates/signature

for Presence will be very close to those of

consciousness.

How does the brain generate unity in perception,

integrating sensory inputs and

previous experiences, to deliver a single

conscious experience?

S t a r l a b

PEACH

75


Who Definition Focus? Research

Martin

Buss

The sense of

being there,

where there can on “being

be a real environment

(telepresence)

or VR

Mel Slater If you respond

as if it were real

in a digitally

generated environment,

it is

presence

There is already

a focus

there”

Yes, needed.

telepresence, haptics, multi-modal interaction,

neuro-prosthesis

Very time consuming research, in relation

to data analysis.

Lack of standards.

Why do people respond in VR as if it were

real when they are fully aware it is not?

Matthew

Lombard

The illusion of Need to use Where is it going?

non-mediation. clear definitions,

What is the role of media (defined as tech-

Broader than

the idennology)

in altering, enhancing or lessening

VR.

tify the core

and focus.

human experience and human communication?

What is at the core of Presence?

Ralph Co-Presence, is

Schroeder being there

together and

being able to

interact

Yes, in technical

aspects.

Not clear on

social focus

yet.

Scientific collaboration at a distance. Social

virtual environments.

Social implications of multi-user systems.

Social interaction in VEs.

Paul Presence is a Yes, badly

Verschure qualia, a state needed.

of being we

cannot measure

as such.

What allows a physical structure like the

brain to construct reality?

Using VR in neuroscience.

How is the body involved? Can we repair

the brain?

To what extent, under what circumstances,

can we replace physical source by virtual

sources and induce the same perceptual,

cognitive and emotional experience? Psychology

question: what are the features that

induce a certain percept?

Using VR in neuro-rehabilitation

S t a r l a b

PEACH

76


Who Definition Focus? Research

Giulio

Ruffini

Phil

Turner

As a phenomenon

Yes. There are Can you do experiments to show that if

is a qualia, problems due you provide simple stimuli, consistent with

the experience to terminology a simple model, they lead to high presence?

of being, being and interdisciplinarity.

somewhere.

Where are you,

What are the principles by which cognitive

systems (human, or artificial ones) model

what your body

and construct reality and self?

is. As a measurable

phenomenon,

it is

more restricted.

Acting as in

real life, is a

measurable

criterion. As a

field, it tries to

predict when

this happens,

and how the

brain constructs

reality.

None No. Multidisciplinary.

How can we use philosophy to think about

presence?

Social Science

and Natural

Science.

S t a r l a b

PEACH

77

More magazines by this user
Similar magazines