CHI LaTeX Extended Abstracts Template - Free

jfleureau.free.fr
  • No tags were found...

CHI LaTeX Extended Abstracts Template - Free

Affective Benchmarcking of Movies

Based on the Physiological

Responses of a Real Audience

Julien Fleureau

Technicolor

Cesson-Sévigné

julien.fleureau@technicolor.com

Philippe Guillotel

Technicolor

Cesson-Sévigné

philippe.guillotel@technicolor.com

Izabela Orlac

Technicolor

Cesson-Sévigné

izabela.orlac@technicolor.com

Abstract

We propose here an objective study of the emotional

impact of a movie on an audience. An affective

benchmarking solution is introduced making use of a

low-intrusive measurement of the well-known

ElectroDermal Answer. A dedicated processing of this

biosignal produces a time-variant and normalized affective

signal related to the significant excitation variations of the

audience. Besides the new methodology, the originality of

this paper stems from the fact that this framework has

been tested on a real audience during regular cinema

shows and a film festival, for five different movies and a

total of 128 audience members.

Author Keywords

Affective computing; ElectroDermal Answer; Signal

Processing; User Experience

ACM Classification Keywords

H.5.1. [Information Interfaces and Presentation]:

Evaluation/methodology.

Copyright is held by the author/owner(s).

CHI’13, April 27 – May 2, 2013, Paris, France.

ACM 978-1-XXXX-XXXX-X/XX/XX.

Introduction

Getting the affective state of an audience watching a

movie may have many potential applications in a video

content creation and distribution context. A movie

director may be interested by such information to validate


his artistic choices and/or adapt the editing to optimize

the emotional flow. It may also help a content distributor

select the best target countries or population.

Nevertheless, obtaining the real-time emotional state of

an audience in an objective and low-intrusive manner is

not easy. One direct method would be to collect, as it is

already done by some studios, a direct self-assessment of

each viewer, minute per minute. This approach is clearly

too intrusive, quite subjective and may bias the reporting

by distracting the subject from the movie. Face analysis

through an adapted camera could be an alternative, but

the associated algorithms [10] may be very sensitive to the

user environment (low lighting conditions) and not always

adapted to natural “close-to-neutral” facial expressions.

Another approach, adopted in this paper, is based on the

recording of physiological signals. Many biological signals

are known to vary with the emotional state of a subject.

Some typical signals are the ElectroDermal Answer

(EDA), the Heart Rate Variability (HRV) or the facial

surface ElectroMyoGram (EMG). The correlation between

such signals and the affective state is well-known in the

literature [8, 2] and the devices to record such biodata are

becoming more and more compact and non-obtrusive

[9, 4].

Furthermore, if evaluating the interest of such signals in

the context of movie viewing is not new [7], more and

more recent works try to correlate more accurately the

user’s emotion with the video content. For example, in

[11], physiological signals (including EDA) are correlated

with audio features extracted from the video stimulus. In

[13], the authors developed a user-independent emotion

recognition method to recover affective tags for videos

using electroencephalogram (EEG), pupillary response and

gaze distance. In [3], the EDA signal is correlated with the

arousal axis in a context of affective film content analysis

with a special focus made on the narrative coherency. In

[5], a real-time approach of the emotional state detection

is alternatively presented where the authors especially

focus on affective events (i.e. fast increase of the

emotional arousal) during a video viewing and try to guess

the associated valence using classification tools.

However, as far as we know, such works do not really

allow to answer the three following questions in the

context of movie viewing: i) When did the audience

significantly react ii) How much the audience react for

each movie highlight and iii) What is the emotional

impact of one movie compared to another

In this paper we propose an original affective

benchmarking solution for movies that may bring an

answer to the three previous questions. It is based on a

dedicated processing of the EDA that produces a

time-variant and normalized affective signal (termed

“Affective Profile”) reflecting the significant arousal

variations of the whole audience over the time in a

quantitative and comparable way. Secondly and for the

first time, an evaluation of this framework on a real

audience during regular cinema shows and film festival is

performed for five different movies and a total of 128

audience members. The methodology as well as the

obtained results are presented in the following.

Affective Benchmarking Solution

EDA as an arousal sensor

Getting the mean reaction of a whole audience over the

time is very related to the study of the arousal

fluctuations of this same audience during the show (cf.

the bi-dimensional “Valence / Arousal” representation of

the emotion [12]). The ElectroDermal Activity (EDA)

measures the local variations of the skin electrical

conductance in the palm area which is known to be highly

correlated with the user affective arousal ([8, 1]). This

signal embeds high-level human-centered information and


might be used to provide a continuous and unconscious

feedback about the level of excitation of the end-user.

More specifically, each shift in the arousal flow of a given

user is correlated with slow (∼ 2s) phasic changes in

his/her EDA consisting in a fast increasing step until a

maximum value (the higher the emotional impact is, the

higher the peak is), followed by a slow return to the initial

state.

Individual Affective Profile

For a given audience member, a time-variant and

normalized affective signal, termed “Individual Affective

Profile” (IAP) may be thus directly obtained from the

EDA as described in Figure 1. After a first low-pass

filtering to remove possible artifacts, the EDA is derivated

and truncated to positive values in order to highlight the

relevant phasic changes. The signal is then temporally

filtered and sub-sampled using overlapping time window

to obtain a time-resolution of 30s, sufficient to analyze

the emotional flow and to take into account possible

delays in the user’s reaction during the movie.

To remove the user-dependent part related to the

amplitude of the EDA derivative (which may vary from

one subject to another), the obtained signal, termed p, is

normalized (area under the curve equal to one) so that it

may be interpreted as a probability of reaction over the

time for the current audience member. The resulting

signal termed p n is called “Individual Affective Profile”

whereas the normalization factor termed x n , is called

“Individual Affective Intensity”.

Raw EDA

signal

Individual

affective

profile p n

Low-pass

filtering

Normalize p by its

individual affective

intensity x n

S(t)

p[i]

[t]

Numerical

derivation

S’(t)

Thresholding

Compute the mean value p[i] of

S’ + on every time window W i

S’ + (t)

Figure 1: Description of the different steps of the algorithm to

compute an Individual Affective Profile from the EDA.

Mean Affective Profile

However, taken alone, p n and x n only reflect the reaction

of one given user. Such quantities are thus very related to

the user’s sensitivity and may be also damaged by

user-specific noises (motions artifacts or user’s external

reactions, for instance). To obtain a more relevant

information concerning the arousal variations of a global

audience during a movie, a “Mean Affective Profile”

(MAP) p n is computed by averaging the individual

affective profiles of every audience member p i n, 1 ≤ i ≤ N

(where N is the size of the audience) and by scaling this

quantity according to:

p n = x n

N

N∑

p i n where x n = 1 N

i=1

N∑

x i n (1)

i=1

As a result, p n takes into account not only the arousal

fluctuations of the audience over the time but also the

“Mean Affective Intensity” (MAI) of those reactions,

contained in the scaling factor x n . By observing the

variations of p n , one can compare different moments of a

movie in an affective way and, as those quantities are

scaled to the mean affective intensity, one may also


compare those values between several movies (see Figure

3).

Relevant Affective Parts

However, as soon as N is small, the strong inter-subject

variability (see Figure 3) makes the interpretation of the

quantity p n more tricky. Indeed, the values of p n may be

not statistically significant. A statistical strategy is thus

adopted to discriminate between the background noise

(no or very little affective reaction) and the relevant

information of the MAP.

More precisely, each time sample k is considered as a

random variable P k associated to the latent unknown

distribution of p i n[k], 1 ≤ i ≤ N. The 10% of variables

with the lowest mean are considered as the background

noise (other approaches based on video meta-data may be

also adopted). Then, for every random variable P k the

mean p-value e k is computed by averaging the p-value of

the bilateral Mann-Whitney-Wilcoxon test performed

between the variable P k and each variables of the

background noise (null hypothesis: the distributions of

both groups are equal). Each P k with an associated e k

value lower than 5% is considered as significantly different

from the background noise and is added to the “Relevant

Affective Parts” (RAP).

Evaluation in Theater

To validate the proposed framework and for the first time

in such a context, experiences were conducted on a real

audience during regular cinema shows but also during

special shows in the context of the film festival, “Festival

du Film Britannique de Dinard”.

More precisely, for two french movies proposed at the

Festival de Cannes (2012), namely, “De rouille et d’os”

from Jacques Audiard (movie #1 - drama involving a

handicapped young girl) and “Le grand soir” from Benoît

Delépine and Gustave Kervern (movie #2 - light comedy

with an off-kilter humor), the EDAs of 3 × 14 audience

members were recorded during three different regular

cinema shows in a partner theater named “Le Sévigné”

(Cesson-Sévigné, France). People were offered a free

ticket for their participation and the ages as well as the

genders of the audience (42 audience members per movie)

were reasonably balanced.

In a very similar way, some other experiments were also

conducted during the film festival “Festival du Film

Britannique de Dinard” in partnership with the

organization committee (Dinard, France). For three other

movies, namely, “Now is good” from Ol Parker (movie #3

- upsetting drama involving a young girl with cancer),

“Life in a day” from Kevin MacDonald (movie #4 -

documentary aggregating real life records of multiple

participants) and “I, Anna” from Barnabe Southcombe

(movie #5 - absorbing thriller), the EDAs of respectively

24, 12 and 12 persons were also recorded with quite

balanced ratios in terms of ages and genders.

In both capture sessions, each subject was free to select

its own position in the room (no control of the placement)

which temperature was regulated by the air-conditioning

system of the theater which theoretically prevent from big

temperature variations.

A consumer BodyMedia Armband sensor [6, 11] was

placed on the palm area of each participant (Figure 2) to

record the skin conductance at a 32Hz rate. The entire

recording process was synchronized and controlled and the

EDA was directly stored on a memory embedded on each

sensor (no wireless transmission). The sensor placed on

the fingers was very well accepted by the users who

reported not to have been disturbed by its presence during

the film.

At the end of each show, a simple questionnaire was

submitted to the participant. They were especially asked


to report the different highlights in the movie they

considered as the most relevant in terms of emotional

impact. From those post-hoc questionnaires combined

with the a real-time vocal annotation (in the projectionist

space) of the movie made by three different persons who

have already watched the movies twice, a “ground truth”

of the different highlights of each film was built up.

Figure 2: Bodymedia sensor placed on the fingers.

Results and Discussion

The IAPs, MAPs and RAPs were respectively computed

for the five different movies and results are proposed in

Figures 3, 4 and 5.

For all movies, one can observe a Mean Affective Profile

with variations and peaks quite well synchronized with the

highlights identified from the audience and the vocal

annotation. Those peaks are significantly different from

the “background affective noise” as it can be observed on

the associated p-value traces (Figure 3), and their

amplitudes are consistent with the intensities of the

events. It is especially the case of the “drowning event” of

the movie #1 but also of the ends of movies #3 and #5

which were reported as very shocking, emotional or

thrilling highlights by all audience members. Those events

have consistently a very high mean value and are also

identified as RAPs. On the contrary, non-highlight parts

have a lower mean value and are generally not identified

as RAPs.

Besides punctual analysis, the MAPs also allow to analyze

the temporal and dynamics aspects of the emotional

arousal during each movies. One can first observe how the

values of the different MAPs tend to increase during the

movie. It is especially the case of movies #1, #3 and #5

where a progressive increase of the emotion strain until a

last ultimate event can be notices. Those quantitative

observations are consistent with the qualitative nature of

the movie creation: it is often the goal of a creator to

make the audience more and more involved in the movie

story and the computed MAPs allow a quantitative

verification of this intent.

In line with this latter observation, one can also notice a

quite high peak on almost each movie around the second

1500. This peak may also illustrate a cinematographic

technique well-known by the directors which consists in

“waking-up” the audience after a certain duration from

the beginning (around one third of the whole movie

duration) to make them be “engaged into the movie”.

As described before, the computed MAPs offer the

possibility to quantify punctual as well as dynamic

tendencies of the audience reaction during a movie.

Furthermore, due to their normalized nature, they also

give a way to quantitatively compare the reaction levels in

between different movies. For instance, when comparing

the MAPs of movies #1 to #5, it appears that movies

#2 and #4 tend to have a lower mean intensity than the

others. Such an observation is consistent with the global

comments from the critics: movies #1, #3 and #5 are

respectively upsetting dramas or absorbing thriller whereas

movies #2 and #4 are respectively a light comedy with

an off-kilter humor and an “experimental” documentary.

Regarding the scattering of the IAPs (Figure 3), even if a


quite high variability may be noticed, the behavior of the

median values seems to confirm the trend already

observed on the mean values and give a rough idea of the

inter-subject dependency. One can however point out a

higher variability for the high mean (or median) values of

the IAPs. This behavior may be partly explained by the

facts that: i) when nothing happens in the movie,

everybody globally don’t react, whereas ii) when an event

occurs in the movie, some of the audience members

significantly react whereas others remain insensitive.

Variability is thus higher for larger RAPs but those parts

remain statistically different from the background noise

according to the associated p-values.

Finally, a last remark can be pointed out regarding the

interest of using the EDA to quantify sustained states. In

such situations, one could think that the EDA only reacts

to quite strong and precise events and thus, that nothing

could be noticeable in the computed MAPs for sustained

parts of the movie which are supposed to be long and

slightly progressive increase of the subject’s arousal. In

fact, the EDA does not react once but repeatedly and

regular peaks may be observed in such situations. Those

peaks are thus detectable in the positive-truncated

derivative and appear in the associated MPAs as one can

especially observe at the thrilling end of the movie #5

where the intensity remains high for more that 300

seconds.

Conclusion and Future Works

In this paper we have proposed a new solution to

affectively benchmark a movie in terms of arousal

variations. It is based on the measurement of the EDA

and a dedicated processing that produces a time-variant

and normalized affective signal. This signal, termed

“Affective Profile”, is easily usable for a content creator

that would be interested by studying its movie in an

objective and affective manner or for studio and

distributors interested by marketing analysis and

commercial predictions. The framework has been carried

out, for the first time in such a context, on a real

audience during regular cinema shows and film festival, for

five different movies and a total of 128 audience

members. The obtained results in such an ecological

context are consistent with the explicit reporting of the

participants and prove that such a solution may offer new

possibilities to the content creators and distributors to

analyze and compare their movies in an affective way.

A deeper validation on more movies has now to be done

as well as a refinement of the analysis on sub-categories of

the audience (by gender, age, ...). Correlation in between

audience member reactions could be indeed interesting

paths to investigate. Furthermore, in this paper, the

questions of when and how much an audience reacts have

been addressed but, the problem of how this same

audience reacts (positively or negatively) has been eluded.

This question of the valence of the reactions is thus

another interesting research direction for the future that

we will have to consider.

References

[1] Boucsein, W. Electrodermal Activity (Second

Edition). Springer, 2012.

[2] Calvo, R., and D’Mello, S. Affect Detection: An

Interdisciplinary Review of Models, Methods, and

their Applications. IEEE Trans. on Affective

Computing 1, 1 (2010), 18–37.

[3] Canini, L., Gilroy, S., Cavazza, M., Leonardi, R., and

Benini, S. Users’ response to affective film content:

A narrative perspective. vol. 1 (2010), 1–6.

[4] Fletcher, R., Dobson, K., Goodwin, M., Eydgahi, H.,

Wilder-Smith, O., Fernholz, D., Kuboyama, Y.,

Hedman, E., Poh, M., and Picard, R. iCalm:


Ali is meeting

his sister

again

emotional

scene

Stéphanie’s

Accident

shocking

scene

The dog is put in

kennels and the son is

extremely sad

shocking scene

Ali is shaking

his son

violently

shocking

scene

Scene of

violent

freefight

shocking

scene

Love scene

with sport

coach

love

scene

First love scene

between Ali and

Stéphanie

(« opé »)

love scene

Moving scene

between Stéphanie

and the son

emotional scene

Sister’s layoff

shocking

scene

Son’s

drowning

shocking

scene

Figure 3: Affective benchmarking solution applied to movies #1. (top) Box plot of every “Individual Affective Profiles”. Each box

represents the distribution of the IAPs for every users at a given time. The central mark is the median, the edges of the box are the

25th and 75th percentiles, the whiskers extend to the most extreme data points not considered outliers, and outliers are plotted

individually. (middle) Mean p-value of the bilateral Mann-Whitney-Wilcoxon test. Time samples with a p-values under the red line (p

= 0.05) are considered as “Relevant Affective Parts”. (down) “Mean Affective Profile”. Each bar represents the mean arousal of the

audience at a given time sample. Red horizontal lines are drawn above “Relevant Affective Parts”. The different highlights in the movie

considered by the audience as the most relevant in terms of emotional impact are also superimposed.


Ali is meeting

his sister

again

emotional

scene

Stéphanie’s

Accident

shocking

scene

The dog is put in

kennels and the

son is extremely

sad

shocking scene

Ali is shaking

his son

violently

shocking

scene

Scene of

violent

freefight

shocking

scene

Love scene

with sport

coach

love

scene

First love scene

between Ali and

Stéphanie

(« opé »)

love scene

Moving scene

between

Stéphanie

and the son

emotional

scene

Sister’s layoff

shocking

scene

Son’s

drowning

shocking

scene

Mother’s

birthday

funny

scene

« Not » is

fidgeting in

front of people

eating at

restaurant

funny scene

The mother is

pushing her

strange trolley

with the baby

inside

funny scene

« Not » is

Jean-Pierre is trying to make

discovering with his brother

his banker that he hired

has been ruined funny scene

by his wife

funny scene

The parking

attendant is

discussing with

the father

funny scene

Scene with the

hanged person

funny scene

Jean-Pierre is trying to

convince a former

colleague to take part

in his « revolution »

funny scene

Figure 4: “Mean Affective Profiles” computed during regular movie shows. Movies #1 (top) and #2 (bottom). Each bar represents

the mean arousal of the audience at a given time sample. Red horizontal lines are drawn above “Relevant Affective Parts”. The

different highlights in the movie considered by the audience as the most relevant in terms of emotional impact are also superimposed.


First kiss

Movie #3

with wig

shoking

Mushroom

consumption

funny

Catheter

removal

shoking

Blackout

close to

the fire

shoking

Adam is

Motorbiking

discovering

with Adam

Tess's cancer

thrilling

and is

Arrested in the

seeking her in supermarket

the forest

thrilling

emotional

Robery in the

supermarket

thrilling

Adam is

introduced to

the father +

On the

Abortion

beach with

confusing

Adam

romantic

Tess's

blackout and

hospitalization

shoking

Yes/No with

mother in Ice-skating

hospital + No

emotional abortion

Tess's

thrilling

nose is

bleeding Discovery

shoking of Tess'

tags

thrilling

Adam is

jumping

through the

window +

love scene

romantic

Tess's dad

is very sad

and is

crying

emotional

Tess's

death

description

emotional

Movie #4

Discussion

with a drunk

guy

funny

A guy is

eating an egg

with a chick

inside

disgusting

Cow in the

abattoir

disgusting

A gay is

annuncing his

homosexuality

to his grandmother

emotional

A funny priest

is re-marrying

a couple

funny

A urban

climber is

climbing

buildings and

a bus

impressive

Lighting and flying

balls are launched

in the sky

emotional

Movie #5

Discovering of

the scene

where the

crime

happened

thrilling

The

policeman is

chasing Anna

close to the

phone box

thrilling

The policeman is

chasing Anna in

his car during the

night

thrilling

Anna is walking

very fast with the

Unvolontary

stroller and is

speed-dating +

pushing an

First meeting

empty swing

between Anna

troubling

and the

policeman

romantic George's son is

chased by the

police

thrilling

Call between

Anna and the

policeman during

the night

romantic

Anna is lying to

The baby Suicide

the policeman

has a attempt

about their first

crash thrilling

The policeman is

meeting

thrilling

discovering the

troubling

empty room

thrilling

Anna with

Sexual scene her knife in

between Anna the kitchen

and Georges + thrilling

Anna is upset in

the elevator

The police is

thrilling

trying to arrest

Anna

thrilling

First speeddating

schemer

Figure 5: “Mean Affective Profiles” computed during the “Festival du Film Britannique de Dinard”. Movies #3 (top), #4 (middle)

and #5 (bottom). Each bar represents the mean arousal of the audience at a given time sample. Red horizontal lines are drawn above

“Relevant Affective Parts”. The different highlights in the movie considered by the audience as the most relevant in terms of emotional

impact are also superimposed.


Wearable Sensor and Network Architecture for

Wirelessly Communicating and Logging Autonomic

Activity. IEEE Trans. Inf. Technol. Biomed. 14, 2

(2010).

[5] Fleureau, J., Guillotel, P., and Huynh-Thu, Q.

Physiological-based affect event detector for

entertainment video applications. IEEE Trans. on

Affective Computing 99, PrePrints (2012).

[6] Jakicic, J., Marcus, M., Gallagher, K., Randall, C.,

Thomas, E., Goss, F., and Robertson, R. Evaluation

of the sensewear pro armband to assess energy

expenditure during exercise. Med. Sci. Sports Exerc.

36, 5 (2004), 897–904.

[7] Kleinkopf, H. A Pilot Study of Galvanic Skin

Response to Motion Picture Violence. Texas Tech

University, 1975.

[8] Lang, P. The emotion probe. Am. Psychol. 50, 5

(1995), 372–385.

[9] Matthews, R., McDonald, N., Hervieux, P., Turner,

P., and Steindorf, M. A Wearable Physiological

Sensor Suite for Unobtrusive Monitoring of

Physiological and Cognitive State. In Conf. Proc.

IEEE EMBS (2007), 5276–5281.

[10] Pantic, M., and Rothkrantz, L. Automatic Analysis

of Facial Expressions: The State of the Art. IEEE

PAMI 22, 12 (2000), 1424–1445.

[11] Rothwell, S., Lehane, B., Chan, C., Smeaton, A.,

OConnor, N., Jones, G., and Diamond, D. The

CDVPlex biometric cinema: sensing physiological

responses to emotional stimuli in film. In ICPA,

Citeseer (2006), 1–4.

[12] Russell, J. A circumplex model of affect. J. Pers.

Soc. Psychol. 39, 6 (1980), 1161–1178.

[13] Soleymani, M., Pantic, M., and Pun, T. Multi-modal

emotion recognition in response to videos. IEEE

Trans. on Affective Computing 3, 2 (2011), 211–223.

Similar magazines