EEG/ECG data fusion using Self-Organising Maps

EEG/ECG data fusion using Self-Organising Maps
Nuno Bandeira 1 , nb@di.fct.unl.pt
Victor Sousa Lobo 2,1 , vsl@di.fct.unl.pt
Fernando Moura-Pires 1 , fmp@di.fct.unl.pt
1 Computer Science Department, Faculty of Science and Technology / New
University of Lisbon, Portugal
2 Portuguese Naval Academy, Lisbon, Portugal
Abstract
Empirical results are presented concerning
data fusion performed over several
combinations of EEG/ECG channel readings of
sport shooting athletes. Our purpose in applying
different data fusion approaches was that of
finding a satisfactory set of features, such that
would allow us to build adequate classifiers on
the data. The resulting data sets were used for
building SOMs (Self-Organising Maps), used
for visual inspection of coherence between
clusters found and shooting accuracy.
Keywords: Self-Organising Maps, EEG, data
fusion
1. Introduction
According to sport shooting experts, the
shooter’s ability to concentrate on the shooting
task is crucial in improving one’s performance,
once high physical technique levels have been
achieved (steady body position, respiration,
muscular and eye-movement control). Since
concentration is mainly a cerebral activity,we
conducted an experiment where EEG and ECG
signals were read and digitised in real-time
during the shooting activity. Previous work (see
[9]) suggested that these could be good
indicators of concentration.
Once we had all the data (around 80Mb -120
Mb per shooting session), we had to devise
adequate pre-processing techniques in order to
handle the high volume of data. Many
techniques are known for transforming EEG
data into feature vectors suitable for clustering
and classification [2][10][11][3]. We opted for
the use of Fast Fourier Transforms (FFT) as
described in the next section.
But the best that the FFT could give us were
different types of channel spectra, therefore
resulting in 20 spectra per shot, one per EEG
channel. Since we wanted to apply SOMs to
visually inspect potential hidden relations in our
data, we also had to find ways of merging all
channels into single feature vectors. Different
approaches were tried and are described in
sections 4 and 5.
2. EEG/ECG signal acquisition
and pre-processing
The subjects from which we recorded our data
are shooters from the sport shooting team of the
Portuguese Navy. So far we have recorded data
from 7 such shooters, but because of difficulties
with the recording software, in this paper we
only present the results of shooters numbers 3
to 7. Each shooter spent one morning at a
shooting range, firing up to 12 rounds of 5
shots. For each shot, besides the EEG and ECG,
we kept the target, and classified the shot
according to the score obtained (10 is right on
the centre, 0 is outside the target). We then
considered that shots with a score of 9 or 10
were good, 7 or 8 were average, and up to 6
were bad.
The electrodes were placed according to the
standard 10-20 system [7][8].
The electrode leads are connected to a
Braintronics ISO1032 preamplifier, that sends
the signal to a Braintronics CONTROL 1032
amplifier. There the signals are amplified,
filtered by a 50Hz notch filter and a 4 th order
70Hz low-pass filter. A DATA TRANS-
LATION DT2821 ADC board is then used to
digitise the resulting signals.
The recorded data consists of 22 signals,
recorded with 12 bit resolution and a 512Hz
sampling rate. Channel 22 is the ECG, from

where the heart beat rate is extracted using a
simple spectra based algorithm. Channel 18 is
the signal of the right ear, that is used as
reference for the differential amplifiers, and
thus contains no information. The remaining 20
channels are all subject to the same initial preprocessing.
First, the last 5 seconds before the
shot are selected (2.5K points). This signal is
then broken up into 9 blocks of 512 points, with
50 % overlap between them (so as to later
obtain a Walsh periodogram). Each of these
blocks is then multiplied by a Hamming
window to reduce frequency leakage, and it’s
spectrum is calculated with a 512 point FFT.
Thus each channel produces 9 spectra with 256
bins of real frequencies and a width of 1Hz.
Since we used a 70Hz low-pass filter and
standard EEG bands range from 1-30Hz we
opted for using only the lower 30 bins.
Thus, when all information is used, we have
20 EEG channels with 9 spectra of 30 bins,
totaling 5400 EEG features, plus one heart beat
rate feature. All subsequent pre-processing is
done on this EEG data.
3. SOM - Self Organising Maps
Self-Organising Maps, also known as
Kohonen Maps, in honour of its creator, are
thoroughly described in [5] and have been
widely used in many applications, including as
a tool for data fusion [4] (an excellent
bibliography can be found in
http://www.cis.hut.fi/nnrc/index.html). The
SOM concept is based on the human brain’s
cortex interactions, simplified in a model in
which different prototypes (neurones) try to
represent the input data by competing with each
and every other neurone in every iteration, for a
better mapping of the input data.
The basic SOM algorithmic procedure is as
follows:
1. For a given training pattern x:
1.1 Calculate the distance of each neurone to
the training pattern x (Calculation phase)
1.2 Find the neurone with smaller distance,
and call it the winner W (Voting phase)
1.3 Change the network neurones with a
function G, which depends on the learning
rate α, the distance d to W (in the output
plane), and the neighbourhood function F.
Due to the nature of the neighbourhood
function, only the neurones closer to W (in
the output space) will be changed. (Update
phase)
2. Update the learning rate α and the
neighbourhood function F according to
some rule
3. Repeat steps 1 and 2 for the next training
pattern, until some stopping criteria is
reached.
In all our analysis we ran our algorithms 6
times with different initial values, to make sure
that the process always converged to the same
final map. Whenever this did not happen, we
simply increased the number of iterations and
the initial learning radius until a stable solution
was found.
A distributed SOM implementation was also
used in building the maps for the largest dataset
(5401 features). A detailed description of this
algorithm and its empirical evaluation can be
found in [1].
To visualise the results of the clustering
performed by SOM, we frequently used Umatrices
[12]. The U-Matrix of a SOM is
obtained by calculating the distance, in the
input space, between neighbouring neurones.
These distances are then represented on a map
in grayscale (black being the greatest distance,
and white the smallest). Clusters can easily be
identified as clear areas (nearby neurones)
separated by dark ridges (large distances to
other clusters).
4. Data fusion
Since our main objective was finding an
adequate set of features that would provide high
visual correlation between the EEG signal and
shooting performance, most of our work at this
stage was concentrated on data fusion.
Due to this fact, data fusion was performed
using different types of feature aggregation,
motivated by several different reasons. Each
choice of feature aggregation led to a different
training set, upon which the clustering
procedure was applied.
The training sets used were:
I) All the features (1 set of 5401 features).
In this training set, all features mentioned in
Section 2 are used. It seems reasonable that
this approach would capture the dynamics of
the signal prior to the shot. In this set, the
heart beat rate is also used.
II) All average spectra. (1 set of 601 features).

In this training set, we averaged the spectra of
each channel, and by doing so assumed that
the signals are stationary during the 5 seconds
before the shot. Each resulting spectrum is the
average of 9 spectra, and thus the signal to
noise ratio is improved considerably. In this
set, the heart beat rate is also used.
III) Average spectra separated by hemisphere (2
sets of 330 features).
In these training sets, we separated the data in
right and left hemisphere. Each hemisphere
consists of 8 EEG channels unique to that
hemisphere, plus the three central channels
(Fz, Cz, Pz [7][8]). This choice of features is
motivated by the fact that the left and right
sides of the brain are reasonably distinct and
all but one of the shooters were right handed
and used the right eye for aiming. We
performed clustering on each side separately,
and later merged the results.
IV) Average spectra by channel (19 sets of
30 features).
In these training sets, we used the spectra of
each channel as a separate training set. By
analysing the ability to cluster the data
sensibly, based on each channel
independently, we tried to determine if there
were any channels more relevant to the task at
hand.
V) Characteristic frequency bands (4 sets of
120 features for the alpha band, 320 for beta,
and 80 for delta and theta).
In these training sets, power spectra within
each band (alpha, beta, delta, and theta [6])
were selected for all channels. Since each
band has a different width, the number of
features selected varies. According to classic
literature in the area [6] these frequency bands
correspond to well established activity
patterns within the brain, and thus are the
natural choice for discriminating between the
shots.
5. Decision fusion
Decision fusion was used for merging the
results obtained in III and IV. The generated
datasets were used in building the
corresponding SOMs, which were then labelled
using the same datasets. Labelling a SOM
consists in finding the winning neurone in a
SOM for each data vector in the dataset and
appending the data vector’s class label to the
neurone’s label. This labelling (usually called
calibration in SOM terms) allowed us to use the
SOMs as classifiers, simply by having each
neurone belong to the class that, in its label, is
most frequent.
Two different strategies were applied in
fusing the SOMs classifications:
- Majority (III, IV). This is the simplest
decision fusion, where the final class is
simply the most ocurring class in the
lower level classifiers. It is used for
evaluation of variation in SOMs
classifications.
- Use of another SOM layer (IV). In this
case, the results of the classification by
the original SOMs are fed as features to a
fusing SOM. It is then used for visual
inspection of dispersion of the first level
SOMs classifications. Higher levels of
agreement on the first level SOMs should
lead to a smooth fusion SOM. If the
decision fusion SOM is messy or has
outliers, then there is disagreement on the
first level SOMs. In such cases, simply by
glancing at the outliers’ neighbours it’s
easy to spot which class most of the first
level classifiers chose for it.
6. Results
With all datasets, with the exceptions of VI
that will be discussed later, and certain channels
of IV, the data was clustered by shooter. We
present these results for the first dataset in
Figure 2, and the others are very similar.
Since the data is clustered by shooter, it
cannot be clustered by score. So as to cluster by
score, it is necessary to join all good shots in
one cluster and bad ones in another, thus
mixing in those clusters the different shooters
(which as in this case does not happen). Thus,
to classify the shots by score we have to analyse
the data of each shooter individually.
None of the datasets tested provides good
clustering by score for all shooters. However, 2
of the individual channels (F7 and T3) provided
reasonable clustering by scores, even when all
shooters are considered simultaneously.
Furthermore, some of the shooters have their
shots clustered by score with some of the
datasets. Shooter 3 has his shots clustered by
score in dataset IV (channel Cz), shooter 4 in
dataset V, shooter 5 in datasets II, IV and V,
and shooter 7 in dataset IV. With shooter 6 no

dataset was capable of clustering his shots by
score.
To visualise the maps produced, we shall
represent the mapped shots as crosses if they
correspond to good shots, triangles if they
correspond to average shots, and circles
otherwise, as shown in Figure 1.
Figure 1 - Legend for the maps
The results for each of the training sets
presented in section 4 are as follows:
I) Training set with all 5401 features.
Different shooters are clearly identified for, as
we can see in Figure 2, shooters 4 and 5 have
very distinct clusters (separated from the others
by dark lines in the U-Matrix), and shooters 6,
7, and 3, while in the same cluster, are mapped
to different areas. To obtain these maps we used
the distributed version of SOM mentioned in
section 3. This allowed us to reduce the total
training time of each map from 2h21m to
1h16m when using 2 machines, and even more
when more machines where available.
6 6 6 6 6 7 7 4 4 4 4
6 6 7 7 7 4 4 4
7 6 6 6 6 6 7 4 4 4
6 6 6 6 6 6 4 4 4
6 6 6 6 6 3 3 3 4
6 5 3 3 3 4 4 4
5 5 5 5 3 3 3
5 5 5 5 3 3 3 4 4 4
Figure 2 - U-Matrix obtained after applying a
SOM to training set I.
If we train SOM maps for each individual
shooter, the results are generally bad.
II) Training set with Average Spectra (601
features)
This training set is only useful for shooter 5.
In Figure 3 we can see that all good shots are
on the upper right corner, while the average
shots are on the bottom left. Furthermore, in
the U-Matrix presented in Figure 4 we can see
that there is a clear distinction between these
two areas. It could be argued that there are
some good shots in the “bad area”, but these
are probably outliers that correspond to lucky
shots that are good despite the bad conditions.
Figure 3 - Map obtained with training set II
for shooter 5
9 9 10 9
8 10 9
10 9 9 9
8 9 8 9
Figure 4 - U-Matrix obtained with training set
II for shooter 5.
III) Training set with average spectra separated
by hemisphere (2 sets of 330 features).
We were unable to obtain good clusters of the
data by scores for any shooter with this dataset.
However, as with all others, we could cluster
rather easily by shooter. When we fused the
results obtained by each hemisphere, we
managed to obtain the results presented in
Table 1.
3 4 5 6 7 All
Best
Hemisphere
50 70 57 53 45 47
Fusion 53 70 70 58 60 55
Gain 3 0 13 5 14 8
Table 1 - Percentage of correct classification
for each shooter, with and without decision
fusion
IV) Training set with average spectra by
channel (19 sets of 30 features).
10

IV.i) Individual Channels
Channels F7 (left frontal), and T3 (left
temporal) proved to be quite good at clustering
by score. Figure 5 shows the results for all
shooters, and we can see an area of bad shots on
the left center, and an area of “confusion” on
the lower right, with good shots on most other
areas.
Figure 5 - Map obtained with training set IV,
channel T3, for all shooters.
When considering individual shooters, shooter
5 again has his shots clustered by score, but
now we can also do the same for shooter 7 and
3, as can be seen in Figure 6 and Figure 7.
Figure 6 - Map obtained with training set IV,
channel Cz, for shooter 3.
Figure 7 - Map obtained with training set IV,
channel Fz, for shooter 7
IV.ii) Fusion by majority
As can be seen in Table 2, the biggest
improvements were achieved with shooter 4
and all shooters together, with an increase of
18% and 17% of correct answers. Average
improvement was 11%.
3 4 5 6 7 ALL
Best
channel
57 72 75 56 60 51
Fusion 67 83 75 69 78 68
Gain 10 11 0 13 18 17
Table 2 - Percentage of correct classifications
for each shooter, with and without decision
fusion
IV.iii) Fusion by another SOM layer
As expected from shooter 4’s low error rate
in fusion by majority, his map is pretty
clean, having only one average shot amidst
the bad shots, as can be seen in Figure 8.
Figure 8 – Results of fusion of the individual
channels by a SOM, for shooter 4.

Figure 9 - Results of fusion with a SOM layer
for shooter 6.
Figure 10 - Results of fusion with a SOM
layer for shooter 3.
On the other hand, the maps of Figure 9 and
Figure 10 show that shooters 3 and 6 are in very
distinct situations, although their results in
fusion by majority are quite similar. In shooter
6’s map there are many neurones that have both
good and average class labels, indicating that
this set of features is not good enough. In
shooter 3’s map, we can observe that this
choice of features led to an embedding of the
average shots amidst the good ones (in the
SOM’s 2D projection). This leads to the
conclusion that, for the data vectors in the
bottom left corner, most of the single-channel
SOMs classified these good shots as below
average. So in this case, what we have is not a
mix-up, but a set of data vectors that should be
put aside and carefully analysed. A possible
solution to this kind of error could be the
addition of another classifier, prior to decision
fusion, that would mainly handle this cluster.
V) Training sets with characteristic frequency
bands (alpha beta, delta and theta).
The alpha band, separated the shooters better
then most others, with the advantage that it uses
only 6 features. It was also useful in separating
shooter 5’s shots by score, as can be seen in
Figure 11.
Figure 11 - Map obtained with training set V,
Alpha band, for shooter 5.
The beta band provided the best separation of
all amongst shooters. It was however useless in
separating by score, as was the theta band.
The delta band provided a reasonable clustering
of shooter 4’s scores, as can be seen in Figure
12.
Figure 12 - Map obtained with training set V,
delta band, for shooter 4.
6. Conclusion
Our main objective in this first phase of our
work was to gain further insight over our data,
which was accomplished. Although most of the
data fusion possibilities had a strong imprint of
each individual’s personal EEG traces, we were
able to find some clues as to what were the
important frequency ranges in each case. These
have already led us to try and find new criteria
that combine different frequency bands, on
which we are currently working.
The data from the ECG did not influence in
any way our results, since it was almost

constant for each shooter, and even amongst
shooters the differences were not significant.
Our use of SOMs as a clustering tool for
visual inspection of our data fusion options was,
as we expected, very useful. Also, decision
fusion wise, we found SOMs to be very useful
as a visual inspection tool of a set of classifiers.
High coherence means cleaner, smoother maps,
messy maps mean lots of variance within the
classifier set and maps with disjoint clusters
could be good indicators that an extra classifier
is needed to handle specific data partitions.
7. Bibliography
[1] Bandeira N, Lobo V, Moura-Pires F:
Training a Self-Organizing Map Distributed on
a PVM Network, Proceedings of IEEE Joint
Conference on Neural Networks, 1998
[2] John ER, Prichep LS : Principles of
Neurometric analysis of EEG and Evoked
Potentials, Electroencephalography, 4th ed
(Niedermeyer E, ed), Williams & Wilkins,
1993, 989-1003
[3] Kalayci T, Özdamar Ö : Wavelet preprocessing
for automated neural network
detection of EEG spikes, IEEE Engineering in
Medicine and Biology, March/April 1995, 160-
166
[4] Lobo VS, Bandeira N, Moura-Pires F:
Distributed Kohonen Networks for Passive
Sonar Based Classification, Proceedings of the
International Conference on Multisource-
Multisensor Information Fusion – FUSION 98
[5] Kohonen T : Self-Organizing Maps,
Springer-Verlag, 1995
[6] Niedermeyer E: The Normal EEG of the
Waking Adult, in Electroencephalography, 4th
ed (Niedermeyer E, ed), Williams & Wilkins,
1993, 131-152
[7] Nuwer MR, et al.: IFCN guidelines for
topographic and frequency analysis of EEGs
and EPs. Report of an IFCM committee.
[8] Reilly E: EEG Recording and Operation of
the Apparatus, in Electroencephalography, 4th
ed (Niedermeyer E, ed), Williams & Wilkins,
1993, 104-124.
[9] Schober F, Schellenberg R, Dimpfel W :
Reflection of mental exercise in the dynamic
quantitative topographical EEG,
Neuropsychobiology 1995;31; 98-112
[10] Silva FL : EEG analysis: Theory and
Practice, Electroencephalography, 4th ed
(Niedermeyer E, ed), Williams & Wilkins,
1993, 1097-1123
[11] Silva FL : Computer-assisted EEG
diagnosis: Pattern recognition and brain
mapping, in Electroencephalography, 4th ed
(Niedermeyer E, ed), Williams & Wilkins,
1993, 1063-1086
[12] Ultsch A: Siemon, H.P.; “Exploratory
Data Analysis Using Kohonen Networks on
Transputers”, Dep.of Comp.Science, Dortmund
FRG, December 1989.