Customized PLE implementations for test-bed scenario ... - ROLE

Integrated Project
Responsive Open
Learning Environments
European Commission Seventh Framework Project (IST-231396)
Deliverable D4.3
Customized PLE implementations for test-bed scenario
validations
Editor
Felix Mödritscher, Alexander Mikroyannidis
Work Package
WP4
Status
Final
Date January 31, 2011
Page 1 of 102

The ROLE Consortium
Beneficiary
Number
Beneficiary name Beneficiary short name Country
1 Fraunhofer FIT FHG DE
2 RWTH Aachen University RWTH Aachen DE
3 Technical University of Graz TUG AT
4 Katholieke Universiteit Leuven K.U.LEUVEN BE
5 University of Koblenz UNI KO-LD DE
6 Uppsala University UU SE
7 École Polytechnique Fédérale de
Lausanne
EPFL
CH
8 University of Leicester ULEIC UK
9 Open University UK OU UK
10 Vienna University of Economics &
Business
WU
AT
11 Festo Lernzentrum Saar GmbH FESTO DE
12 imc AG IMC DE
13 British Institute for Learning and
Development
14 Shanghai Jiao Tong University,
China
BILD
SJTU
UK
RPC
15 Zentrum für Soziale Innovation ZSI AT
16 U&I Learning UIL BE
Page 2 of 102

Document Control
Title:
Editors:
E-mail:
Customized PLE implementations for test-bed scenario
validations
Felix Mödritscher (WU), Alexander Mikroyannidis (OU)
felix.moedritscher@wu.ac.at, a.mikroyannidis@open.ac.uk
Amendment History
Version Date Author/Editor Description/Comments
1 Oct 6, 2010 Felix Mödritscher Preparation of document and first structure
2 Nov 3, 2010 Felix Mödritscher Incorporated feedback on the four perspectives
(WP1, WP3, WP6, WP7)
3 Nov 24, 2010 Felix Mödritscher,
Barbara Krumay
Incorporated feedback on questions
4 Dec 22, 2010 Felix Mödritscher Method of test-bed studies
5 Dec 28, 2010 Felix Mödritscher,
Alexander
Mikroyannidis
6 Dec 30, 2010 Felix Mödritscher,
Barbara Krumay,
Carsten Ullrich
7 Jan 3, 2011 Felix Mödritscher,
Barbara Krumay, Bodo
von der Heiden
8 Jan 4, 2011 Felix Mödritscher,
Barbara Krumay,
Michael Werkle, Manuel
Schmidt
9 Jan 10, 2011 Felix Mödritscher,
Barbara Krumay, Sarah
Wills, Joe Dobbs
10 Jan 14, 2011 Felix Mödritscher,
Alexander
Mikroyannidis
Summary on pilot test-bed study (OU)
SJTU test-bed summary
RWTH test-bed summary
FESTO test-bed summary
BILD test-bed summary
Executive Summary, Introduction, Concluding
remarks
11 Jan 23, 2011
Jan 27, 2011
Denis Gillet, Milos
Kravcik
Internal review
12 Jan 31, 2011 Felix Mödritscher,
Barbara Krumay
Incorporation of reviewer comments, finalisation
of document
Page 3 of 102

Contributors
Name
Felix Mödritscher, Barbara Krumay
Alexander Mikroyannidis
Alexander Nussbaumer
Katrien Verbert, Sten Govaerts
Effie Law
Zinayida Petrushyna, Bodo von der Heiden
Carsten Ullrich
Michael Werkle, Manuel Schmidt
Sarah Wills
Joe Dobbs
Institution
WU
OU
TUG
K.U.Leuven
ULEIC
RWTH Aachen
SJTU
FESTO
BILD
Etestme (BILD member)
Reviewers
Name
Denis Gillet
Milos Kravcik
Institution
EPFL
RWTH Aachen
Legal Notices
The information in this document is subject to change without notice.
The Members of the ROLE Consortium make no warranty of any kind with regard to this
document, including, but not limited to, the implied warranties of merchantability and fitness for
a particular purpose. The Members of the ROLE Consortium shall not be held liable for errors
contained herein or direct, indirect, special, incidental or consequential damages in connection
with the furnishing, performance, or use of this material.
Page 4 of 102

Executive Summary
In order to provide customized personal learning environment (PLE) implementations
for real-world test-beds, this deliverable aims at elaborating a methodology to validate
educational settings according to the ROLE vision. Basically, the overall approach
consists of two parts. In the first part the most relevant results from the first 18 months
of the ROLE project are summarised in order to outline the key concepts and to
formulate questions for analysing and improving real-world scenarios with respect to
ROLE vision. Thereby, project outcomes are presented along four dimensions related to
PLEs, namely the psycho-pedagogical, the community, the technical, and the
development perspective. In the second part of the deliverable, the ROLE test-beds and
their results from former project milestones are analysed through this PLE scenario
validation method. The study gives an overview of starting points, ROLE-related
improvements and future plans for each test-bed. Additionally, experiences and
limitations of realising the ROLE vision in the test-beds are manifested.
Overall, the findings of the test-bed validation study can be summarised as follows:
• In university settings, like OU, SJTU and RWTH, learning is driven by the
organisation, primarily through the efforts of teaching staff and by applying LMS
technology. Yet, higher and further education settings are open to new
pedagogical, community-oriented, or technical concepts. At the university testbeds,
various ROLE ideas have been realised before or within the project.
• Corporate settings, like FESTO and BILD, are highly focussing on competences
related to their businesses, thus being praxis-oriented but also very
instutionalised. However, it requires more efforts to open up or personalise
learning environments. Various aspects of the ROLE vision are restricted by
organisational policies or boundaries. On the other side, these test-beds also
offer great opportunities for exploiting ROLE concepts commercially.
With respect to work package 4 (‘ROLE services’), this document proposes a
methodology to validate real-world learning scenarios under consideration of psychopedagogical
and community issues and according to the technical infrastructure, the
personal learning services, and novel software development methods. Additionally, it is
shown how this validation can be conducted in praxis and with the aim to analyse and
improve real-world test-beds.
This deliverable relates to the five ROLE objectives (cf. ROLE-DoW, p.6), as it
summarises technical approaches (RO1: support of individual assembly of learning
environments), psycho-pedagogical aspects (RO2: psycho-pedagogically sound
framework), community-oriented PLE development (RO3: new engineering methods
considering the different stakeholder communities), and evaluation methodologies
(RO4: evaluation of the effectiveness of PLEs in the test-beds) within the ROLE project.
Moreover, this document indicates possible exploitation strategies (RO5), e.g. within the
BILD test-bed. The deliverable is related to task 4.4 (‘Test-bed Technical Platforms’)
and a joint outcome of the work packages 1, 2, 3, 4, 5, 6, 7, and 9. It also includes
contributions of an external partner (Etestme).
Page 5 of 102

Table of Contents
EXECUTIVE SUMMARY 5
1 INTRODUCTION 7
2 SCENARIO-DRIVEN PLE DEVELOPMENT AND APPLICATION 8
2.1 Vision and goals 8
2.2 The psycho-pedagogical perspective (RO2, RO4) 9
2.3 The community perspective (RO3, RO4) 12
2.4 The technical perspective (RO1, RO4) 16
2.5 The development perspective (RO1, RO3, RO4) 21
3 TEST-BED VALIDATION STUDY 24
3.1 Methodology of the study 24
3.2 OU test-bed: Open, community-oriented but LMS-based and driven by the
organisation 24
3.3 SJTU test-bed: Flexible, interactive, rich in instructional variety but highly
teacher-driven 27
3.4 RWTH test-bed: Innovative, praxis and outcome-oriented but driven by teachers
and the organisation 30
3.5 FESTO test-bed: Web-based, workplace and praxis-oriented but teacher-driven,
hardly collaborative and not personalised 33
3.6 BILD: A gateway to many test-beds 36
4 CONCLUDING REMARKS 40
APPENDIX A: QUESTIONNAIRE FOR THE TEST-BED SURVEY 42
A.1 ROLE - Test-bed validation, community-perspective 42
A.2 ROLE - Test-bed validation, psycho-pedagogical level 55
A.3 ROLE - Test-bed validation, SRE 67
A.4 ROLE - Test-bed validation, technical perspective 72
APPENDIX B: RESULTS OF THE CHRISTMAS, EASTER, AND STONEHENGE PROJECT 91
B.1 OU test-bed 91
B.2 SJTU test-bed 93
B.3 RWTH test-bed 95
B.4 FESTO test-bed 97
B.5 BILD test-bed 99
REFERENCES 101
Page 6 of 102

1 Introduction
The primary goal of task 4.4 (‘Test-bed Technical Platforms’) is to provide customized
personal learning services (PLS) for the ROLE test-beds. As a step towards this goal,
this deliverable aims at elaborating a methodology for validating real-world learning
scenarios according to the vision of the ROLE project, i.e. by analysing and improving
learning through the application of personal learning environment (PLE) concepts and
technology. After two years of research and development, this vision has been made
concrete by various deliverables published in the first 18 months of the project.
Therefore, the overall approach in this deliverables is split into two parts.
In the first part the former outcomes of ROLE are reflected and summarised with
respect to the core idea of the research project. Precisely, section 2 starts with
presenting the vision and the goals of the research project and highlights the key
concepts of the former deliverables. Thereby, the project outcomes are presented along
four ROLE-related dimensions, namely the psycho-pedagogical, the community, the
technical, and the development perspective. At the end of each section, concrete
questions for validating learning scenarios according the development and application
of PLEs are formulated.
The second part of the deliverable (section 3) shows how this PLE validation
methodology can be applied in practice. Therefore, the ROLE test-beds and their results
from former project milestones (Christmas, Easter, and Stonehenge Project) are
analysed through this PLE scenario validation method. The study gives an overview of
starting points, ROLE-related improvements and future plans for each test-bed.
Additionally, findings on and limitations of realising the ROLE vision in the test-beds are
manifested. Finally, section 4 draws conclusions from the test-bed study.
Page 7 of 102

2 Scenario-driven PLE development and application
This section aims at drafting a methodology for developing and applying PLEs in
educational praxis by bringing together the approaches and results from the former
ROLE deliverables. Thus, the following subsection summarises the vision and goals of
the ROLE project. Then, the theoretical approaches and driving paradigms relevant for
ROLE are highlighted along four different perspectives, and possible questions and
indicators for examining the test-beds are manifested at the end of each subsection.
2.1 Vision and goals
According to the DoW (p.6), the ROLE vision comprises five grand challenges in
learning technologies. First of all, a ‘Responsive Open Learning Environments’ aims at
providing scaffolding structures and personalised learning experiences, i.e. by
adapting resources, services, and tools according to learners and their specific
situations and needs. Second, the adaptive behaviour of ROLEs should be based on a
widely agreed psycho-pedagogical model in order to guarantee pedagogically sound
and meaningful responsiveness. Third, ROLE requires new engineering
methodologies considering development communities, learners, and educational
technologies on a global scale and achieving to build useful real-world environments for
learning. Fourth, the ROLE vision also includes novel evaluation methodologies to
prove that responsive and open learning environments improve individual learning,
particularly when facing transitions between learning contexts like switching between
formal and informal learning, from a job to another job position, from university to a
company, from individual to shared competence development, etc. Finally, ROLE aims
at having impact on large markets and communities, which requires an active
involvement of developers and learners in development and application of ROLEs.
Consequently, the DoW (p.6) defines five objectives to set this vision into practice:
• RO1: support the individual assembly of accessible learning services, tools and
resources in responsive open learning environments (ROLE)
• RO2: research and develop a psycho-pedagogically sound framework for supporting
the individual composition of learning services in ROLE
• RO3: create new engineering methodologies to enable significant contributions to
ROLE from learner and developer communities from outside the project consortium
• RO4: develop and sustain an evaluation methodology to systematically demonstrate
the effectiveness of different ROLE in test-beds focused on the transition of learners
• RO5: exploit and disseminate the ROLE results to wider communities and markets
In summary it can be said that ROLE builds upon the vision of learners who are capable
to design and use their personal learning environments in order to achieve their goals
efficiently and in a pedagogical sound way, being successful in lifelong learning across
institutional boundaries. Due to the large number of possible application areas
Page 8 of 102

(workplace learning, higher and further education, private interests, etc.), the learning
context is supposed to be defined in a very simple, generic way, for instance in the form
of activities or scenarios. Therefore, this deliverable uses the notion of scenarios of
learners being involved into activities in which they collaborate on shared artefacts
together with other actors (peers) and by using a variety of learning tools (cf. the learner
network model behind PLEs). All in all, this section aims at compiling a methodology for
scenario-driven PLE development and application on the basis of the ROLE results
available so far.
This vision behind ROLE is based on many different research topics which have been
elaborated throughout all work packages in the first 18 months of the project and are
well documented in the deliverables (see also http://www.role-project.eu/Deliverables).
With respect to the work package structure of the project (cf. DoW, p.28), the paradigms
driving the development of the ROLE framework can primarily be identified in the work
packages 1 (‘ROLE Framework Analysis’), 5 (‘Testbeds of Learner Driven Service
Composition’), 6 (‘Psycho-Pedagogical Integration Model, Evaluation, and Validation’),
and 7 (‘Learning Community Building and Sustaining’). The following subsections
summarise the most relevant theoretical approaches and key concepts from different
perspectives, namely pedagogy, learning communities, technology, and PLE
development. Furthermore, each of these subsections elaborates how these aspects of
the ROLE vision can be evaluated in the test-beds.
2.2 The psycho-pedagogical perspective (RO2, RO4)
The pedagogical approach of ROLE is being developed in work package 6 in close
collaboration with WP1 and WP2 and based on observations in the test-beds.
The Psycho-Pedagogical Integration Model (PPIM) and self-regulated learning
As a first WP6 result, the deliverable D6.1 (‘Common psycho-pedagogical framework’)
introduces the Psycho-Pedagogical Integration Model (PPIM), i.e. a theoretical model
which aims at supporting learners in learning self-regulated in the context of open
responsive learning environments, which includes different learning phases, such as
selecting appropriate tools and in designing and using their learning environments in a
pedagogically and psychologically sound way. Core components of this model are the
concept of self-regulated learning (SRL), a comprehensive learner model, a model of
learning activities and the underlying skills, as well as guidelines for learner monitoring
and providing recommendations.
Figure 2-1 gives an overview of the iterative SRL process model, in which learners have
a certain degree of freedom but are supported by guidance and personalised learning
experiences on the basis of motivational aspects, collaboration and practice sharing, as
well as meta-cognition and awareness. Applying this process model for learning, D6.1
identifies three kinds of key activities for domain, tool, and self-regulated learning. In
practice, real-world activities include parts or all of these key activities and consist of
four stages (see Figure 2-1): (a) definition of a learner profile, (b) finding and selecting
resources, (c) working with these resources, (d) reflecting learning and the outcomes.
Page 9 of 102

Figure 2-1: Self-regulated learning process model, taken from D6.1
An intelligent learning environment could support learners by monitoring them and
providing recommendations within each of the four phases on the basis of the PPIM
structure (and without considering community data).
Assessment procedures and guidance principles for supporting self-regulation
With regard to the SRL process model, the internal deliverable ID6.1 (‘Draft assessment
procedures and guidance principles for supporting self-regulation’) deals with strategies
to foster self-regulated learning and with modelling and assessing self-regulatory skills.
This deliverable builds upon the notion of skills and differentiates between domainrelated
skills, tool skills, and SRL skills. In further consequence, the SRL skills are
examined closer, and strategies to monitor and assess these skills are elaborated.
Draft common psycho-pedagogical principles for PLEs
On a more technological level, internal deliverable ID6.2 (‘Draft common psychopedagogical
principles in the context of individually compiled learning environments’)
examines psycho-pedagogical approaches in the context of related research projects
and the ROLE test-beds and identifies the following concepts as being relevant for the
ROLE approach: (1) self-regulated learning, (2) guidance and freedom, (3) skill
assessment and monitoring of learning activities, (4) meta-cognition, (5) motivation, (6)
collaboration, (7) personalisation, (8) recommendations, (9) informal and non-formal
learning, (10) lifelong learning, and (11) blended learning. These aspects are
considered to be relevant for individually compiled environments and the test-beds.
Training material collections of the test-bed
Related to WP6, the deliverable D5.3 (‘Test-bed Training Material Collection’) manifests
the requirements on training materials for ROLE and examines the test-beds according
to these requirements and to the existing materials. Concluding D5.3, training materials
for the ROLE approach must be designed for highly heterogeneous groups of learners
and support different pedagogical and didactical strategies (online collaboration, talking
heads, synchronous communication, animations and simulations, game-based learning,
etc.). The materials should also be customizable and available in different media forms.
Page 10 of 102

Psycho-pedagogical evaluation methodology
According to the internal deliverable ID6.3/ID6.4 (‘Draft evaluation methodology from
the technical, usability and psycho-pedagogical perspective including guidelines for the
empirical realisation’) and referring the ROLE objectives RO2, psycho-pedagogical
evaluation focuses on the areas (1) self-regulated learning, (2) guidance and
recommendations, (3) reflection and monitoring, and (4) activity and skill models. As a
detailed method is not given, this deliverable considers psychological tests (i.e.
questionnaires and assessments) as the key instrument for evaluating these four
aspects. The primary target group for psycho-pedagogical evaluation are the learners.
Test-bed validation strategy
For validating the ROLE vision on a psycho-pedagogical level, the following questions
are supposed to be answered by the test-bed responsible and/or selected learners:
• What does learning look like in this test-bed (classroom/blended learning vs. pure
online activities; formal, informal, or non-formal learning)?
• How important is self-regulated learning in the scope of the test-bed activities?
What are the benefits for having self-regulated learning skills in a test-bed?
Name and describe them in brief! Are there any support facilities for SRL or tool
skills provided by a tool or platform? Name and describe the support facilities for
SRL or tool skills provided by a tool or platform in brief! Is it an explicit goal to
foster SRL, or is this a side-aspect of the test-bed activities?
• How much freedom in learning is given (structured course content and predefined
lists of learning materials, strict navigation according to instructional units
of the courses, or goals and tools to find material beyond pre-defined course
content)? Which kinds of guidance are available by a facilitator or through
technological tool?
• How does personalisation of learning look like in the test-bed activities? Are there
any pedagogical recommendations (next instruction, background literature,
peers, or others) either by facilitators or by a system? Describe the pedagogical
recommendations you meant by 'other' and whether they are given by facilitators
or the system)!
• How are learners monitored (technically and by facilitators)? Describe the
technical monitoring process in brief! Describe the monitoring process by
facilitators in brief! Is there technical support for reflecting learning (e.g.
automated analysis of free-text answers, quizzes for self-assessment, indicator
for the learning progress, visualisation of skill mastery, or others)? Name and
describe the technical support for reflecting learning stated in 'other' in brief!
• Which role does collaborative learning play? Do learners benefit from
collaborative situations, and do they get support from peers? How are learning
groups established?
• Are learners being assessed, and is the learning outcome measured? If yes, by
whom are learners assessed? If yes, how is this assessment done? Does
Page 11 of 102

technology support assessment? If yes, describe in brief how technology
supports assessment. How is the assessment outcome used for which
purposes?
• Are activities and skills (domain-specific, tool-specific, SRL-specific) modelled
and assessed? How are domain-specific activities and skills modelled and
assessed? How are tool-specific activities and skills modelled and assessed? Is
there a learner profile in electronic form which includes learning goals (to-be
states) and skill/competence development over time (current states)?
• Who performs the planning for learning and who defines the expected outcome
(learning goals)? Is this done formally and explicitly?
• Which of these five aspects (SRL, guidance, recommendations, monitoring and
assessment, activities and skill profiles / none / other) are planned to be
addressed in the future? Where are the restrictions concerning these points in
the test-bed? Where are the restrictions concerning these points (SRL, guidance,
recommendations, monitoring and assessment, activities and skill profiles) in the
test-bed? Add more information, if available.
• Which kinds of training and learning materials are available in the test-bed? Are
these materials customizable according to individual needs of learners? Describe
how! Is the learning material modelled, in order to be used by services (e.g. to be
recommended)? Describe in brief, how the learning material is modelled in order
to be used by services (e.g. to be recommended).
2.3 The community perspective (RO3, RO4)
In contrary to the PPIM, work package 7 aims at developing a bottom-up approach
based on good practices in communities.
Model and methodology for PLE-based collaboration in learning ecologies
Based on PLE literature and an in-depth analysis of the five test-beds, the deliverable
D7.1/ID7.2 (‘Model and Methodology for PLE-Based Collaboration in Learning
Ecologies’) introduces a theoretical model of PLE-based learning ecologies in which
PLE users (actors) interact with entities like artefacts, media (collections of artefacts),
processes (activities), other agents (peers or computer programs), and communities
(see also left-hand side of Figure 2-2). The learners are involved in different activities in
which they use their learning environments (i.e. specific tools or tool combinations) to
connect to learner networks and collaborate on shared artefacts in order to achieve
personal or group goals (see right-hand side of Figure 2-2).
Page 12 of 102

Figure 2-2: Interaction and activity model for learning ecologies, taken from D7.1/ID7.2
Additionally, D7.1/ID7.2 proposes a user-centred PLE design methodology which
empowers learners to design and use their environments, e.g. through facilities to
mash-up existing learning tools and by applying further concepts, like providing PLE
recommendations or sharing good PLE practices. Overall, the WP7 approach aims at
identifying and strengthening Communities of Practice (CoPs).
Mash-up personal learning environments
As a consequence of the PLE design methodology, the internal deliverable ID7.1 (‘Draft
Prototype of a Mash-up PLE’) describes four concrete prototypes – i.e. two platforms
and two client-sided solutions – developed in the scope of the ROLE consortium and
realising the considerations of PLE-based collaboration partially or fully.
Good practice sharing through activity patterns
In order to supporting learners to use the proposed PLE technology, deliverable D7.2
comes up with an approach towards sharing good PLE practices, i.e. experiences given
from peer actors. This approach builds upon user interaction capturing and facilities for
sharing and retrieving patterns of learning activities (cf. Figure 2-3). Furthermore, D7.2
also argues for generating and providing recommendations for PLE users.
Figure 2-3: Good PLE practice lifecycle, taken from D7.2
Page 13 of 102

Consequently, the deliverable deals with important aspects concerning this practice
sharing approach, like privacy and trust, the dimensions of capturing learner
interactions, competence development issues, etc., and proposes an infrastructure to
enable good PLE practice sharing on different kinds of PLE solutions (platforms and
client-sided prototypes).
Community evaluation methodology
The evaluation of this community-based bottom-up approach can be achieved through
very different aspects, which are manifested in ROLE deliverables or declared as next
steps in the project.
On the one hand, the internal deliverable ID6.3/ID6.4 (‘Draft evaluation methodology
from the technical, usability and psycho-pedagogical perspective including guidelines for
the empirical realisation’) summarise generic methodologies (plus factors and methods)
for evaluating the success of information systems, technology acceptance, and
motivation and Community of Practice (CoP) success. Here, also aspects from
evolutionary biology would be of interest. For instance, the usage of PLEs and single
tools could be examined on basis of a Darwinist model, leading to the evaluation of the
evolvability and fitness for purpose of PLE technologies in CoPs. A workshop to be held
in March 2011 (cf. http://augur.wu.ac.at/EFEPLE11/) will address this special topic.
Additionally, trust and privacy are tackled but not explored in detail. Particularly for
capturing interaction data, it is necessary to consider privacy while applying new PLE
technology and generating recommendations require trustful software.
On the other hand, the community-driven PLE approach by WP7 requires also the
evaluation of typical Web 2.0 issues, like the quality of user-given input (Folksonomies),
the accuracy and novelty of PLE recommendations, or the dynamics and structure of
networked collaboration and the CoPs themselves. Furthermore, the PLE practice
sharing approach documented in the deliverable D7.2 enables the evaluation on the
basis of concrete data available in the pattern repository, e.g. by analysing the number
of activity patterns shared, explicit user feedback on activities and activity outcomes,
etc. Regarding these considerations, work has started in the second year of the ROLE
project. First results have been published e.g. at the Workshop on Recommender
Systems for Technology Enhanced Learning (Manouselis et al., 2010) and the
EFEPLE’11 Workshop (see above).
Test-bed validation strategy
In order to validate the community considerations of ROLE, the following questions are
supposed to be answered by the test-bed responsible and/or selected learners:
• How can learning activities be characterised in the test-bed? Is learning restricted
by the boundaries of an organisation? How is it restricted? Describe the
restrictions given by the boundaries of an organisation! Is learning restricted by
the boundaries of a context? How is it restricted? Describe the boundaries given
by the context! Do learners elaborate the outcomes on their own or do they have
to collaborate with others? Are they involved in passive lecturing activities, or are
Page 14 of 102

they motivated to collaboratively achieve their goals? How are they motivated to
collaboratively achieve their goals?
• Do learners use new media technologies to learn and work together? Which new
media technologies do they use? Are there any recommendations or restrictions
on selecting ICT tools, materials, and peer learners? Describe the restrictions
and recommendations concerning selection of ICT tools, materials, and peer
learners in brief!
• Is there learning software (i.e. a platform) which has to be used by all learners?
Name and describe the learning software (i.e. platform) which has to be used by
all learners! Does the test-bed provide and promote a technical PLE solution by
which learners can integrate and select the learning tools they need for their
everyday activities? Name and describe the technical PLE solution by which
learners can integrate and select the learning tools they need for their everyday
activities, provided and promoted by the test-bed. Are there any restrictions
concerning other tools and platforms available (e.g. Facebook)? Name and
describe the restrictions concerning other tools and platforms available.
Restrictions by whom (the organisation, the learner, or somebody else)?
• Does the test-bed technology capture learner interactions e.g. in terms of the logfiles
of tools, interaction recordings of a PLE solution, or user-given input like
social tags? Which data is available (for analysis)? Which data has already been
analysed and how? Which interactions are captured (activities, artefacts, media
repositories, communities, peer actors or software agents, none, or others)?
• Are there any organisational policies in your test-bed? Is it obligatory to agree to
the organisational policies? Are the privacy policies restricted by a third party
(external partner)? Are the privacy policies ROLE-specific? Do the organisational
policies in the test-bed allow capturing and analysing user data of test-bed
learners? Is it possible to install new PLE software (web-based platforms or client
applications) or use third-party tools within the test-bed? Is it allowed to provide
an API to user data or to transfer this data to external systems? Please describe
advantages/disadvantages of the privacy policies in your test-bed!
• Are interaction recordings analysed or used in some way, e.g. for giving visual
feedback or optimising/adapting learning? Describe, how interaction recordings
are analysed or used, e.g. for giving visual feedback or optimising/adapting
learning. Is there an attempt to generate and provide PLE recommendations for
learners on the basis of the learner interaction recordings? Describe the attempt
to generate and provide PLE recommendations for learners on the basis of the
learner interaction recordings.
• Does the test-bed allow sharing and reusing good practices of PLE-based
activities amongst the learners? If yes, which practices can be shared? How
does practice sharing work? How much effort is it for the learners? Do you
capture the ways of practice sharing? Please provide some numbers concerning
practice sharing (How many test-bed learners did share good practices in the last
3 months? In your opinion, what is the amount of good practices which could be
Page 15 of 102

shared in the last three month? In your opinion, what is the ratio of good
practices shared in comparison to the sharing potential (in %)?)
2.4 The technical perspective (RO1, RO4)
From a technical perspective, research and development is driven by the work
packages 1 to 5 being influenced by WP6 and WP7. The relevant concepts reach from
theoretical models over the notion of the ROLE space to user interface issues.
Existing and applicable theoretical models
The WP2 deliverables (D2.1: ‘Survey on existing Models’, D2.2: ‘Applicability of
Theoretical Models’) examine existing models from the field of technology-enhanced
learning with respect to their applicability for the ROLE approach. In a first sketch, D2.1
identifies the most relevant modelling areas for ROLE, namely learner modelling,
domain and content modelling, competences and skill modelling, and modelling of
learning scenarios. Parts of these models have influenced the WP6 PPIM (skills, learner
profiles, key activities, etc.) and the WP7 approach (PLE-based activities), as shown
before. D2.2 describes concrete TEL models from literature and research projects which
are applicable for ROLE. Moreover, the deliverable outlines that recommendations are
the most important instrument to support the test-bed scenarios. Again, work on
recommendations has been started in the scope of the WP 6 (top-down, model-driven,
pedagogical) and WP7 (bottom-up, data-driven, community-aware).
Learning-related services
In work package 3, the first deliverable D3.1 (‘Survey of learning-related services’)
examined 64 tools and services which can be used for learning. Hereby, the survey was
conducted along five functional categories: (1) communication and collaboration tools,
(2) learning domain or planning services, (3) authoring tools, (4) enabling services, and
(5) LMS and PLE solutions. Classifying tools and services according to these categories
can be useful e.g. for generating recommendations. Moreover, D3.1 proposes to open
up the learning tools, not only to provide data interoperability with other tools and
services but also to be able to use selected features of monolithic systems like LMSes.
Guidelines for the choice of integration technologies
As another outcome of work package 3, D3.2 (‘Guidelines for the choice of integration
technologies’) summarises guidelines for evaluating ROLE-like solutions according to
four perspectives: (1) application type, (2) tool and tool assembly for web-based
environments, (3) protocols and service design, and (4) data representation and
formats. Additionally, the deliverable evaluates concrete technologies and formats for
each of the four section.
ROLE specifications
Deliverable D3.3 (‘Version 1 of the ROLE specifications’) gives an overview of existing
standards and specifications relevant for five aspects of ROLE, namely (1) data
structures (i.e. user data, content, metadata, and intellectual properties and digital
rights), (2) architectures and interfaces (i.e. for repositories, tool interoperability, and
Page 16 of 102

school interoperability), (3) communication protocols (i.e. XMPP or Open Application),
(4) tool and service syndication framework (i.e. widget technologies and repositories,
widget preference server, widget engines, and widget containers), and (5) tracking
services (e.g. being based on the Contextual Attention Metadata (CAM) schema).
As a conclusion, D3.3 presents standards and specifications that are relevant to enable
interoperability and possible synergies between tools and services. A few of them –
such as XMPP for realising a publish/subscribe mechanism or multi-user chats, the
Open Application API for inter-widget communication, or CAM for capturing learner
interactions – seem to be promising for the ROLE interoperability framework.
ROLE space
The notion of the ROLE space is mentioned in various deliverables, mainly from WP4
and WP5. Deliverable D4.1 (‘Personal Learning Service Bundles’) coins the term
“personal space” as a technical solution for managing learning functionality in the form
of widgets thus defining it as a widget container. Consequently, this deliverable
describes the first iteration of the ROLE Personal Learning Service, i.e. a PLE solution
for realising a language learning scenario on the basis of a web-based widget
environment (see Figure 2-4). A set of widgets for a specific scenario (an activity with a
specific purpose) is called “PLE bundle”. This prototype is considered to be a first step
towards building a ROLE PLE solution applicable for other scenarios.
Figure 2-4: ROLE Christmas project PLE with three learning widgets, taken from D4.1
Deliverable D4.2 (‘Service Syndication Platform’) defines the ROLE space as a
“personal learning mash-up environment for end-users” (p.9), i.e. a platform for
integrating and managing existing tools (widgets) for learning scenarios. The ROLE
space relies on the widgets provided by a “widget store” – a platform for end-users to
search services and bundle them on the basis of recommendations –, a “widget
Page 17 of 102

epository” – the community database containing all services and service bundles –,
and a “widget preference server” – a platform component which allows to store service
preferences of users and enables sharing of services and bundles.
In further consequence, the deliverable elaborates the most relevant use cases around
this architecture, related work from literature and case studies, as well as
implementation details. Moreover and in accordance with WP3, D4.2 highlights the
importance of widget and service interoperability for usability enhancements like singlesign-on
or for end-user aspects like combining two widgets with each other, i.e. to
automatically synchronise data between two widgets.
Learner interaction capturing and PLE recommendations
Beside other deliverables in the work packages 1, 2, 3, 6, and 7, WP4 also deals with
the idea of capturing learner interactions and generate recommendations for PLE users
from this data. Therefore, D4.1 reports on the Camera widget which monitors learners
and captures user interactions with the PLE (e.g. the Christmas prototype) in the
Contextualized Attention Metadata (CAM) format. Deliverable D3.3 explains the
architecture behind the Camera widget in detail. On a pragmatic level, D2.2 and ID6.2
propose deriving pedagogical recommendations while ongoing work in WP7 elaborates
how PLE recommendations can be mined from these user-given interaction recordings.
User profiles, authentication, authorisation, single-sign-on
Other issues addressed in various work packages are user (learner) profiles and
aspects of authentication and authorisation. Amongst others, D2.2 and D6.1 give first
details of what a learner profile should contain in order to satisfy the requirements of the
PPIM. D3.3 gives an overview of standards and specification in connection with user
and learner profiles while D3.1 even surveys platforms containing user management
(e.g. LMSes) or authorisation and authentication services (e.g. OpenID and Shibboleth).
D4.2 highlights the importance of a central authentication mechanism and a single-signon
solution in order to not annoy end-users by being in need of and managing various
user accounts.
User interface and usability issues
Concrete user interface and usability issues are primarily addressed in the work
packages which are close to the end-users, namely in WP4 and WP5. D4.1 describes
the Christmas prototype (see Figure 2-4) and thus proposes a concrete user interface
for end-users. Moreover, D4.2 elaborates the most relevant use cases in connection
with the widget-based PLE infrastructure and review existing technology (Appstore,
iGoogle, Netvibes, etc.). The deliverable D5.1/D5.2 (‘Participatory design and
implementation plan of Personal Learning Test-beds’) present PLE like solutions
available or planned within the five ROLE test-beds. All in all, the user interface and
usability issues of a ROLE PLE have being discussed within an own taskforce in the
project. Results are partially published at former conferences (e.g. by Isaksson &
Palmer, 2010), or will be documented in upcoming deliverables.
Page 18 of 102

Technical evaluation methodology
Evaluation of the technical perspective is mainly concerned with the technical feasibility,
the usefulness, and the usability for end-users (learners). Hereby, the success and
usefulness of a ROLE PLE is related to the other perspectives elaborated in the
subsections 2.2 and 2.3 and e.g. described in ID6.3/ID6.4.Furthermore, D3.2 shows
how technology (protocols, standards, etc.) can be reviewed and selected appropriately.
The user-related aspects of PLEs are tackled as part of the development process which
is addressed in the upcoming subsection.
Test-bed validation strategy
The following questions are of interest for validating the test-beds:
• Are there technical tools for creating or visualising pedagogical or community
models? Name and describe the technical for creating or visualising pedagogical
or community models. Who is working on such models? Describe the
role/organisational unit and responsibility of the person, working on such models!
What are the pedagogical or community models used for in the test-bed?
• Which technologies are used for (a) communication and collaboration, (b)
learning domain and learning planning, (c) authoring of training materials, (d)
enabling services (authentication, authorisation, automatic metadata generation,
content management, searching, mashing up and integrating different tools, etc.),
and (e) learning management (LMS) and personalised learning (PLE)?
• What types of applications (client-sided programs, platforms, and hybrids, none,
or others) are utilised in the test-bed? Provide an example or a comment. Which
tool assembly techniques can be identified (webtop and visual mash-ups, linking
and mashing up data, inter-widget communication, social networking and
sharing, widget-based activity flows, none, or others)? Provide an example or a
comment. Which protocols and service designs (i.e. SOAP, REST, XML-RPC,
XMPP, none, or others) are used? Provide an example or a comment. Which
kinds of data representations and formats (IEEE LOM, XML, RDF/OWL, JSON,
none, or others) are identifiable? Provide an example or a comment.
• Describe three concrete sets of learning tools (personal learning service bundles)
which are used for specific learning scenarios within the test-bed! One is
mandatory, two and three are optional.
• In which way have the PLS bundles of the ROLE projects (Christmas, Easter,
Stonehenge) been applied in the test-bed? Summarise experiences given by the
organisation and by (selected) learners!
• Which of the ROLE specifications are used for (a) structuring user and content
data (SCORM, iCalender, unspecified, none, or others), (b) specifying
architectures and interfaces (Apache Shinding, Wookie, GadgetTabML,
OpenSocial Gadgets, OpenAjaxs Metadata, unspecified, none, or others), (c)
communication between systems (MPP, Event API, unknown, unspecified, none,
or others), (d) tool and service syndication (widget technology) (OAI-PMH, SPI,
Pubsubhubub, unspecified, none, or others), (e) capturing learner interactions
Page 19 of 102

(Google Wave, unspecified, none, or others), (f) digital rights management
(consider 'capturing learner interactions', too) (Creative commons, ODRL,
unspecified, none), (g) structuring/describing content artefacts (IEEE LOM,
MPEG7, unspecified, none, or others), and (h) structuring user data (IEEE RCD,
HR-XML, unspecified, none, or others)? Is there or will there be any
standardisation initiatives for one of these areas? Can this data be accessed
through some API? If not, is it planned to open up the repositories in the test-bed
and provide this data via API?
• Does the test-bed provide a ROLE space like solution which allows users to
select and assemble learning tools in a widget-based way? Is it planned to
introduce such a PLE solution by rolling out a widget infrastructure? Is it planned
to introduce such a PLE solution by extending an existing platform in this
direction? Describe the plans for introducing such a PLE solution.
• Are there any approaches towards learner monitoring or user interaction
capturing within the test-bed? If yes, which format for storing these interaction
recordings is used? What is happening to this data (analysis, sharing, generation
of recommendations, offer new possibilities, no usage at the moment.)? Is it
planned to implement this kind of learner monitoring? Please describe your plans
to implement learner monitoring or user interaction capturing! Are there any
privacy restrictions concerning interaction recordings (policies implemented by
the organisation, or by user settings, no privacy restrictions concerning learner
monitoring, or other).
• What do the user profiles (of the learners) look like? Is there some authentication
(verify that someone is who they claim they are) or authorisation (finding out if
one is permitted to have the resource) service used for accessing courses and
content in the test-bed? If there are any other tools, it is necessary to have
additional user accounts? Can learners access and modify their profiles? How
can learners access and modify their profiles? Please describe the process of
accessing and modifying profiles! What is planned regarding the user profiles
and the authentication? Please describe the plans according user profiles and
authentication!
• Apart from user profiles and interaction records, which other sensitive data is
available at the test-bed? Are there any privacy restrictions (laws, policies by the
organisation or settings given by users) concerning these data sets? Have users
the possibility to protect their privacy, e.g. through modifying settings in their user
profiles? Name and describe in brief, how users can protect their privacy in their
user profiles! Are there any approaches for creating awareness of privacy issues
amongst the learners? Name and describe the approaches for creating
awareness of privacy issues amongst the learners!
• How are user interfaces and features of the learning software developed in the
test-bed? Can new technology (software and hardware) be integrated into
existing systems? Are there any user experiences on such integrations?
Describe how new technology (software and hardware) can be integrated into
existing systems! Describe the user experiences on such integrations of new
Page 20 of 102

technology (software and hardware)! Are user feedback and requests considered
appropriately and taken up by developers quickly? Are there any discussions or
evaluations of typical usability issues (efficiency, learnability, subjective
satisfactory, etc.)? Describe the discussions or evaluations of typical usability
issues (efficiency, learnability, subjective satisfactory, etc.)!
2.5 The development perspective (RO1, RO3, RO4)
Particularly driven by the ROLE Objective 3 (‘new engineering methodologies’),
research and development focuses on novel software development approaches
throughout various work packages in the project.
The Social Requirements Engineering Methodology
The deliverable D1.3/D1.4 (‘Functional and non-functional requirements analysis and
specification’) proposes to involve end-users and other stakeholder communities
(developer, researchers) into the requirements engineering and development phases.
Therefore, the social requirements engineering (SRE) process aims at an active
participation of learning communities into requirements engineering, which is based on
the actor-network theory (ANT) and modelled by the i* framework to describe the
interactions and influences between human agents, technologies, and resources (see
also deliverable D7.1/ID7.2).
Overall, this engineering methodology should strengthen communities of practice
(CoPs), i.e. groups of peoples who share a concern of a passion and who interact
regularly to learn how to improve. By applying the SRE methodology, the ROLE testbeds
should be enhanced by concepts and technology developed by researchers and
software engineers in and outside the ROLE project. Consequently, D1.3/D1.4
summarises the requirements gathered by the ROLE work packages, by selected focus
groups, by internal and external-oriented workshops, as well as by questionnaires,
interviews, and other technical tools.
Specifically for the five ROLE test-beds, the deliverable D5.1/D5.2 (‘Participatory design
and implementation plan of Personal Learning Test-beds’) analyses the test-beds and
highlights specific use-cases and requirements. Furthermore, this deliverable proposes
a participatory design approach for developing ROLE solutions which aim at bridging
gaps identified in the test-beds. This SRE like approach is applied within the test-beds
in order to gather relevant requirements and plan implementing solutions.
Combining learning tools to personal learning service (PLS) bundles
From a more technical perspective, D3.1 (‘Survey of learning-related services’)
classifies learning tools and services for educational purposes thus enabling developers
and end-users to select them for specific learning activities. Moreover, D3.2 (‘Guidelines
for the choice of integration technologies’) proposes guidelines for selecting appropriate
integration technologies so that developers can design the learning tools and services
accordingly.
D4.1 (‘Personal Learning Service Bundles’) describes not only concrete bundles of
(test-bed related) services but also gives details on the development process and the
Page 21 of 102

ROLE technical platform (a widget-based approach). This deliverable shows how SRE
is realised in the practice of the ROLE project by outlining three iterations as milestones
to build up a ROLE infrastructure.
Dissemination and exploitation
Another aspect of the SRE methodology deals with involving external stakeholders and
disseminating and exploiting results of the ROLE research and development. Next to
traditional instruments like the ROLE website (D8.1), a media-kit (D8.2), and a
dissemination plan (D8.3), WP8 proposes promotion activities of PLS bundles and
prototypes if the SRE process leads to successful outcomes (see ID8.1). Similarly, WP9
elaborates an exploitation plan (D9.1) and introduces the ROLE Alliance Programme as
an instrument for involving companies to observe and exploit results of the project.
Concepts beyond the SRE methodology
D7.1/ID7.2 proposes a software engineering methodology going beyond SRE and
considering aspects of end-user development and opportunistic design, i.e. shifting real
development tasks to learners in a way that they can handle them. This user-driven PLE
design method is restricted to supporting end-users in formalising their requirements
within a learning activity, selecting appropriate (web-based) tools, and integrating them
into their learning environments. An important preliminary of this user-centred approach
deals with supporting end-users in an adequate way.
Finally, the deliverables D4.2 (‘Service Syndication Platform’) and D7.2 (‘Strategies and
Facilities for Activity Pattern Sharing’) introduce concepts beyond the must-have
requirements which are called “delighters” in D1.3/D1.4 and should motivate end-user
to work with ROLE technology (e.g. a widget store, a preference server or a practice
sharing back-end infrastructure).
Evaluation methodology
In the former subsections, an evolution methodology has been depicted for each
perspective of the ROLE approach. Evaluating the SRE process can be according to
any of these three aspects but also on metrics like successful uptake in the test-beds,
dissemination of research results into other scientific communities, exploitation of ROLE
software by companies, satisfaction of certain stakeholder groups, etc. In the context of
the test-beds, success of the SRE approach can be measured best by evaluating if the
targeted improvements have been reached.
Test-bed validation strategy
The SRE process can be validated for each test-bed through the following questions:
• Which stakeholders (learners, developers, researchers, none, or other) are
involved in developing and providing educational technology in the test-bed? If
there are other stakeholders, describe them. Are requirements claimed by
learners considered and realised? If yes, how are these end-user requirements
gathered? Describe the process and tools in brief! Who is responsible for
implementing them?
Page 22 of 102

• Can external stakeholders (e.g. researchers or open source providers) bring own
ideas or technology into a test-bed? How does this (formal or informal) process
of collaborating with external stakeholders look like?
• Which idea or tool of the ROLE project has been taken up already? What is
planed as future work?
• Can learners integrate external tools (web-applications or third party software)
into the technical infrastructure of a test-bed on their own, without requesting
permissions, only with support from the provider of the technical infrastructure, or
they cannot integrate external tools at all? If yes, are there any restrictions on
external software?
• Does the test-bed include a repository for tools/widgets (like the AppStore) or
one for sharing and providing best (learning/teaching) practices? If yes, describe
the technical solution of the repository!
• How is the success of a technical learning infrastructure measured or evaluated
in the test-bed? Which feedback do learners and teachers (facilitators) give?
Page 23 of 102

3 Test-bed validation study
To evaluate the applicability of the ROLE vision within the test-beds, a study was
conducted in year 2 of the project (M18-M24). The main goal of this study is to validate
(1) which aspects of the ROLE vision have been realised in each test-bed so far, (2)
what is planned within the next year, and (3) where test-beds are restricted regarding
concepts and approaches of the ROLE project. Furthermore, this study also documents
results of former project milestones, namely of the Christmas, the Easter, and the
Stonehenge milestone (cf. iterative software development plan depicted in the
deliverables D1.3/D1.4 and D5.1/D5.2).
3.1 Methodology of the study
The test-bed validation study was implemented in the following way. In a first step, the
deliverables published in the first 18 months were reviewed, and key concepts were
identified and summarised. This literature review led to the scenario-driven PLE
development and application approach consisting of four perspectives and to concrete
questions for each perspective (see section 2). The second phase was realised in the
form of an online survey in which the key questions of the last section were broken
down to concrete questionnaire items (see Appendix A).
The online questionnaire provided with the open source survey tool ‘LimeSurvey’ (cf.
http://www.limesurvey.org/) were filled out by the test-bed partners. The results of the
survey can be found under https://fit-bscw.fit.fraunhofer.de/bscw/bscw.cgi/39069778
(closed project repository; authentication and authorisation required). To get a complete
picture on the situation in each test-bed, partners were interviewed according to
selected findings from the survey (see Appendix B).
The data collected was analysed and summarised in the following way. First, a
description of each test-bed and its goal, one of the learning transitions given in the
ROLE-DoW (p.31), is outlined. Then, the results concerning the four perspectives of the
ROLE vision are elaborated briefly and in this sequence: (a) psycho-pedagogical
perspective, (b) community perspective, (c) technical perspective, and (d) development
perspective. Additionally, concrete learning scenarios from each test-bed are sketched
before the results of the former ROLE milestones (Christmas, Easter, and Stonehenge
project) are summarised and next steps are highlighted.
3.2 OU test-bed: Open, community-oriented but LMS-based and
driven by the organisation
Open University UK (OU) has gathered the interest of a wide audience ranging from
governmental and non-governmental entities interested in promoting continuing
professional development, public and private higher education institutes, academic
teachers, training course designers, graduate and postgraduate students, educational
researchers, and generally anyone interested in informal learning. This test-bed at the
Open University UK is characterised by fostering informal learning activities for which
Page 24 of 102

self-regulated learning (SRL) is considered to be important (formal to informal
learning, F2I). Learners in this test-bed use the customized Moodle platform
“OpenLearn” (http://www.open.ac.uk/openlearn, see also appendix B.1) to learn at their
own pace, keep a learning journal in order to monitor their progress, complete selfassessment
exercises, and discuss with other learners in forums. Typically, they have
full control over their learning processes.
In this context, the psycho-pedagogical approach of ROLE could be an enabler for
SRL, e.g. through adequate guidance mechanisms. So far, SRL is not yet supported by
technology, and it is not an explicit goal but an interesting side-aspect within this testbed.
Next to the LMS solution OpenLearn, learners have the freedom to select goals
and use tools to find materials beyond the pre-defined course content. Hereby, help can
be found at best by peers within a forum. Personalisation of the learning process and
pedagogical recommendations (next instruction, background literature) are not realised
while possible peers are suggested via the OpenLearn platform. Learner monitoring is
implemented through the logging mechanism of the LMS which is also used for the
activity reports provided to learners.
Reflecting on the learning process is mainly supported by two instruments, quizzes for
self-assessment and other indicators for learning (progress, scores). Collaboration
between learners is facilitated (but not enforced) through tools like learning clubs and
FlashMeeting. Particularly thematically oriented learning clubs aim at bringing together
learners from different OpenLearn courses. Although supporting only informal learning,
all learners of the test-bed and their outcomes are being assessed, mainly through
automated quizzes in the OpenLearn platform and for self-reflection reasons.
Furthermore, there is no learner profile (beyond the one in OpenLearn), and the goals
achieved and the learners’ different skills are not measured at all. Generally, learning is
driven by the course creators who also define the expected outcomes in a formal and
explicit way. Training materials are available as user guides and forums.
Regarding the community perspective, the OU test-bed is not restricted to the
boundaries of an organisation or a specific context. Learners have to achieve the
outcomes on their own, are involved into passive lecturing activities, and they can
collaborate with peers. There is no strategy in place for motivating collaboration
amongst learners. The OU test-bed offers various new media technologies to work and
learn together, e.g. through collaboration tools (blogs, forums, FlashMeeting, FlashVlog,
etc.) and sensemaking tools (Compendium or Cohere). These tools and the learning
materials are recommended in a rather static way, i.e. as part of the OpenLearn
courses. Learners are free to work with these or other external tools according to their
needs and preferences. However, the Moodle-based LMS does not support the
integration of all these learning tools.
Capturing learner interactions is more or less restricted to the log files of the OpenLearn
platform and the internal tools utilised in the test-bed. The log files are analysed with
regard to activities and communities of learners (learning clubs, FlashMeeting groups).
Moreover, organisational policies allow capturing and analysing user data, installing new
PLE software (desktop and web-applications), and the usage of third-party tools while
the sensitive data may not be accessed from outside or given away. Generally, the test-
Page 25 of 102

ed has an organisational policy for which an agreement is obligatory. Practice sharing
amongst learners is not fostered and done informally if at all.
From the technical perspective of the ROLE vision, the OU test-bed does not support
the theoretical (psycho-pedagogical and community) models through facilities for
creating or visualising them. On the other hand, communication and collaboration is
supported by various tools like FlashMeeting, FlashVlog, learning journals (blogs),
learning clubs (for users with common interests), and forums. Furthermore,
Compendium is suggested as a tool for creating learning path maps in order to
sequence resources that remix content and activities from OpenLearn and other
sources. Training materials for tools are authored in HTML (provided as Moodle
courses). Participation in forums is another way of providing support to learners.
All learning services are either provided by the core Moodle installation or through plugins.
An integrative PLE infrastructure does not exist. The test-bed allows the usage of
desktop applications (e.g. Compendium) and provides server-sided platforms (e.g.
Moodle or FlashMeeting) as well. However, it is not possible to combine the different
technologies through UI mash-ups, linkage of data, system interoperability, or other
mechanisms. Similarly, protocols and service designs for enabling interoperability and
mash-ups are not used in the test-bed. Addressing data representation types and
formats, resources are described with IEEE LOM and RDF/OWL and can be harvested
according to the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH).
No other standards for structuring data and specifying architectures, interfaces, or
communication between systems are utilised. User profiles are based on the Moodle
functionality (standard profiles). Course content can be accessed without authentication.
For an active participation in the courses users have to log in. This account can be used
for other tools in the test-bed as well. Moreover, OpenLearn builds upon the creative
commons license for digital rights management issues. Learner interactions are logged
by OpenLearn but this data is only used for providing activity reports to learners. Due to
privacy considerations, learners can restrict monitoring and choose what parts of their
profiles are public and if they wish to be contacted for research purposes.
A typical scenario in the test-bed is a sensemaking activity in which learners use
Cohere (web-application) or Compendium (desktop tool) to describe strategies for
studying open educational resources (OER) or to create learning path maps.
Furthermore, tools are used to communicate and collaborate within learning activities.
From the development perspective, the OpenLearn technical team, comprised both
academics and developers, observes the platform and tool usage, analyses user
feedback from evaluations and decides on changes to the learning software
accordingly. Feedback by users comprises new requirements as well as usability
issues. New learning tools are added to OpenLearn from time to time. Requirements
are quite loose; the new tool needs to be associated with learning. External
stakeholders cannot bring in own ideas or technology into the test-bed. Finally, the testbed
has already a repository of tools for learning. Access to these tools is provided in
the OpenLearn platform (http://openlearn.open.ac.uk/course/category.php?id=14). The
tools maintained by the OU are evaluated according to different criteria (utility, usability)
and with state-of-the-art techniques (inspection and test methods, eye-tracking, etc).
Page 26 of 102

Results from the Christmas, the Easter, and the Stonehenge Project
The English language learning scenario was not of use for the OU test-bed, so the
prototypes from the Christmas Project were not utilised. An OpenLearn learning
scenario was produced for the Easter Project, which was then translated into prioritised
development requirements. Furthermore, widgets were embedded and used in Moodle
courses. The Stonehenge Project has provided useful outcomes concerning the support
of self-regulated learning and recommendations.
Next Steps
With respect to the psycho-pedagogical perspective it is planned to support selfregulated
learning, i.e. by guidance facilities, recommendations, as well as monitoring
and assessment mechanisms. To achieve this and apply results from the Stonehenge
Project, the OpenLearn team is evaluating the possibility to extend the existing Moodle
platform by integrating widget technology. Adding widget container functionality to
OpenLearn would allow the integration of widgets into the Moodle courses. In this way,
learners could utilise various developments of the ROLE project, e.g. the CAM-based
learner monitoring, facilities for supporting guidance and self-regulated learning, the
provision of pedagogical and community recommendations, an improvement of the
social interaction between learners, etc. Furthermore, learners could also use third-party
widgets within their learning environment.
Additionally, the OU Linked Data project (http://data.open.ac.uk) aims at opening up and
structuring (user and content) data in a controlled manner and over a public API.
Restrictions concerning the ROLE visions primarily deal with learner monitoring and
user data, as the test-bed has strict policies for protecting the users’ privacy. Amongst
others, learners can decide to exclude themselves from research activities.
Furthermore, widgets and services (plus the user data captured) must be hosted and
controlled by the Open University.
3.3 SJTU test-bed: Flexible, interactive, rich in instructional variety
but highly teacher-driven
The test-bed at the Shanghai Jiao Tong University can be described as a formal
educational setting for adult learners who have a job and want to improve their
competencies via degree education or certificate training (transition between two
companies, C2C). Typical elements are lectures, online courses, formal assessment,
and certification. Courses are primarily held in the form of blended learning consisting of
classroom and online activities. Thus, learning occurs in a formal and an informal way.
From the psycho-pedagogical perspective, self-regulated learning (SRL) is
considered to be a side-aspect of ROLE due to various reasons, like a lack of time for
help-seeking and self-organising activities, clear distributions of roles (teacher,
students), a lack of communication with peers, etc. Thus, SRL is not a concept
supported by educators or by the learning platform but it is practiced by some learners
in a weaker form (e.g. learning diaries) and it is occasionally requested by them.
Generally, learners in the SJTU test-bed have less freedom in learning and follow a
Page 27 of 102

structured course with pre-defined materials. Guidance within the learning platform is
restricted to the pre-given courses, while teachers point to helpful resources and tools.
Personalised learning experiences are mainly realised by the teachers who offer
materials beyond the course content and recommend next instructions, background
literature or other resources (e.g. cultural artefacts like music, helpful tools for
mathematics, etc). Additionally, learner monitoring is done on a technical level (log files
of school LMS and PLE platform) and by teachers (feedback in class and through
homework). Reflection of learning is mainly supported by quizzes for self-assessment.
Collaborative learning is considered to be beneficial, as learners get support from peers.
Thus, teachers try to foster collaboration but face different problematic issues (varying
levels of social competences and skills of students, traditional teacher-centred setting).
In the SJTU test-bed learners as well as learning outcomes are assessed by teachers
and through exams (multiple-choice quizzes and free text answers), whereby the
learning platform is not used for this purpose. Course-related skills and competences
are assessed by teachers manually and rather informally while competences beyond
the course domain are not considered at all. A learner profile is not given in this testbed.
Learning is primarily planned by the administration of the university and the
teachers. The learning outcomes (goals) of the courses are defined by the teachers in a
formal and explicit way. Restrictions of ROLE concepts in the test-bed are given through
the size of classes, a lack of tools and time, differences in the skills of learners, and a
lack of self-directedness of students. Training materials comprise slides, written
homework artefacts, course books, video files, and lecture streams. Materials are not
customizable, not based on a content model, or being analysed through ICT.
With respect to the community perspective, learning is restricted to the organisation,
the university, and to the context of formal education, i.e. learners are supposed to get
an academic degree. Students are learning on their own and in groups but they are also
motivated to collaborate, e.g. by being encouraged to solve exercises together. In the
test-bed, new media technology is applied. Precisely, in some courses mash-up
technology is used. Teachers suggest appropriate learning tools in the form of pre-built
PLEs while there is no focus on recommending peers or artefacts within the learning
activities. The SJTU test-bed utilises two platforms, a learning management system
(LMS) provided by the university and a PLE solution called Liferay and introduced by
teachers (cf. appendix B.2). Various Web 2.0 platforms are blocked by the Chinese
government; other applications are not accessible at the learners’ workplaces.
Learner interactions are captured in the form of log files within the two platforms and
CAM data within Liferay as given by the OpenApp mechanism (see also ROLE
deliverable D3.4: ‘Prototypical implementations’). Thereby, OpenApp provides
information on user interactions through events, i.e. the activity a learner is involved in,
the tool (gadget) that is used, and some details on the event (type, time, etc). This data
is used for analysing simple aspects, like the number of students using the platforms,
and for novel approaches like PLE recommendations, as explained later. From the
organisational perspective, it is possible to collect and analyse user data, to install any
kind of software, to open up the environment, and to store data externally (e.g. in Web
Page 28 of 102

2.0 platforms). Particularly privacy issues are, at this point of time, of minor importance.
Practice sharing amongst learners is not fostered but planned as future work.
On a more technical level, the theoretical models developed in the ROLE project are
not supported in this test-bed. On the other hand, collaboration and communication are
fostered by typical functionality, like chats, forums or message boards, while planning
and running the learning activities is realised within the LMS provided by SJTU (learning
management) and the Liferay portal (personalised learning). Training materials are
created and offered in the form of Powerpoint slides and web-based forms. Basically,
both platforms are provided via web-server and accessible via browser. Liferay supports
visual assembly of different learning tools (gadgets), whereby linking and mashing data
is not possible but interoperability between two gadgets is enabled through the
OpenApp mechanism. Social networking, sharing learning experiences, and creating
widget flows is not realised.
Concerning standardisation, the SJTU test-bed mainly builds upon the standards and
specifications used for the OpenApp mechanism (XMPP for service design, XMPP and
Event API for communication between systems, CAM schema for capturing learner
interactions) and within Liferay (XML and JSON as data formats, Apache Shindig and
OpenSocial for specifying the architecture and user interface, API for accessing some
data in Liferay). Structuring user and content data, tool and service syndication, digital
rights management, and user data and profiles are not standardised. Learner
monitoring and user interaction capturing is realised by ROLE mechanisms, through
OpenApp and in the form of CAM-based data. There are no privacy restrictions in the
test-bed. User profiles are managed by the LMS and include personal information and
the grades. Authentication is necessary but the infrastructure does not support singlesign-on.
The personal user information can be accessed and edited by learners.
One important scenario of the SJTU test-bed is language learning, for which various
web-based tools are provided as a Liferay page by the teacher. Here, learners can
practice a foreign language with the help of translations from an online dictionary,
applying a spell checker, using another gadget to read phrases out loud, listen to audio
or video sequences, or self-record conversations in this language.
From the PLE development perspective, learners, developers, and researchers of the
test-bed are involved into developing and proving educational technology, whereby
development is mainly driven by research staff. New requirements from learners are
normally evaluated and realised if considered valuable. Therefore, taskforce meetings
and interviews with the learners are conducted.
External stakeholders mainly comprise researchers, e.g. partners in the ROLE project.
Here, collaboration and communication is done in a rather informal way through
exchanging emails and joint research, and by setting up a joint proposal and integrate
technology later. As far as being compatible, technology can be integrated into existing
systems. Moreover, also learners can bring in external tools with the support from the
provider of the technical infrastructure. Finally, the test-bed already integrated the ROLE
widget store in a light way, i.e. by offering the widgets at the Liferay pages. The
feedback on the PLE technology is generally positive but the uptake and usage needs
to be increased.
Page 29 of 102

Results from the Christmas, the Easter, and the Stonehenge Project
So far, the test-bed has already taken up the concept of Personal Learning
Environments and introduced the Liferay platform as a technical PLE solution for
language learning. Having a widget container particularly enables the test-bed to
integrate results from the ROLE project, which is the primary aim of the Stonehenge
Project. Besides extending the SJTU widgets according to the CAM event model, the
test-bed managed to integrate various external prototypes in the form of gadgets.
Amongst others, Liferay users can add a gadget (CAM SVN) which collects and stores
the user interactions to a CAM repository. Moreover, another gadget (PLEShare) allows
actively sharing such interaction recordings and provides PLE recommendations
(appropriate activities, peers, actions, or URLs) on the basis of the data shared by
community users. A third gadget suggests pedagogically sound recommendations for
the current situation of a learner. Finally, ROLE partners also provide other functionality,
e.g. two federated search gadgets (Binocs, Objectspot), a FlashMeeting gadget, and
many more (cf. ROLE Widget Store, http://widgetstore.role-demo.de).
Next Steps
Future plans in this test-bed concern a wider uptake of ROLE technology, and if seen as
valuable by students and teachers, a stronger focus on self-regulated learning in order
to strengthen self-organisation and help-seeking skills of learners. For the later purpose,
further functionality from the ROLE project has to be integrated (e.g. guidance), and the
existing features, like learner monitoring, recommendations, assessment, etc, have to
be evaluated and improved. Technically, the focus on PLEs and widget technology
(Liferay) is considered to be a valid and useful strategy but it is planned to improve
standardisation of the user interface, communication between systems, as well as tool
and service syndication.
3.4 RWTH test-bed: Innovative, praxis and outcome-oriented but
driven by teachers and the organisation
The test-bed at the RWTH Aachen comprises a formal educational setting at a
university, in which learning can be characterised by lectures occasionally enriched with
project tasks, i.e. to prepare students for praxis-oriented situations in profession life
(university to company, U2C). The lectures analysed in the test-bed typically apply a
blended learning approach, combining classroom teaching and online activities in a
course. Moreover, learning occurs in a formal, informal, and non-formal way, which
means that learners also have to develop competencies beyond the official curriculum.
With respect to the psycho-pedagogical approach of ROLE, self-regulated learning
(SRL) is an important side-aspect in the test-bed, e.g. for succeeding in the final exam
or in business life. RWTH provides a structured course and a pre-defined list of learning
materials. Furthermore, SRL should be fostered through the Web 2.0 Knowledge Map
(see appendix B.3), a tool to enrich courses with an improved structure and topical,
relevant background information for the project tasks. Personalisation of learning is not
supported in the test-bed. Recommendations on the next instructions or relevant peer
learners are given by facilitators. Monitoring of learners is achieved by teachers (exams)
and technically (log files and recording CAM events within the platform). As a
Page 30 of 102

mechanism to motivate learners and reflect learning, RWTH uses a bonus point system
for the project work. On the other hand, all offers (lecture, course, knowledge map) are
not obligatory, so students have full control over the learning process.
Collaborative learning has been evidenced as an element for improving the learners’
grades, whereby the project groups are created by facilitators and according to former
achievements. Learners as well as the project outcome are assessed through exams
which are corrected by teaching assistants. Assessment, however, is not supported by
ICT. Furthermore, domain, tool and SRL-specific skills are not measured and stored
within a learner profile. Learning is planned by teaching assistants while the goals and
outcomes are given by the professor responsible for a lecture. Learning outcomes are
defined in a formal and explicit way. Training materials consist of lecturer notes, exams
of the last year, Wiki-based content (eClara), and the above-mentioned knowledge map.
At least the Knowledge Map is customizable according to the needs of learners.
Addressing the community perspectives, the test-bed is restricted to higher education
at a university. However students are allowed and even motivated to learn about issues
beyond the own organisation within the project tasks. Learning and the creation of
learning outcomes is done alone and in groups, whereby the project tasks even force
collaborative learning. Within the test-bed, different new media technologies (e.g. Wikis,
search websites, various websites, or the Knowledge Map) are used. Furthermore,
these tools, concrete materials (lecture notes), and the peer learners in the project
groups are suggested by the facilitators. Here, restrictions mainly concern the group
building, as the project members are selected according to pre-knowledge.
As a learning platform RWTH provides the Web 2.0 Knowledge Map and a Wiki system
eClara. Yet, the test-bed does not utilise real PLE technology. Thus, learner interactions
are captured and analysed with the open source tool Piwik and, since summer term
2010, with the CAM widget which stores learner interactions. Further analysis has been
done at a taskforce meeting at RWTH Aachen (October 18, 2010). Organisational
policies allow capturing and analysing anonymous user data. However it is not possible
to bring new client (desktop) applications into the test-bed, while providing new
platforms is not a problem. Additionally, users are not restricted to specific software and
can use any third-party tools. The privacy policy also allows open APIs and giving away
user data which has been de-personalised before. The RWTH test-bed is considering
privacy only from the legal perspective; there are no technical solutions for creating
privacy awareness. Practice sharing is realised in the Knowledge Map tool which
enables learners to generate and share content.
From a technical perspective, theoretical models of the ROLE project are not
supported by tools, although they are applied in the test-bed. For collaboration and
communication purposes, a chat widget and a forum called “Messageboard” is used.
Learning is planned and supported by the before-mentioned Knowledge Map, the
eClara Wiki, and the forum. The learning environment consists of these web-based
tools whereby it is planned to introduce a LMS platform in 2011 in order to add missing
functionality (learning management, personalised learning). Next to the web-based
tools, the learners can use client software installed at the computer pool at RWTH. So
far, the test-bed does not have a real mash-up (widget) solution but integrated a widget
Page 31 of 102

container solution into the existing infrastructure. Users can link and mash data in the
Knowledge Map and the Wiki. Furthermore, inter-system interoperability and activity
flows are supported by these tools.
Besides the first work on capturing learner interactions in the form of CAM data, no
other protocols or specifications of ROLE are used. So, this test-bed relies on XMPP
and PMRPC (service design), XML and JSON (data representation and formats), and
XMPP and Event API (communication between systems). No standardisation has been
or will be addressed concerning structuring user and content data, digital rights
management, the description of learning resources, and the management of user data
and profiles. Moreover, widget technology has been brought into the test-bed. User
interactions are recorded in the form of CAM data and, on a lower level, as log files.
This data is analysed but not used for more sophisticated features like
recommendations or practice sharing. User profiles are stored on a LDAP server in a
minimalistic variant (unique identifier and data necessary for authentication). The
identifier and authentication is used by all tools but learners cannot modify their profiles.
Privacy restrictions are only given by law.
The primary scenario for using learning tools deals with project tasks within a lecture.
Here, students can use the tools provided within the test-bed to collaboratively work on
a small project in which theoretical issues of the lecture have to be applied in a practical
setting. Typically, the Web 2.0 Knowledge Map application provides the content of the
lecture in a structured way and gives background information on the topics.
Furthermore, learners can generate in the Wiki and share new content through the
Knowledge Map.
Finally and regarding the development perspective, new technical learning solutions
are realised by researchers and developers at RWTH. For instance, the Web 2.0
Knowledge Map is an outcome of a former project called WeKnow. Thereby, learners
are involved into the development process in terms of requirements gathering (surveys,
learner feedback) and user tests (utility, usability). Due to collaborative projects by
RWTH, external stakeholders in the development process of learning technology mainly
comprise researchers from other organisations who can bring in new concepts and
prototypes if being considered useful by the test-bed responsible.
New technology can be integrated on behalf of the test-bed responsible, as already
shown with various approaches (widgetisation of the Knowledge Map, integration of a
widget container and two ROLE gadgets, etc). Learners, however, cannot bring in new
tools into the learning environment. Beside the tools given in the test-bed, RWTH does
not have an own tool repository. The success of technical learning solutions is
evaluated through surveys at the end of each semester. The feedback on the existing
tools is mainly positive.
Results from the Christmas, the Easter, and the Stonehenge Project
In the Christmas Project, the chat and the CAM widget were firstly developed. However,
these tools have not been used within the test-bed, as there was no lecture at this time.
The Easter Project led to an implementation and introduction of a widget container for
the RWTH learning platform. By using this container the three before-mentioned
Page 32 of 102

gadgets were shown to the learners, and feedback on this solution was gathered.
Finally and after planning further development, the Stonehenge Project was dedicated
to develop a first version of the new Web 2.0 Knowledge Map, a PLE like solution based
on a widget container and having integrated the chat and the CAM gadget per default.
In this way, learner interaction recording is realised within the test-bed. These results
have not been evaluated yet due to the fact that there is no lecture in the winter term.
Next Steps
In the RWTH test-bed is open for proposals of ROLE-partners to address psychopedagogical
concepts developed in the ROLE project although SRL and
recommendations are considered to be valuable and worth a closer investigation. Thus,
the test-bed responsibles want to evaluate ROLE approaches and prototypes towards
their usefulness and applicability. Although the ideas and implementations, like
recommendations or practice sharing, seem to be interesting the uptake will start if they
are ready for a productive system.
In 2011 it is planned to introduce the learning platform L2P which is based on IMC CLIX
and should provide necessary learning management functionality and serve as a
personal learning environment at the same time. Concerning the latter issue the focus is
set to widget technologies which would foster also standardisation efforts, e.g. of the
user interface or tool interoperability.
3.5 FESTO test-bed: Web-based, workplace and praxis-oriented but
teacher-driven, hardly collaborative and not personalised
The FESTO test-bed aims at transferring good practices from experienced knowledge
workers to those who are seeking a job or trying to acquire another position within a
company (job opportunity in a company, J2J) and, subsequently, try to increase or
sustain their employability in a rapidly changing working environment. Thus, the testbed
primarily deals with formal education delivered through classroom, blended, and
online learning experiences.
From a psycho-pedagogical perspective, self-regulated learning (SRL) is considered
to be an important side-aspect of the test-bed, as learners are responsible for their
learning activities and have to select and elaborate the contents on their own. Facilities
to support SRL comprise course contents structured according to company-related
educational topics and catalogue services for learners. Moreover, freedom in learning is
restricted to navigating the catalogue and the courses. In this context, guidance is given
by a content sequencing mechanism (suggesting next instruction if available) and,
rarely, through tutor support. Besides the sequencing feature, no more personalisation
and recommendations features are given in the test-bed. Due to privacy restrictions,
automated learner monitoring is only realised in FESTO offices outside of Germany.
Reflection of learning is implemented through self-assessment quizzes in FESTO’s
learning management system (see appendix B.4), the Virtual Academy (VA), as well as
various learning progress indicators.
Collaboration is of minor importance in the FESTO test-bed and primarily part of
blended learning activities. Furthermore, the learning outcomes are assessed through
Page 33 of 102

computerised quizzes. Skills and competence development are not considered if not
explicitly related to the course curriculum. Overall, the learning goals and achievements
are not measured by the system but are visible for the learners themselves. The user
profiles contain only basic data of the learners but do not capture information on
learning. Planning learning and defining the outcomes is done by employees together
with the responsible line managers in the yearly employee talks. Concerning training
materials, the test-bed provides web-based trainings, videos, presentations, rapid e-
learning, tests, and simulations. However, these materials are not customizable, and the
preparation for learning is hardly supported by adequate services and tools. Restrictions
on ROLE-specific, psycho-pedagogical concepts are given through the workload of
employees and privacy policies of FESTO.
Concerning community aspects of the ROLE vision, learning is restricted to the
organisational context of the test-bed in the way that the content is given by the internal
business unit that is responsible for a course. Additionally the test-bed is strongly
focussing on web-based trainings and courses which include interactive elements like
simulations or quizzes. Learners are normally working alone on the outcomes while
collaborative activities are rare. Collaboration per se is not motivated by facilitators. New
media platforms and technologies (Facebook, Twitter, Mash-ups, etc.), are not utilised
at all. Thus, the VA platform also provides no recommendations on selecting ICT-based
tools, materials or peer learners in certain situations (except the before-mentioned
content sequencing). On the other hand, only software approved by FESTO can be
used within the test-bed.
In principle, learners in this test-bed have to use the FESTO VA platform which is based
on IMC’s CLIX Learning Management System. Yet, this software does not enable
learners to integrate external tools into their environment. Moreover, certain web-based
platforms, like Facebook, are blocked within the company. Concerning learner
interaction capturing, FESTO cannot evaluate this issue since the system only logs
error messages for support reasons. It is also not an explicit goal to realise such a
strategy. Consequently, hardly any approaches on analysing or exploiting interaction
data can be identified. At best, learners get feedback on exams and the final marks. A
strong reason for the low priority of interaction capturing is given by organisational
policies which allow neither collecting user data nor accessing it from outside or storing
it to an external repository. Generally, all operations on user data have to be approved
by the data protection officer. Additionally, the installation of third-party software is
controlled by FESTO. Learners have to agree to these organisational policies.
From a technical perspective, test-bed users have no tools to use or visualise the
models developed in ROLE. Communication and collaboration is supported as part of
the VA platform, by a component called Forum. Management and planning of learning
activity is also a core functionality of the LMS-based platform of FESTO. For content
creation, various tools like the EasyProf authoring tool, Flash Captivate or Lecturinity
are provided. Overall, the FESTO test-bed fully builds upon the IMC CLIX based LMS
solution for all other services (authorisation, authentication, searching, etc), and further
client-sided software is not available. Younger concepts like mash-ups, linked data,
inter-widget communication, social networking and sharing, etc. are technically not
realised at the moment. The application of standards and specifications is restricted to
Page 34 of 102

the IMC CLIX platform supports, which mainly comprises aspects of describing learning
content (AICC, SCORM, QTU) and a few standards for special functions (SMTP for
sending emails, LDAP for authentication, etc).
Learner interactions are not captured by means of full log files. Only error messages are
available, and this data is not used for further purposes. In general, standardisation is
not of high importance in the test-bed. Furthermore, any step towards exploiting usage
data is problematic due to the privacy policy of the organisation. Privacy per se is fully
driven by FESTO. Users must agree to the policy and have no control over privacy
settings. In order to increase privacy awareness, FESTO offers a course on data
protection in the test-bed. The learner profiles contain only a basic set of data about a
user, like the name, the email address, the department, etc, whereby only a few data
fields can be modified by the users themselves. Users have to authenticate at the
system to access courses and content. Authentication is realised by a LDAP server. No
further accounts are necessary within the test-bed.
Three important application scenarios can be identified in the test-bed. First of all, pure
online activities comprise primarily web-based trainings which can include passive
materials, like readings or web-based content, as well as interactive or multimedia
content, like simulations or videos. Secondly, blended learning activities aim at
combining classroom lecturing and online activities. Finally, quizzes are of importance
for grading but also for formative assessment methods, i.e. to enable self-reflection of
the learning process.
The development perspective is characterised by one technology partner, IMC, that is
planning and implementing new features of the platform and research partners, e.g. of
the ROLE consortium, that are involved into identifying new requirements through
taskforce activities. In principle, learners, developers (from FESTO and IMC), and
researchers are involved. New technology can only be integrated after being approved
by FESTO. Users are not allowed to integrate own tools. Feedback and requests of the
test-bed learners are considered after being reviewed and approved by test-bed
responsibles. The usability of the platforms is evaluated regularly, which is conducted
by questionnaires (‘happiness sheets’) and has a strong focus on the learning materials.
New features are implemented by the VA administration in collaboration with the
technology partner. External stakeholders in the development process mainly comprise
the above-mentioned research and technology partners. The test-bed also does not
provide a widget store or a practice sharing repository.
Results from the Christmas, the Easter, and the Stonehenge Project
The result of the Christmas project was a ROLE PLE prototype based on a dashboard
with interoperable widgets. For FESTO these results were a first showcase on how a
PLE solution could look like and which functionalities it offers. Hereby, the main goal is
to improve the existing learning management system (LMS Clix) by terms of
personalisation and individualisation for the learners. The outcomes of the Christmas
project (for example the Language Resource Browser widget, the translator widget and
the vocabulary widget) showed what is possible regarding widget-interfaces and
communication between widgets and the platform. The testing and the analysis of this
Page 35 of 102

prototype especially concerning the business requirements of FESTO have been
implemented in the planning of the next steps of the test-bed within ROLE.
The Easter project grouped the test-beds into three bundles corresponding to three
kinds of users, LMS users, higher-education students and life-long learners. The
FESTO test-bed focused on LMSes and especially on LMS users in a company. This
target group has special needs which hardly can be supported in workplace settings
due to different problematic aspects. Learners in business environments primarily have
to fulfil their job-role, and learning is mostly to support them in doing so. Due to high
workloads it is often hard to learn on the job. Business learners are a highly
heterogeneous target group with big age differences (from 16 to 65), different
qualifications, pre-knowledge, job-roles, learning requirements, preferences and goals.
Derived from this spectrum of learners, FESTO created a list of widgets that are
beneficial for the target group of the test-bed. Then, this list has been converted into a
roadmap (subdivided in short-term planned, medium planned and long term planned
widgets) together with the developers of IMC. The first widget from this list has been
realised within the Stonehenge project. It is a federated search widget that allows
retrieval of learning content from internal and external databases and repositories. This
widget has been integrated into the live system of the test-bed.
Next Steps
In order to provide a more open and responsive learning environment, the general
strategy is to follow an incremental improvement of the LMS platform in the direction of
a personalised LMS instead of heading for a radical change of technology. Therefore, it
is planned to extend the existing platform into the direction of a widget infrastructure,
thus leading to a hybrid between a LMS and a PLE (the vision of a PLMS). Additionally,
further concepts of the ROLE project, such as fostering self-regulated learning, PLE
recommendations, activity and skill profiles, etc, are being investigated for their
relevance and applicability in the test-bed. Single aspects could be realised through a
broad variety of open social widgets developed within or brought in by the ROLE
project. Most probably, recommendations for learners are of interest. However such an
approach might be problematic due to the privacy policy on user data, which has to be
clarified by the data protection officer.
3.6 BILD: A gateway to many test-beds
The BILD test-bed focuses on the development of professional competencies beyond
isolated, organisational-driven learning (one to many, O2M), i.e. through providing rich
learning experiences, which includes so-called transcompetences, i.e. tool skills and
social competencies. Different to the other ROLE test-beds, the British Institute of
Learning Disabilities (BILD) does not provide specific courses or educational content but
specialised services for their over 2000 members. Thus, BILD cannot be characterised
as a test-bed but rather as a gateway to many potential test-beds which can be
enriched by aspects of and results from the ROLE project. Basically, BILD can be seen
as an organisation that could transfer research results of ROLE into real-world
application scenarios, for instance by applying the test-bed validation methodology
elaborated in this deliverable.
Page 36 of 102

In order to show how this could work in praxis, this section evaluates one of these
potential BILD test-beds according to the ROLE vision. Fort this purpose, the BILD
member Etestme (http://www.etestme.co.uk/, cf. appendix B-5) was chosen for the
study. A typical scenario for the development of professional competencies is supported
by the English language learning service “E-Testme”, an online system for practising the
IELTS examination (http://www.ielts.org/) by consuming related materials and
submitting exams, i.e. voice recordings or written texts, to the platform. After the
outcome has been assessed by language professionals feedback is given to the user.
With respect to the psycho-pedagogical approach of ROLE, self-regulated learning
(SRL) is considered to be relevant but not of primary importance. SRL is hardly
supported by the platform. At the very best learners get feedback on their outcomes. In
this test-bed learners are free to navigate the relevant online courses which consists of
goals, course-related content and tools to find and access the materials. Guidance is
realised within the Etestme platform while personalisation of the learning process is not
considered at all. Additionally, the platform does not provide any recommendations or
background literature beyond the pre-defined course materials.
Learner monitoring and supporting self-reflection is restricted to assessment (quizzes
for self-assessment and examinations) and grading. Collaboration is considered
important but insufficiently supported in the BILD test-bed. Particularly, a lot of effort is
put into the assessment of learners and the measurement of learning outcomes, both
conducted via email by language professionals. Activities and skills of learners are not
modelled or captured. Furthermore, the test-bed includes no learner profiles in
electronic form. All training materials are given by IELTS while learners are responsible
for learning and achieving the goals.
From the community perspective, learning in the BILD test-bed is not restricted by
organisational boundaries but to the context of informal learning through e-learning and
assessment. Learners are working alone on the outcomes but can consult peers or
Etestme staff via forum. Collaboration is not necessary at all. Beside the forum, no other
new media technology and Web 2.0 platforms are utilised in this test-bed scenario.
Additionally, recommendations are restricted to pre-given course content while it is left
to the users which tools they use and with which peers they collaborate. The test-bed
neither provides or promotes a PLE solution nor can learners bring in and integrate own
tools. Outside the Etestme platform there are also no restrictions concerning ICT.
Etestme is using learner interactions for improving the learning process, for instance by
applying voice recordings to support reflection and grading of oral language tests. Yet,
analysis and the derivation of improvements is done by human experts and not
supported by ICT-based tools. Additionally, no other interaction data, not even log-files,
is exploited for analysis or related features (reflection, recommendations).
Organisational policies concern privacy and aim at preserving the anonymity of learners
and language professionals behind the Etestme service. Within this test-bed it is
possible to use third-party software and, to a small extend, to have data in external
systems, e.g. a system of the learners’ organisation. Finally, practice sharing among
learners is also not realised or supported in the test-bed.
Page 37 of 102

On a technical level, hardly any of the developments of the ROLE project are applied
in this exemplary BILD test-bed. For communication and collaboration, a forum is used.
Curriculum planning and learning content authoring is not necessary due to the given
training materials. Even management and personalisation of the learning process is
redundant within the Etestme platform because of the simplicity of the learning scenario.
Furthermore, the test-bed neither provides mash-up platforms nor the possibility to link
data (user and content data) from different sources. Consequently, interoperability
issues like inter-widget communication are also not supported.
Besides the IELTS standards and basic web-based technology (HTML), standardisation
efforts in any of the ROLE-relevant areas (protocols and service designs, data
representation, content and user data, architectures and interface, etc) are not
recognisable. Moreover, it is not planned to open repositories or provide APIs to access
data due to privacy reasons. Here restrictions mainly concern the preservation of the
users’ anonymity. Currently, the test-bed does not provide a ROLE space like
infrastructure.
In this test-bed, software development per se is triggered by Etestme, whereby the
company decides about stakeholders to be involved in development process. Thus, it is
not possible to determine if feature requests by learners are considered or realised.
Similarly, Etestme has to decide if ideas and technology from outside should be taken
up within the environment and how collaboration with these external stakeholders
should look like. All other issues, like utility and usability evaluation or tool and practice
sharing repositories, are dependent on the strategy of the test-bed responsible.
Results from the Christmas, the Easter, and the Stonehenge Project
The BILD and its members have been part of several dissemination activities relating to
the progress of the ROLE project. These have included events arranged specifically for
BILD members and other major industry events such as World of Learning
(http://www.learnevents.com/) and Online Educa Berlin (http://www.online-educa.com/).
Hereby, it was demonstrated what the ROLE project hopes to achieve overall as well as
specific reports from both the Christmas and Easter projects. This has taken place
through face to face BILD events and major industry conferences. BILD members have
also given their views relating to the aims and purpose of the project and this has been
fed back to ROLE project leaders. Some specific technologies relating to ROLE such as
Flashmeeting have also been tested during activities such as online meetings.
Throughout activities relating to the Christmas, Easter and Stonehenge projects, the
BILD has provided learning scenario examples from its members as well as
demonstrated and disseminated ROLE results to BILD members and other industry
professionals. Results were collected in the form of ‘sound bytes’ (voice recordings),
and these were fed back to ROLE, i.e. made available on the BSCW site. The BILD also
received more informal feedback which has also been communicated to the project
members in meetings.
Particularly relevant for ROLE was the BILD member, Etestme. Etestme developed an
online system by which English language learners can practise for their IELTS
examination. There are four parts to the test; listening, reading, speaking and writing.
Page 38 of 102

For evaluation in ROLE, Etestme produced a widget sized version of their online
language assessment system, namely an ‘E-Testme results thermometer’ (see
appendix B.5). A learner is able to input their email address and visualise their IELTS
practice progress - the thermometer rises according to their best ever results score.
This widget is designed to be used in conjunction with other supportive widgets to
enhance the Etestme learner experience. The gadget has been made available in
iGoogle and will be embedded into the Etestme results page and email feedback.
Following on from the Easter project work with Etestme, the BILD arranged a taskforce
meeting on October 19 th in Bristol, UK with other ROLE partners and Etestme to
observe the learning system in full and discuss and evaluate how ROLE could be
extended and tested through this scenario. The main benefit was seen in overcoming
the shortcoming of the current scenario that the learner is currently working in isolation
and not necessarily as part of a group. They do not have access to an Etestme tutor
except for marking and feedback if it is requested. Quite often the learner does not
access the forum for advice and general collaboration with other students.
The benefit ROLE could bring would be to link it to learner profiles, goal setting, reading
materials, multi-media content as well as other learners. In essence, Etestme could be
used as one widget within a supportive bundle, for instance as evaluated in the SJTU
test-bed (cf. subsection 3.3). Regarding dissemination, the Etestme system could push
individuals towards ROLE. This could be done at the end of the test via email. This push
could direct them towards a pre-made bundle of widgets to assist them further in their
learning. Also, at the point of sale, Etestme could try and make the customers use a
gadget sized version of the system. This gadget could be linked to a bundle of other
related gadgets. Etestme felt that a widget can help to establish partnerships with
learning material producers, by providing their measurement tool as a service.
Next Steps
The BILD will ensure that Etestme features the scoring widget, which was developed as
part of the Stonehenge Project, in the results and feedback information which is sent to
their customers. Etestme wishes to test the widget in ROLE Space with the guidance of
the ROLE developers. Etestme is also keen to promote bundles of widgets through their
learning systems and email communications with customers.
The BILD feels it would be beneficial to make members aware of the ROLE results and
the work completed for the Stonehenge Project. More business related scenarios, like
the FESTO test-bed (see subsection 3.5), may initiate a greater understanding of ROLE
by the BILD members. It is hoped that showing the members the work completed in the
ROLE project, for instance by analysing a potential test-bed in the way described in this
deliverable, will encourage more membership participation in ROLE and inspire other
BILD members to provide scenarios for the next stages of the project.
Overall, the BILD as a provider of educational services could play a key role in
exploiting ROLE results in real-world educational settings. Even more, BILD could apply
ROLE technologies (e.g. a pattern repository or widget store) to support this exploitation
strategy as well as to offer new services (practice sharing, recommendations or
educational widgets) to their members.
Page 39 of 102

4 Concluding remarks
Validating the test-beds according to the questions elaborated in section 2, has led to
some interesting findings concerning PLE development and application in the praxis of
educational settings.
First and foremost, test-beds can be categorised into educational and corporate
settings. While in both settings learning seems to be driven by the organisation, i.e. by
using LMS technology, universities and higher education institutes tend to be more
open regarding PLE-related concepts. For instance, the OU was already investigating
community approaches and collaborative learning before the start of the ROLE project,
and is now intensifying the work in this area. Besides, the OU is also opening up their
data repositories through the Linked Data project. SJTU, moreover, introduced a PLElike
solution via the Liferay platform without facing a lot of barriers or problematic
aspects. Particularly, language learning could be enriched by interactive and helpful
tools in this way. RWTH uses PLE technology to support the transition to corporate
learning and realises more realistically, praxis-oriented learning experiences.
On the other hand, corporate settings underlie restrictions, like organisational policies,
so that realising certain aspects of the ROLE vision might require more efforts. Although
offering interactive and praxis-oriented content to their users, learning in the FESTO
test-bed is not personalised, mainly driven by teachers and being based on pre-defined
materials. Bringing in other tools is not very easy due to organisational rules while the
exploitation of user data is prohibited due to privacy and intellectual property issues.
The BILD test-bed is particularly challenging, as the organisation does not have real
learning scenarios but offers services to their members. However, the case of the BILD
indicates clear benefits for exploiting ROLE solutions. As a gateway to many potential
test-beds, BILD can validate the test-beds of their members according to PLE
development and application. Furthermore, BILD can even provide specialised services,
like practice sharing repositories or a learning widget store, to them.
Secondly, the validation of the test-bed scenarios showed that each (existing and
potential) test-bed has its own characteristics and underlies preliminaries, so that it
requires a customized strategy for introducing PLE technology. While it was feasible to
provide a PLE solution next to other platforms at SJTU and RWTH, the other test-beds
reported that such a radical approach is not feasible. Hereby, some responsibles asked
for a stepwise realisation of ROLE ideas, i.e. by extending LMSes with a widget
container or integrating learner monitoring and recommender components into existing
platforms. The PLE validation methodology presented in this deliverable could be useful
to shape a roadmap to introduce customized PLE implementations in test-beds.
Thirdly, self-regulated learning (SRL) is generally considered important by all test-beds.
In the cases of the OU, RWTH and FESTO, SRL is of high priority as the learners of
these test-beds are responsible for their learning activities. Nevertheless, none of the
test-beds reported any technology-assisted SRL taking place, which provides an
excellent opportunity for supporting SRL through ROLE technological solutions. With
Page 40 of 102

egards to building and sustaining communities of learners, this is mostly restricted by
the organisational context of each test-bed, either in higher education or business.
However, collaboration tools are employed in many cases and social interactions are
supported to a certain degree.
Finally, the test-bed studies evidenced that certain aspects, like privacy and trust, are
essential for user-centred approaches like PLEs but have not been addressed properly
within ROLE. Here, work has been started in the second year of the project and is
documented in the M24 deliverable D3.4 (‘Prototypical implementations’).
Page 41 of 102

Appendix A: Questionnaire for the Test-bed Survey
The following questionnaire was used for the test-bed survey. Results of this survey can
be found under the URL https://fit-bscw.fit.fraunhofer.de/bscw/bscw.cgi/39069778
(closed project repository; authentication and authorisation required).
A.1 ROLE - Test-bed validation, community-perspective
Test-bed validation strategy
In order to validate the community considerations of ROLE, the following questions are
supposed to be answered by the test-bed responsible and/or selected learners
Welcome
There are 40 questions in this survey
Demographic information
Provide data about you, your test-bed and your partners
1 Which test-bed are you evaluating?
Please choose only one of the following:
Festo Lernzentrum Test-bed (Internal Job Opportunity in a Company)
British Institute for Learning and Development test-bed (Continuing Education for
Transition between two Jobs)
OU test-bed by the Open University UK (Continuing Education for Transition between
two Jobs)
RWTH Aachen University Course “Web 2.0 Knowledge Map for academic teaching”
(University to Company)
Jiao Tong University Test-bed (Continuing Education for Transition between two Jobs)
Other
2 Information about the test-bed.
Please write your answer(s) here:
Test-bed is active since (please enter the year/month/day in the format YYYYMMDD
Number of registered test-bed learners after 3 months
Number of registered test-bed learners today
Page 42 of 102

Number of test-bed learners concurrently online (daily average)
Number of test-bed learners concurrently online (peak-time)
Peak-time (please enter hh.mm)
Number of test-bed learners concurrently online (weekly average)
Maximum of test-bed learners concurrently online
If there are no numbers available, please enter -1
3 Information about you
Please write your answer(s) here:
Your Name
Your responsibility/role in the project
4 Choose the institutional partner you are working for
Please choose only one of the following:
FHG - Fraunhofer FIT
RWTH Aachen - RWTH Aachen University
TUG - Technical University of Graz
K.U.LEUVEN - Katholieke Universiteit Leuven
UNI KO-LD - University of Koblenz
UU - Uppsala University
Page 43 of 102

EPFL - École Polytechnique Fédérale de Lausanne
ULEIC - University of Leicester
OU - Open University UK
WU - Vienna University of Economics & Business
FESTO - Festo Lernzentrum Saar GmbH
IMC - imc AG
BILD - British Institute for Learning and Development
SJTU - Shanghai Jiao Tong University, China
ZSI - Zentrum für Soziale Innovation
UIL - U&I Learning
Other
Learning activites
How can learning activities be characterised in the test-bed? Is learning restricted by the
boundaries of an organisation or a context? Do learners elaborate the outcomes on their own or
do they have to collaborate with others? Are they involved in passive lecturing activities, or are
they motivated to collaboratively achieve their goals?
5 How can learning activities be characterised in the test-bed
Learning activities - examples:
• formal learning (classes, courses, lectures, assessment, certification)
• informal learning (semi-structured activities at home or at work, occasionally)
• non-formal learning (learning in a formal setting but without typical structures and
methods; e.g. learning in communities, online courses, workshops, etc.)
Please write your answer here:
6 Is learning restricted by the boundaries of an organisation?
Example - boundaries of an organisation:
• learning is restricted to one organisation, i.e.
o all materials and courses are provided by and about this organisation;
Page 44 of 102

o external materials have to be verified by an organisation
Please choose only one of the following:
Yes
No
7 How is it restricted? Describe the restrictions given by the boundaries of an
organisation!
Please write your answer here:
8 Is learning restricted by the boundaries of a context?
Example - boundaries of a context:
• learners are mainly involved into formal, informal or non-formal activities
Please choose only one of the following:
Yes
No
9 How is it restricted? Describe the boundaries given by the context!
Please write your answer here:
10 Do learners elaborate the outcomes on their own or do they have to
collaborate with others?
Please choose the appropriate response for each item:
Yes
No
Do learners elaborate the outcomes on their own?
Do learners have to collaborate with others?
11 Are they involved in passive lecturing activities, or are they motivated to
collaboratively achieve their goals?
Please choose the appropriate response for each item:
Yes
No
Are they involved in passive lecturing activities?
Page 45 of 102

Yes
No
Are they motivated to collaboratively achieve their goals?
12 How are they motivated to collaboratively achieve their goals?
Please write your answer here:
New Media Technologies
Do learners use new media technologies to learn and work together? Are there any
recommendations or restrictions on selecting ICT tools, materials, and peer learners?
13 Do learners use new media technologies to learn and work together?
Please choose the appropriate response for each item:
Yes
No
Do learners use new media technologies to learn and work together?
14 Which new media technologies do they use?
Please choose all that apply:
Facebook
Twitter
Picasa
RSS Feader
ICal
Wiki
Blogs
MashUps
StumbleUpon
Delicious
Digg
YouTube
MySpace
StudiVZ
Page 46 of 102

Citeulike
Google
Diigo
None
Other:
15 Are there any recommendations or restrictions on selecting ICT tools,
materials, and peer learners?
Please choose all that apply and provide a comment:
Recommendations on selecting ICT tools
Recommendations on selecting materials
Recommendations on selecting peer learners
No recommendations at all
Restrictions on selecting ICT tools
Restrictions on selecting materials
Restrictions on selecting peer learners
No restrictions at all
Other:
Page 47 of 102

You can add more information using the field to the right.
16 Describe the restrictions and recommendations concerning selection of ICT
tools, materials, and peer learners in brief!
Please write your answer here:
Learning Software
Is there learning software (i.e. a platform) which has to be used by all learners? Does the testbed
provide and promote a technical PLE solution by which learners can integrate and select
the learning tools they need for their everyday activities? Are there any restrictions concerning
other tools and platforms available (e.g. Facebook)?
17 Is there learning software (i.e. a platform) which has to be used by all
learners?
Please choose only one of the following:
Yes
No
18 Name and describe the learning software (i.e. platform) which has to be used
by all learners!
Please write your answer here:
19 Does the test-bed provide and promote a technical PLE solution by which
learners can integrate and select the learning tools they need for their everyday
activities?
Please choose only one of the following:
Yes
No
20 Name and describe the technical PLE solution by which learners can integrate
and select the learning tools they need for their everyday activities, provided and
promoted by the test-bed.
Please write your answer here:
Page 48 of 102

21 Are there any restrictions concerning other tools and platforms available (e.g.
Facebook)?
Please choose only one of the following:
• Yes
• No
22 Name and describe the restrictions concerning other tools and platforms
available.
Please write your answer here:
23 Restrictions by whom? (The organisation, the learner, somebody else ..)
Please write your answer here:
Capturing of learner interactions
Does the test-bed technology capture learner interactions e.g. in terms of the log-files of tools,
interaction recordings of a PLE solution, or user-given input like social tags? Which data is
available? Which interactions are captured (activities, artefacts, media repositories,
communities, peer actors or software agents)?
24 Does the test-bed technology capture learner interactions e.g. in terms of the
log-files of tools, interaction recordings of a PLE solution, or user-given input like
social tags?
Please choose all that apply:
• Log-files of tools
• Interaction recording of a PLE solution
• User-given input like social tags
• None
• Other:
25 Which data is available (for analysis)?
Please choose all that apply and provide a comment:
Page 49 of 102

Log-files
Interaction recordings
User-given input
None
Other:
26 Which data has already been analysed and how?
Please choose all that apply and provide a comment:
Log-files
Interaction recordings
User-given input
None
Other:
27 Which interactions are captured (activities, artefacts, media repositories,
communities, peer actors or software agents)?
Please choose all that apply and provide a comment:
Activities
Page 50 of 102

Artefacts
Media repositories
Communities
Peer actors / co-learners (add the number of peer actors / co-learners in the field to the
right)
Software agents
Personal contacts (network nodes) (add the number of personal contacts in the field to
the right)
None
Other:
Privacy Policy
Do the organisational policies in the test-bed allow capturing and analysing user data of testbed
learners? Is it possible to install new PLE software (web-based platforms or client
applications) or use third-party tools within the test-bed? Is it allowed to provide an API to user
data or to transfer this data to external systems?
28 Do the organsational policies in the test-bed allow capturing and analysing
user data of test-bed learners? Is it possible to install new PLE software (webbased
platforms or client applications) or use third-party tools within the testbed?
Is it allowed to provide an API to user data or to transfer this data to
external systems?
Please choose the appropriate response for each item:
Page 51 of 102

Yes
No
Do the organisational policies allow capturing user data of test-bed
learners?
Do the organisational policies allow analysing user data of test-bed
learners?
Is it possible to install new PLE software (client applications)
within the test-bed?
Is it possible to install new PLE software (web-based platforms)
within the test-bed?
Is it possible to use third-party tools within the test-bed?
Is it allowed to provide an API to user data?
Is it allowed to transfer this data to external systems?
29 Please describe advantages/disadvantages of the privacy policies in your testbed!
Please write your answer here:
30 Organisational and privacy policies in the test-bed
Please choose the appropriate response for each item:
Yes
No
Are there any organisational policies in your test-bed?
Is it obligatory to agree to the organisational policies?
Are the privacy policies restricted by a third party (external partner)?
Are the privacy policies ROLE-specific?
Analysis of interaction recordings
Are these interaction recordings analysed or used in some way, e.g. for giving visual feedback
or optimising/adapting learning? Is there an attempt to generate and provide PLE
recommendations for learners on the basis of the learner interaction recordings?
31 Are interaction recordings analysed or used in some way, e.g. for giving visual
feedback or optimising/adapting learning?
Please choose only one of the following:
Yes
Page 52 of 102

No
32 Describe, how interaction recordings are analysed or used, e.g. for giving
visual feedback or optimising/adapting learning.
Please write your answer here:
33 Is there an attempt to generate and provide PLE recommendations for learners
on the basis of the learner interaction recordings?
Please choose only one of the following:
Yes
No
34 Describe the attempt to generate and provide PLE recommendations for
learners on the basis of the learner interaction recordings.
Please write your answer here:
Sharing and reusing of good practicess
Does the test-bed allow sharing and reusing good practices of PLE-based activities amongst
the learners? If yes, which practices can be shared? How does practice sharing work and how
much effort is it for the learners?
35 Does the test-bed allow sharing and reusing good practices of PLE-based
activities amongst the learners?
Please choose only one of the following:
Yes
No
36 Which practices can be shared?
Please write your answer here:
37 How does practice sharing work?
Please write your answer here:
Page 53 of 102

38 How much effort is practice sharing for the learners?
Please write your answer here:
39 Do you capture the ways of practice sharing?
Please choose only one of the following:
Yes
No
40 Please enter some numbers concerning practice sharing
Please write your answer(s) here:
How many test-bed learners did share good practices in the last 3 months?
In your opinion, what is the amount of good practices which could be shared in the last three
month?
In your opinion, what is the ratio of good practices shared in comparison to the sharing
potential (in %)?
Submit your survey.
Thank you for completing this survey.
Page 54 of 102

A.2 ROLE - Test-bed validation, psycho-pedagogical level
Test-bed validation strategy
For validating the ROLE vision on a psycho-pedagogical level, the following questions
are supposed to be answered by the test-bed responsible and/or selected learners
Welcome
There are 45 questions in this survey
Identification
Please enter your name and organisation
1 Information about you
Please write your answer(s) here:
Your Name
Your responsibility/role in the project
2 Which test-bed are you evaluating?
Please choose only one of the following:
Festo Lernzentrum Test-bed (Internal Job Opportunity in a Company)
British Institute for Learning and Development test-bed (Continuing Education for
Transition between two Jobs)
OU test-bed by the Open University UK (Continuing Education for Transition between
two Jobs)
RWTH Aachen University Course “Web 2.0 Knowledge Map for academic teaching”
(University to Company)
Jiao Tong University Test-bed (Continuing Education for Transition between two Jobs)
Other
3 Choose the institutional partner you are working for
Please choose only one of the following:
Page 55 of 102

FHG - Fraunhofer FIT
RWTH Aachen - RWTH Aachen University
TUG - Technical University of Graz
K.U.LEUVEN - Katholieke Universiteit Leuven
UNI KO-LD - University of Koblenz
UU - Uppsala University
EPFL - École Polytechnique Fédérale de Lausanne
ULEIC - University of Leicester
OU - Open University UK
WU - Vienna University of Economics & Business
FESTO - Festo Lernzentrum Saar GmbH
IMC - imc AG
BILD - British Institute for Learning and Development
SJTU - Shanghai Jiao Tong University, China
ZSI - Zentrum für Soziale Innovation
UIL - U&I Learning
Other
Learning - categories
What does learning look like in this test-bed (classroom/blended learning vs. pure online
activities; formal, informal, or non-formal learning)
4 What does learning look like in this test-bed?
Examples:
- Classroom: lectures in a classroom, face-to-face instruction
- Blended: mixing different learning environments (classrooms and educational technologies)
- Online: courses provided over ICT and educational technologies, e.g. VLE or LMS platforms
- formal learning (classes, courses, lectures, assessment, certification),
- informal learning (semi-structured activities at home or at work, occasionally),
- non-formal learning (learning in a formal setting but without typical structures
- Other: occasional learning at any possible place
Please choose all that apply:
Classroom learning
Page 56 of 102

Blended learning
Pure online activities
Formal learning
Informal learning
Non-formal learning
Other:
Self-regulated learning
How important is self-regulated learning in the scope of the test-bed activities? What are the
benefits for having self-regulated learning skills in a test-bed? Are there any support facilities for
SRL or tool skills provided by a tool or platform? Is it an explicit goal to foster SRL, or is this a
side-aspect of the test-bed activities?
5 How important is self-regulated learning in the scope of the test-bed activities?
Please choose the appropriate response for each item:
Very
important
Very
unimportan
t
How important is self-regulated learning in
the scope of the test-bed activities
6 What are the benefits for having self-regulated learning skills in a test-bed?
Name and describe them in brief!
Please write your answer here:
7 Are there any support facilities for SRL or tool skills provided by a tool or
platform?
Please choose only one of the following:
Yes
No
8 Name and describe the support facilities for SRL or tool skills provided by a
tool or platform in brief!
Page 57 of 102

Please write your answer here:
9 Is it an explicit goal to foster SRL, or is this a side-aspect of the test-bed
activities?
Please choose the appropriate response for each item:
Yes
No
It is an explicit goal to foster SRL in the test-bed activities
SRL is a side-aspect of the test-bed activities
Freedom in learning
How much freedom in learning is given? Which kinds of guidance are available by a facilitator or
through technological tool?
10 How much freedom in learning is given?
Please choose the appropriate response for each item:
Structured course
content and predefined
lists of
learning materials
Strict navigation
according to
instructional units of
the courses
Goals and tools to find
material beyond predefined
course content
How much
freedom in
learning is given?
Page 58 of 102

11 Which kinds of guidance are available by a facilitator or through technological
tool?
Please write your answer here:
Personalisation
How does personalisation of learning look like in the test-bed activities? Are there any
pedagogical recommendations (next instruction, background literature, peers, etc.) either by
facilitators or by a system?
12 Is there a possibility for personalisation of learning given in the test-bed?
Please choose only one of the following:
Yes
No
13 How does personalisation of learning look like in the test-bed activities?
Please write your answer here:
14 Are there any pedagogical recommendations (next instruction, background
literature, peers, etc.) either by facilitators or by a system?
Check any that apply:
Next instruction
Background literature
Peers
Other
Facilitators System None
15 Describe the pedagogical recommendations you meant by 'other' and whether
they are given by facilitators or the system
Please write your answer here:
Monitoring of learners
Page 59 of 102

How are learners monitored (technically and by facilitators)? Is there technical support for
reflecting learning (e.g. automated analysis of free-text answers, quizzes for self-assessment,
indicator for the learning progress, visualisation of skill mastery, etc.)?
16 How are learners monitored (technically and by facilitators)?
Please choose the appropriate response for each item:
Yes
No
Technically
By facilitators
17 Describe the technical monitoring process in brief!
Please write your answer here:
18 Describe the monitoring process by facilitators in brief!
Please write your answer here:
19 Is there technical support for reflecting learning (e.g. automated analysis of
free-text answers, quizzes for self-assessment, indicator for the learning
progress, visualisation of skill mastery, etc.)?
Please choose only one of the following:
Yes
No
20 Which technical support for reflecting learning (e.g. automated analysis of
free-text answers, quizzes for self-assessment, indicator for the learning
progress, visualisation of skill mastery, etc.) is given?
Please choose the appropriate response for each item:
Yes
No
Automated analysis of free-text answers
Quizzes for self-assessment
Indicator for the learning progress
Visualisation of skill mastery
Page 60 of 102

Yes
No
There is no technical support
Other
21 Name and describe the technical support for reflecting learning stated in
'other' in brief
Please write your answer here:
Role of collaborative learning
Which role does collaborative learning play? Do learners benefit from collaborative situations,
and do they get support from peers? How are learning groups established?
22 Which role does collaborative learning play?
Please write your answer here:
23 Do learners benefit from collaborative situations, and do they get support from
peers?
Please choose the appropriate response for each item:
Yes
No
Do learners benefit from collaborative situations
Do learners get support from peers
24 How are learning groups established?
Please write your answer here:
Assessment and measurement
Are learners being assessed, and is the learning outcome measured? If yes, how and by whom
is this done? Does technology support assessment, and how is the assessment outcome used
for which purposes?
25 Are learners being assessed, and is the learning outcome measured?
Please choose the appropriate response for each item:
Page 61 of 102

Yes
No
Are learners being assessed
Is the learning outcome measured
26 If yes, by whom are learners assessed?
Please write your answer here:
27 If yes, how is this assessment done?
Please write your answer here:
28 Does technology support assessment?
Please choose only one of the following:
Yes
No
29 If yes, describe in brief how technology supports assessment.
Please write your answer here:
30 How is the assessment outcome used for which purposes?
Please write your answer here:
Modelling and assessing of activities and skills
How are activities and skills (domain-specific, tool-specific, SRL-specific) modelled and
assessed? Is there a learner profile in electronic form which includes learning goals (to-be
states) and skill/competence development over time (current states)?
31 Are activities and skills (domain-specific, tool-specific, SRL-specific) modelled
and assessed?
Please choose only one of the following:
Page 62 of 102

Yes
No
32 How are domain-specific activities and skills modelled and assessed?
Please write your answer here:
33 How are tool-specific activities and skills modelled and assessed?
Please write your answer here:
34 How are SRL-specific activities and skills modelled and assessed?
Please write your answer here:
35 Is there a learner profile in electronic form which includes learning goals (tobe
states) and skill/competence development over time (current states)?
Please choose all that apply and provide a comment:
Learning goals (to-be states)
Skill/competence development over time (current states)?
There is no learner profile
Other:
Planning for learning
Who performs the planning for learning and who defines the expected outcome (learning
goals)? Is this done formally and explicitly?
Page 63 of 102

36 Who performs the planning for learning and who defines the expected
outcome (learning goals)?
Please write your answer(s) here:
Who performs the planning for learning ?
Who defines the expected outcome (learning goals)?
37 Is this done formally and explicitly?
Please choose the appropriate response for each item:
Yes
No
Formally
Explicitly
Future plans
Which of these five aspects (SRL, guidance, recommendations, monitoring and assessment,
activities and skill profiles) are planned to be addressed in the future? Where are the restrictions
concerning these points in the test-bed?
38 Which of these five aspects (SRL, guidance, recommendations, monitoring
and assessment, activities and skill profiles) are planned to be addressed in the
future?
Please choose all that apply and provide a comment:
SRL
Guidance
Recommendations
Monitoring and assessment
Page 64 of 102

Activities and skill profiles
None
Other:
39 Where are the restrictions concerning these points (SRL, guidance,
recommendations, monitoring and assessment, activities and skill profiles) in the
test-bed?
Please choose all that apply and provide a comment:
SRL
Guidance
Recommendations
Monitoring and assessment
Activities and skill profiles
None
Other:
40 Where are the restrictions concerning these points (SRL, guidance,
recommendations, monitoring and assessment, activities and skill profiles) in the
test-bed? Add more information, if available.
Page 65 of 102

Please write your answer here:
Training material
Which kinds of training materials are available in the test-bed? Are these materials
customisable according to individual needs of learners? Is the learning material modelled, in
order to be used by services (e.g. to be recommended)?
41 Which kinds of training materials are available in the test-bed?
Please write your answer here:
42 Are these materials customisable according to individual needs of learners?
Please choose only one of the following:
Yes
No
43 Are these materials customisable according to individual needs of learners? -
Describe how.
Please write your answer here:
44 Is the learning material modelled, in order to be used by services (e.g. to be
recommended)?
Please choose only one of the following:
Yes
No
45 Describe in brief, how the learning material is modelled in order to be used by
services (e.g. to be recommended)?
Please write your answer here:
Page 66 of 102

Submit your survey.
Thank you for completing this survey.
A.3 ROLE - Test-bed validation, SRE
Test-bed validation strategy
The SRE process can be validated for each test-bed through the following questions:
Welcome
There are 18 questions in this survey
Identification
1 Information about you
Please write your answer(s) here:
Your Name
Your responsibility/role in the project
2 Which test-bed are you evaluating?
Please choose only one of the following:
Festo Lernzentrum Test-bed (Internal Job Opportunity in a Company)
British Institute for Learning and Development test-bed (Continuing Education for
Transition between two Jobs)
OU test-bed by the Open University UK (Continuing Education for Transition between
two Jobs)
RWTH Aachen University Course “Web 2.0 Knowledge Map for academic teaching”
(University to Company)
Jiao Tong University Test-bed (Continuing Education for Transition between two Jobs)
Other
3 Choose the institutional partner you are working for
Page 67 of 102

Please choose only one of the following:
FHG - Fraunhofer FIT
RWTH Aachen - RWTH Aachen University
TUG - Technical University of Graz
K.U.LEUVEN - Katholieke Universiteit Leuven
UNI KO-LD - University of Koblenz
UU - Uppsala University
EPFL - École Polytechnique Fédérale de Lausanne
ULEIC - University of Leicester
OU - Open University UK
WU - Vienna University of Economics & Business
FESTO - Festo Lernzentrum Saar GmbH
IMC - imc AG
BILD - British Institute for Learning and Development
SJTU - Shanghai Jiao Tong University, China
ZSI - Zentrum für Soziale Innovation
UIL - U&I Learning
Other
Stakeholders
Which stakeholders (learners, developers, and researchers) are involved in developing and
providing educational technology in the test-bed? Are requirements claimed by learners
considered and realised? If yes, how are these end-user requirements gathered, and who is
responsible for implementing them?
4 Which stakeholders (learners, developers, and researchers) are involved in
developing and providing educational technology in the test-bed?
Please choose all that apply:
Learners
Developers
Researchers
Page 68 of 102

None
Other:
5 Describe the stakeholders, mentioned in others
Please write your answer here:
6 Are requirements claimed by learners considered and realised?
Please choose only one of the following:
Yes
No
7 If yes, how are these end-user requirements gathered? Describe the process
and tools in brief!
Please write your answer here:
8 Who is responsible for implementing these end-user requirements?
Please write your answer here:
Input of external stakeholders
Can external stakeholders (e.g. researchers or open source providers) bring own ideas or
technology into a test-bed? How does this (formal or informal) process of collaborating with
external stakeholders look like?
9 Can external stakeholders (e.g. researchers or open source providers) bring
own ideas or technology into a test-bed?
Please choose all that apply and provide a comment:
Researchers
Open source providers
Page 69 of 102

External stakeholders are not able to bring own ideas or technology into a test-bed
Other:
10 How does this (formal or informal) process of collaborating with external
stakeholders look like?
Please write your answer here:
Last ideas and future work
Which idea or tool of the ROLE project has been taken up already? What is planed as future
work?
11 Which idea or tool of the ROLE project has been taken up already?
Please write your answer here:
12 What is planned as future work?
Please write your answer here:
Integration of external tools by learners
Can learners integrate external tools (web-applications or third party software) into the technical
infrastructure of a test-bed on their own and without requesting permissions? If yes, are there
any restrictions on external software?
13 Can learners integrate external tools (web-applications or third party software)
into the technical infrastructure of a test-bed on their own and without requesting
permissions?
Please choose only one of the following:
On their own and without requesting permissions
Page 70 of 102

On their own after requesting permissions
Only with support from the provider of the technical infrastructure
The learner cannot integrate external tools at all
Make a comment on your choice here:
14 If yes, are there any restrictions on external software?
Please write your answer here:
Repositories for tools/widgets
Does the test-bed include a repository for tools/widgets (like the AppStore) or one for sharing
and providing best (learning/teaching) practices? If yes, describe the technical solution!
15 Does the test-bed include a repository for tools/widgets (like the AppStore,
Widget store, tool store ..) or one for sharing and providing best
(learning/teaching) practices?
Please choose all that apply:
Repository for tools / widgets
Repository for sharing / providing best learning practices
Repository for sharing / providing best teaching practices
There is no repository at all
Other:
16 If yes, describe the technical solution of the repository!
Please write your answer here:
Success measurement
How is the success of a technical learning infrastructure measured or evaluated in the test-bed?
Which feedback do learners and teachers (facilitators) give?
17 How is the success of a technical learning infrastructure measured or
evaluated in the test-bed?
Page 71 of 102

Please write your answer here:
18 Which feedback do learners and teachers (facilitators) give?
Please write your answer here:
Submit your survey.
Thank you for completing this survey.
A.4 ROLE - Test-bed validation, technical perspective
Test-bed validation strategy
Welcome
There are 68 questions in this survey
Identification
1 Information about you
Please write your answer(s) here:
Your Name
Your responsibility/role in the project
2 Which test-bed are you evaluating?
Please choose only one of the following:
Festo Lernzentrum Test-bed (Internal Job Opportunity in a Company)
British Institute for Learning and Development test-bed (Continuing Education for
Transition between two Jobs)
OU test-bed by the Open University UK (Continuing Education for Transition between
two Jobs)
Page 72 of 102

RWTH Aachen University Course “Web 2.0 Knowledge Map for academic teaching”
(University to Company)
Jiao Tong University Test-bed (Continuing Education for Transition between two Jobs)
Other
3 Choose the institutional partner you are working for
Please choose only one of the following:
FHG - Fraunhofer FIT
RWTH Aachen - RWTH Aachen University
TUG - Technical University of Graz
K.U.LEUVEN - Katholieke Universiteit Leuven
UNI KO-LD - University of Koblenz
UU - Uppsala University
EPFL - École Polytechnique Fédérale de Lausanne
ULEIC - University of Leicester
OU - Open University UK
WU - Vienna University of Economics & Business
FESTO - Festo Lernzentrum Saar GmbH
IMC - imc AG
BILD - British Institute for Learning and Development
SJTU - Shanghai Jiao Tong University, China
ZSI - Zentrum für Soziale Innovation
UIL - U&I Learning
Other
Technical tools
Are there technical tools for creating or visualising pedagogical or community models? Who is
working on such models? What are the models used for in the test-bed?
4 Are there technical tools for creating or visualising pedagogical or community
models?
Page 73 of 102

Please choose the appropriate response for each item:
Yes
No
Are there technical tools for creating pedagogical models?
Are there technical tools for creating community models?
Are there technical tools for visualising pedagogical models?
Are there technical tools for visualising community models?
5 Name and describe the technical tools for creating pedagogical models
Please write your answer here:
6 Name and describe the technical tools for creating community models
Please write your answer here:
7 Name and describe the technical tools for visualising pedagogical models
Please write your answer here:
8 Name and describe the technical tools for visualising community models
Please write your answer here:
9 Who is working on such models? Describe the role/organisational unit and
responsibility of the person, working on such models!
Please write your answer(s) here:
Who is working on technical tools for creating pedagogical models?
Who is working on technical tools for creating community models?
Page 74 of 102

Who is working on technical tools for visualising pedagogical models?
Who is working on technical tools for visualising community models?
10 What are the models used for in the test-bed?
Please write your answer(s) here:
• What are the pedagogical models used for in the test-bed?
•
• What are the community models used for in the test-bed?
•
Technologies
Which technologies are used for (a) communication and collaboration, (b) learning domain and
learning planning, (c) authoring of training materials, (d) enabling services (authentication,
authorisation, automatic metadata generation, content management, searching, mashing up
and integrating different tools, etc.), and (e) learning management (LMS) and personalised
learning (PLE)?
11 Which technologies are used for communication and collaboration?
Please write your answer here:
12 Which technologies are used for learning domain and learning planning?
Please write your answer here:
13 Which technologies are used for authoring of training materials?
Please write your answer here:
14 Which technologies are used for enabling services (authentication,
authorisation, automatic metadata generation, content management, searching,
mashing up and integrating different tools, etc.)?
Please write your answer here:
Page 75 of 102

15 Which technologies are used for learning management (LMS) and
personalised learning (PLE)?
Please write your answer here:
Application types
What types of applications (client-sided programs, platforms, and hybrids) are utilised in the
test-bed? Which tool assembly techniques can be identified? Which protocols and service
designs (i.e. SOAP, REST, XML-RPC, XMPP, etc.) are used? Which kinds of data
representations and formats (IEEE LOM, XML, RDF/OWL, JSON, etc.) are identifiable?
16 What types of applications (client-sided programs, platforms, and hybrids) are
utilised in the test-bed? Provide an example or a comment.
Please choose all that apply and provide a comment:
Client-sided programs
Platforms
Hybrids
None
Other:
17 Which tool assembly techniques can be identified? Provide an example or a
comment.
Please choose all that apply and provide a comment:
Webtop and visual mash-ups
Page 76 of 102

Linking and mashing up data
Inter-widget communication
Social networking and sharing
Widget-based activity flows
None
Other:
18 Which protocols and service designs (i.e. SOAP, REST, XML-RPC, XMPP, etc.)
are used? Provide an example or a comment.
Please choose all that apply and provide a comment:
SOAP
REST
XML-RPC
XMPP
None
Other:
Page 77 of 102

19 Which kinds of data representations and formats (IEEE LOM, XML, RDF/OWL,
JSON, etc.) are identifiable?
Provide an example or a comment.
Please choose all that apply and provide a comment:
IEEE LOM
XML
RDF/OWL
JSON
None
Other:
Sets of learning tools
Describe three concrete sets of learning tools (personal learning service bundles) which are
used for specific learning scenarios and purposes within the test-bed!
20 Describe set one (1) of three concrete sets of learning tools (personal learning
service bundles) which are used for specific learning scenarios and purposes
within the test-bed!
Please write your answer here:
Please add a name, a target or purpose and a technical perspective!
21 Describe set two (2) of three concrete sets of learning tools (personal learning
service bundles) which are used for specific learning scenarios and purposes
within the test-bed!
Page 78 of 102

Please write your answer here:
Please add a name, a target or purpose and a technical perspective!
22 Describe set three (3) of three concrete sets of learning tools (personal
learning service bundles) which are used for specific learning scenarios and
purposes within the test-bed!
Please write your answer here:
Please add a name, a target or purpose and a technical perspective!
PLS bundles
In which way have the PLS bundles of the ROLE projects (Christmas, Easter, Stonehenge)
been applied in the test-bed? Summarise experiences given by the organisation and by
(selected) learners!
23 Case Christmas:
In which way has Christmas been applied in the test-bed? Summarise
experiences given by the organisation!
Please write your answer here:
24 Case Christmas:
In which way has Christmas been applied in the test-bed? Summarise
experiences given (selected) learners!
Please write your answer here:
25 Case Easter:
In which way has Easter been applied in the test-bed? Summarise experiences
given by the organisation!
Please write your answer here:
Page 79 of 102

26 Case Easter:
In which way has Easter been applied in the test-bed? Summarise experiences
given by (selected) learners!
Please write your answer here:
27 Case Stonehenge:
In which way has Stonehenge been applied in the test-bed? Summarise
experiences given by the organisation!
Please write your answer here:
28 Case Stonehenge:
In which way has Stonehenge been applied in the test-bed? Summarise
experiences given by (selected) learners!
Please write your answer here:
Usage of ROLE specifications
Which of the ROLE specifications are used for (a) structuring user and content data, (b)
specifying architectures and interfaces, (c) communication between systems, (d) tool and
service syndication (widget technology), and (e) capturing learner interactions? Will there be
any standardisation initiatives for one of these areas? Can this data be accessed through some
API? If not, is it planned to open up the repositories in the test-bed and provide this data via
API?
29 Which of the ROLE specifications are used for structuring user and content
data
Please choose all that apply:
SCORM
iCalender
Unspecified
None
Other:
Page 80 of 102

30 Which of the ROLE specifications are used for specifying architectures and
interfaces
Please choose all that apply:
Apache Shinding
Wookie, GadgetTabML(!) and OpenSocial Gadgets
OpenAjaxs Metadata
Unspecified
None
Other:
31 Which of the ROLE specifications are used for communication between
systems
Please choose all that apply:
XMPP
Event API
Unknown
Unspecified
None
Other:
32 Which of the ROLE specifications are used for tool and service syndication
Please choose all that apply:
OAI-PMH
SPI
Unspecified
None
Other:
33 Which of the ROLE specifications are used for capturing learner interactions
Page 81 of 102

Please choose all that apply:
Google Wave
Unspecified
None
Other:
34 Which of the ROLE specifications are used for digital rights management
(consider 'capturing learner interactions', too)
Please choose all that apply:
Creative commons
ODRL
Unspecified
None
Other:
35 Which of the ROLE specifications are used for structuring/describing content
artefacts
Please choose all that apply:
IEEE LOM
MPEG7
Unspecified
None
Other:
36 Which of the ROLE specifications are used for structuring user data
Please choose all that apply:
IEEE RCD
HR-XML
Unspecified
Page 82 of 102

None
Other:
37 Is there or will there be any standardisation initiatives for one of these areas?
Please choose the appropriate response for each item:
Yes
No
Specifications for structuring user data
Specifications for structuring content data
Specifying architectures
Specifying interfaces
Communication between systems
Tool syndication (widget technology)
Service syndication (widget technology)
Capturing learner interactions
38 Can this data be accessed through some API?
Please choose only one of the following:
Yes
No
39 If not, is it planned to open up the repositories in the test-bed and provide this
data via API?
Please choose only one of the following:
Yes
No
40 Describe how this data can be accessed through some API or your plans to
open up the repositories in the test-bed and provide this data via API.
Please write your answer here:
ROLE space like solution
Page 83 of 102

Does the test-bed provide a ROLE space like solution which allows users to select and
assemble learning tools in a widget-based way? Is it planned to introduce such a PLE solution,
either by rolling out a widget infrastructure or extending an existing platform in this direction?
41 Does the test-bed provide a ROLE space like solution which allows users to
select and assemble learning tools in a widget-based way?
Please choose all that apply and provide a comment:
The test-bed provides a ROLE space like solution which allows users to select and
assemble learning tools in a widget-based way.
It is planned to introduce such a PLE solution by rolling out a widget infrastructure.
It is planned to introduce such a PLE solution by extending an existing platform in this
direction.
Other:
42 Describe the plans for introducing such a PLE solution
Please write your answer here:
Monitoring and capturing
Are there any approaches towards learner monitoring or user interaction capturing within the
test-bed? If yes, which format for storing these interaction recordings is used? What is
happening to this data (analysis, sharing, generation of recommendations, etc.)? If no, is it
planned to implement this kind of learner monitoring? Are there any privacy restrictions (policies
by the organisation or settings given by users) concerning interaction recordings?
43 Are there any approaches towards learner monitoring or user interaction
capturing within the test-bed?
Please choose only one of the following:
Yes
No
44 If yes, which format for storing these interaction recordings is used?
Page 84 of 102

Please write your answer here:
45 What is happening to the data generated by learner monitoring?
Please choose the appropriate response for each item:
Yes
No
Analysing
Sharing
Used to generate recommendations
Used to offer new possibilities
No usage at the moment
46 What is happening to the data captured from user interaction?
Please choose the appropriate response for each item:
Yes
No
Analysing
Sharing
Used to generate recommendations
Used to offer new possibilities
No usage at the moment
47 Is it planned to implement some kind of learner monitoring?
Please choose only one of the following:
Yes
No
48 Please describe your plans to implement learner monitoring or user interaction
capturing!
Please write your answer here:
49 Are there any privacy restrictions concerning interaction recordings?
Page 85 of 102

Please choose all that apply and provide a comment:
The privacy restrictions are policies, implemented by the organisation.
There are privacy restrictions implemented by user settings.
There are no privacy restrictions concerning learner monitoring.
Other:
User profiles
What do the user profiles (of the learners) look like? Is there some authentication or
authorisation service used for accessing courses and content in the test-bed? If there are any
other tools, it is necessary to have additional user accounts? How can learners access and
modify their profiles? What is planned regarding the user profiles and the authentication?
50 What do the user profiles (of the learners) look like?
Please write your answer here:
51 Is there some authentication (verify that someone is who they claim they are)
or authorisation (finding out if one is permitted to have the resource) service
used for accessing courses and content in the test-bed?
Please choose all that apply and provide a comment:
There is an authentication service used for accessing courses in the test-bed
There is an authentication service used for accessing content in the test-bed
There is an authorisation service used for accessing courses in the test-bed
There is an authorisation service used for accessing content in the test-bed
Page 86 of 102

If there are any other tools, is it necessary to have additional user accounts?
Other:
52 Can learners access and modify their profiles?
Please choose only one of the following:
Yes
No
53 How can learners access and modify their profiles? Please describe the
process of accessing and modifying profiles!
Please write your answer here:
54 What is planned regarding the user profiles and the authentication? Please
describe the plans according user profiles and authentication!
Please write your answer here:
Sensitive data
Next to user profiles and interaction records, which other sensitive data is available at the testbed?
Are there any privacy restrictions (laws, policies by the organisation or settings given by
users) concerning these data sets? Have users the possibility to protect their privacy, e.g.
through modifying settings in their user profiles? Are there any approaches for creating
awareness of privacy issues amongst the learners?
55 Apart from user profiles and interaction records, which other sensitive data is
available at the test-bed?
Please write your answer here:
Page 87 of 102

56 Are there any privacy restrictions (laws, policies by the organisation or
settings given by users) concerning these data sets?
Please choose all that apply:
Laws
Policies by the organisation
Settings given by users
None
Other:
57 Have users the possibility to protect their privacy, e.g. through modifying
settings in their user profiles?
Please choose only one of the following:
Yes
No
58 Name and describe in brief, how users can protect their privacy in their user
profiles!
Please write your answer here:
59 Are there any approaches for creating awareness of privacy issues amongst
the learners?
Please choose only one of the following:
Yes
No
60 Name and describe the approaches for creating awareness of privacy issues
amongst the learners!
Please write your answer here:
Development of user interfaces and features
Page 88 of 102

How are user interfaces and features of the learning software developed in the test-bed? Can
new technology (software and hardware) be integrated into existing systems, and are there any
user experiences on such integrations? Are user feedback and requests considered
appropriately and taken up by developers quickly? Are there any discussions or evaluations of
typical usability issues (efficiency, learnability, subjective satisfactory, etc.)?
61 How are user interfaces and features of the learning software developed in the
test-bed?
Please write your answer here:
62 Can new technology (software and hardware) be integrated into existing
systems?
Please choose only one of the following:
Yes
No
63 Are there any user experiences on such integrations?
Please choose only one of the following:
Yes
No
64 Describe how new technology (software and hardware) can be integrated into
existing systems!
Please write your answer here:
65 Describe the user experiences on such integrations of new technology
(software and hardware)!
Please write your answer here:
66 Are user feedback and requests considered appropriately and taken up by
developers quickly?
Please choose only one of the following:
Yes
Page 89 of 102

No
67 Are there any discussions or evaluations of typical usability issues (efficiency,
learnability, subjective satisfactory, etc.)?
Please choose only one of the following:
Yes
No
68 Describe the discussions or evaluations of typical usability issues(efficiency,
learnability, subjective satisfactory, etc.)!
Please write your answer here:
Submit your survey.
Thank you for completing this survey.
Page 90 of 102

Appendix B: Results of the Christmas, Easter, and
Stonehenge Project
B.1 OU test-bed
The OU test-bed regards the transition from a traditional and rather static Learning
Management System (LMS) towards a responsive and dynamic Personal Learning
Environment (PLE).
Christmas Project
The results of the Christmas project demonstrated how widget bundles can be
developed and used for specific learning tasks. This served as the basis for the
development of bundles explicitly targeted to the needs of the OpenLearn user in the
following two internal projects.
Easter Project
During the Easter project, the first attempt was made to inject ROLE technologies
directly into OpenLearn, in the form of additional or alternative tools to the ones already
present in this environment. In particular, a widget bundle was developed for finding
open educational resources and working collaboratively with them.
Figure B-1: ROLE and third-party widgets within an OpenLearn course
Page 91 of 102

This bundle consisted of the ObjectSpot search widget, the FlashMeeting widget, as
well as the EtherPad widget. The ObjectSpot widget was used for performing queries
into a variety of online repositories, including OpenLearn, iTunesU, Wikipedia,
YouTube, and Slideshare. The FlashMeeting widget allowed learners to hold
videoconferences with their peers and the EtherPad widget allowed them to
collaboratively work on a document in real time. These widgets were used together with
external Google gadgets, as shown in Figure B-1.
Stonehenge Project
During the Stonehenge project, a number of areas of improvement within OpenLearn
were identified, offering opportunities for the deployment of new ROLE widgets and
services. More specifically, the lack of support to the learner for self-regulated learning
was pointed out as an opportunity for an application of the ROLE psycho-pedagogical
model. In addition, the need for better integration of learning tools with the learning
content offered by OpenLearn was identified. Ongoing development work has been
focused on addressing these issues through the incorporation of OpenSocial
technologies into the Moodle-based OpenLearn. The aim of this work is to allow widgets
employing the social capabilities of the OpenSocial API to be integrated into any Moodle
installation.
Relevant learning tools and systems (plus public demonstrators):
• OpenLearn LabSpace (learning management platform for research and
educational experiments at the OU)
o URL: http://labspace.open.ac.uk/
• PLE demonstrator in LabSpace (a first attempt to integrate widget technology
into LabSpace)
o URL: http://labspace.open.ac.uk/mod/resource/view.php?id=420341
• Flashmeeting (browser-based tool to conduct and manage virtual meetings)
o URL: http://flashmeeting.open.ac.uk/http://fm.ea-tel.eu/
• FlashVlog (a video blog, i.e. a weblog which allows recording and accessing
video recordings of users)
o URL: http://flashmeeting.open.ac.uk/flashvlog/
• Cohere (web-based sensemaking tool)
o URL: http://cohere.open.ac.uk/
• Compendium (desktop version of a sensemaking tool)
o URL: http://compendium.open.ac.uk/
Page 92 of 102

B.2 SJTU test-bed
From the viewpoint of SJTU, the Christmas and Easter Projects mainly explored
fundamental technology where the assembled prototypes served as proof of concepts,
and were not designed for usage in the classrooms. Thus, SJTU contribution to these
projects was limited, and SJTU rather explored usage of their Liferay PLE system in
teaching.
Stonehenge Project
As a part of the Stonehenge Project, SJTU extended the Liferay PLE with ROLE
technologies, especially CAM/OpenApp. Most of the widgets integrated in the Liferay
PLE were extended to publish CAM events. Furthermore, the recommendation services
provided by ROLE partners were included in the PLE.
Figure B-2: Example Liferay PLE (widget mash-up) for language learning
Figure B-2 and B-3 show (partial) screenshots of the Liferay PLE. In Figure B-2, the
text-to-speech widget is on the left-hand side, the spell checker on the top right. Below it
is a recording tool that allows the student to practice their pronunciation. Not visible in
the figure are the translation and the uploading tool. Figure B-3 shows content from an
existing website embedded in PLE using an iframe.
Page 93 of 102

Figure B-3: External content embedded into the Liferay PLE
Relevant learning tools and systems (plus public demonstrators):
• Liferay-based PLE (a PLE solution for creating and providing tool mash-ups for
specific learning situations, like language learning):
o URL: http://202.120.34.34/
o Screen name: student
o Password: !student!
Page 94 of 102

B.3 RWTH test-bed
Similarly to SJTU, the RWTH test-bed mainly explored fundamental ROLE technologies
in the scope of the Christmas Project. Thus, the contribution to this ROLE milestone
was limited.
Easter Project
As a part of the Easter Project, RWTH and Frauenhofer FIT developed a first new
version of the knowledge map. This version was the first test-bed version which
includes new ROLE techniques like inter- and intra-widget communication.
Figure B-4: Screenshot of Easter Project prototype, a widgetised version of the Knowledge Map
Figure B-4 shows a (partial) screenshot of the Easter Project version. It extends the
original Knowledge Map tool by a widget infrastructure and includes the “Personal
History” widget, a version of the CAM widget developed by FIT, as well as a chat widget
developed by RWTH University. These three widgets communicate with each other.
The chat widget also communicates with other widgets in other Browser-Instances by
using XMPP-techniques.
Stonehenge Project
The Stonehenge Project aimed at maturing the concepts and developments of the
Easter Project with no fundamentally new contribution for the RWTH test-bed users.
Page 95 of 102

Relevant learning tools and systems (plus public demonstrators):
• Knowledge Map (a tool to enrich courses with an improved structure and topical,
relevant background information for project tasks):
o URL: http://role-is.dbis.rwth-aachen.de/role/wkm-public/
o Registration required
Page 96 of 102

B.4 FESTO test-bed
Christmas Project
The result of the Christmas project was a ROLE PLE prototype based on a dashboard
with interoperable widgets. For FESTO, these results were a first showcase on how a
PLE could look like and which functionalities it offers. The prototype was used to
demonstrate users, internal knowledge providers and trainers in which direction learning
management systems could be developed in the future. The outcomes of the Christmas
project (for example the language Resource Browser widget, the translator widget and
the vocabulary widget) showed what is possible regarding widget-interfaces and
communication between widgets and the platform.
Easter Project
The FESTO test-bed is focused on LMS users and especially LMS users in a company.
This target group has special needs and the surrounding conditions in companies are
not as flexible as those predominant, for example, in universities. Learners in business
environments have primarily to fulfil their job role, and learning is mostly to support them
in doing so. Due to high workloads, it is often hard to learn on the job. Business learners
are a very heterogeneous target group with big age differences (from 16 to 65), different
educational backgrounds and previous knowledge, job-roles, learning requirements,
learning preferences and learning goals. Therefore FESTO decided to personalise
learning through ROLE research outcomes and made learning more demand-oriented
and service-oriented for the users of the FESTO Virtual Academy.
Derived from this spectrum of learners, FESTO created a list of widgets that benefit the
target group of the test-bed. This list has then been converted into a roadmap
(subdivided in short-term planned, medium planned and long term planned widgets)
together with the developers of IMC. The first widget from this list the a federated
search widget, that searches internal and external databases and repositories for
learning content, has now been fed into the Stonehenge project as a development goal.
Stonehenge Project
The taskforce members involved were the ROLE partners from IMC, University of
Koblenz, RWTH Aachen, TU Graz and KU Leuven. There was a taskforce meeting
which took place at Festo Lernzentrum in St. Ingbert, where researchers and
developers met with FESTO learners to discuss how they learn. Three different learners
demonstrated and explained how they use the FESTO Virtual Academy in three
different learning scenarios (new employee, foreign language training, product training).
There was also a brainstorming of ideas and a ranking of their feasibility within the
Stonehenge project and test-bed benefit. The result of the Stonehenge Project was the
implementation of the federated search widget (see Figure B-5).
Page 97 of 102

Figure B-5: ROLE’s search widget (left-hand side) and FESTO’s Media Search Widget (right-hand side)
FESTO and IMC agreed upon realising the federated search widget on the Live-System
of the FESTO Virtual Academy for testing issues. KU Leuven supported IMC in the
development and improvement of the search widget. In the first step, the widget
searches external Internet databases for learning content, in the beginning of next year
the focus will be on bringing internal FESTO resources into the widget.
Relevant learning tools and systems (plus public demonstrators):
• Virtual Academy (learning management system of FESTO based on the
commercial product CLIX, only accessible for FESTO employees)
Page 98 of 102

B.5 BILD test-bed
The BILD is a membership organisation representing the interests of over 2000 learning
and development professionals. Each member represents a different organisation
utilising potentially different learning systems and servicing varying types of learners
working in different contexts. The members selected to participate in the test bed
activity represent scenarios where there are individual learners and where there is a
potential to make a transition from working in isolation to sharing competencies with
others and entering into a collaborative situation.
Christmas Project
The BILD was mainly involved with dissemination activities during this period and was
showcasing the technological methods available by utilising the ROLE system. A
language learning scenario was demonstrated and discussed at industry events and
with BILD members at BILD events. Feedback was given where appropriate and
possible.
Easter Project
The BILD member Etestme produced a widget sized launch page to their site. This was
the first time that a BILD member had explored the possibilities of using gadget based
learning in their systems. The member found this productive and went on to contribute a
more enhanced widget for the Stonehenge Project.
Stonehenge Project
During the Stonehenge Project the BILD member Etestme experimented with gadget
based technology further by producing an Etestme results display gadget (cf. Figure B-
6). The idea being that the learners could embed this in their learning system. The tool
was made available in iGoogle and there are plans to test and evaluate it within the
Etestme system later in the year.
Figure B-6: Gadget for displaying Etestme results
Page 99 of 102

Relevant learning tools and systems (plus public demonstrators):
• E-Testme platform (an online service for learners to practice IELTS tests)
o URL: http://www.etestme.co.uk/
o Registration required
Page 100 of 102

References
Chatterjee, A., and Law, E.: Draft evaluation methodology from the technical, usability and
psycho-pedagogical perspective including guidelines for the empirical realisation. ROLE
Deliverable ID6.3/ID6.4, 2010.
Chatterjee, A., Law, E., and Verbert, K.: Functional and Non-Functional requirements analysis
and specification. ROLE Deliverable D1.3/D1.4, 2009.
Dahrendorf, D.: Service Syndication Platform. ROLE Deliverable D4.2, 2010.
Ferdinand, P., and Kiefel, A.: Survey on existing Models. ROLE Deliverable D2.1, 2009.
Ferdinand, P., and Kiefel, A.: Applicability of Theoretical Models. ROLE Deliverable D2.2, 2010.
Govaerts, S., and Dahrendorf, D.: Prototypical implementations. ROLE Deliverable D3.4, 2011.
Höbelt, C., and Zimmermann, V.: Exploitation and use/technology transfer plan and report.
ROLE Deliverable D9.1, 2009.
Isaksson, E., and Palmer, M.: Usability and inter-widget communication in PLE. Proceedings of
MUPPLE Workshop at the European Conference on Technology Enhanced Learning (EC-
TEL), Barcelona, 2010.
Kroop, S.: ROLE Portal. ROLE Deliverable D8.1, 2009.
Kroop, S., and Hofer, M.: Dissemination and Awareness plan for ROLE. ROLE Deliverable
D8.3, 2010.
Kroop, S., and Scheffel, M.: Promotional ROLE Media Kit. ROLE Deliverable D8.2, 2009.
Manouselis, N., Drechsler, H., Verbert, K., and Santos, O.C.: Proceedings of the 1st Workshop
on Recommender Systems for Technology Enhanced Learning (RecSysTEL 2010).
Procedia Computer Science 1(2), Elsevier, 2010, pp. 2773-2998.
Mikroyannidis, A., and Gillet, D.: Participatory design and implementation plan of Personal
Learning Test-beds. ROLE Deliverable D5.1/D5.2, 2009.
Moiszi, S., and Kroop, S.: Setup and Extension of ROLE ALLIANCE Program. ROLE
Deliverable D9.2, 2010.
Mödritscher, F., and Petrushyna, Z.: Model and Methodology for PLE-Based Collaboration in
Learning Ecologies. ROLE Deliverable D7.1/ID7.2, 2009.
Mödritscher, F., and Wild, F.: Draft Prototype of a Mash-up PLE. ROLE Deliverable ID7.1, 2010.
Mödritscher, F., Wild, F., and Petrushyna, Z.: Strategies and Facilities for Activity Pattern
Sharing. ROLE Deliverable D7.2, 2010.
Nussbaumer, A.: Draft assessment procedures and guidance principles for supporting selfregulation.
ROLE Deliverable ID6.1, 2010.
Nussbaumer, A.: Draft common psycho-pedagogical principles in the context of individually
compiled learning environments. ROLE Deliverable ID6.2, 2010.
Nussbaumer, A., and Fruhmann, K.: Common psycho-pedagogical framework. ROLE
Deliverable D6.1, 2009.
Palmer, M., Isaksson, E., and Naeve, A.: Personal Learning Service Bundles. ROLE Deliverable
D4.1, 2010.
Page 101 of 102

Palmer, M., Nilsson, M., and Naeve, A.: Guidelines for the choice of integration technologies.
ROLE Deliverable D3.2, 2009.
Renzel, D.: Test-bed Training Material Collection. ROLE Deliverable D5.3, 2010.
Schmitz, H.-C., and Kroop, S.: Promotion activity of PLS Bundle and Prototype of a Mash-up
PLE. ROLE Deliverable ID8.1, 2010.
Verbert, K.: Version 1 of the ROLE specifications. ROLE Deliverable D3.3, 2010.
Verbert, K., Dahn, I., and Kiefel, A.: Survey of learning-related services. ROLE Deliverable D3.1,
2009.
Page 102 of 102