TutorHome evaluation summary (34 kb) - The Open University


TutorHome evaluation summary (34 kb) - The Open University

Evaluation of



20 December 2004

Katherine Prince

AL World Project Coordinator


Survey Design

TutorHome has been available since February 2003, with all ALs required to use it since

January 2004. ALs are expected to log on at least once a week, depending on the level of

computing usage associated with the course. Student details have been delivered solely

online since March 2004.

The TutorHome team organised a survey to evaluate the site’s current usefulness and to

help set direction for the next round of developments to take place during 2005. The

survey was administered by IET.

A random sample of 2,148 ALs were contacted via their OU e-mail addresses and asked to

complete an online questionnaire published using IET’s electronic survey administration

system (ELSA). They had the option of requesting a paper version of the questionnaire.

The sample included the 13 ALs working on web-intensive courses and was stratified to

help ensure representation across ALs working on web-enhanced and web-focused


The survey remained open from 16 September 2004 through 18 October 2004. An e-mail

reminder was sent to non-respondents on 28 September 2004.

Evaluation of TutorHome, Autumn 2004 -- 2

Questionnaire Design

The questionnaire explored ALs’ experience in using several of the key areas of

TutorHome. The areas covered:

general usability

working with student groups

working with course resources

regional information

support for working online

employment-related information.

Respondents were asked to indicate their agreement with statements on a five-point

scale, where five equalled Strongly Agree and one equalled Strongly Disagree. Each

area included an overall question.

Possible developments were also explored:

providing more facilities for working with student groups

expanding online recruitment facilities

providing more opportunities for participating in community.

Respondents were asked to indicate interest in new developments using a five-point

scale, where five equalled Very Interested and one equalled Not at All Interested.

Each content-specific section and a final section on overall impressions also provided

opportunities to make suggestions for improvement through open-ended questions.

Evaluation of TutorHome, Autumn 2004 -- 3

Summary – Overview of


Of the 2,148 ALs invited to participate in the survey, 622 completed the questionnaire,

for a response rate of 29%. Only 6% of respondents completed the paper version of the


61% of respondents indicated agreement with the statement, All in all, I find TutorHome

easy-to-use, by giving it a rating of four or five. The average rating for the question was


In evaluating aspects of TutorHome usability, respondents generally agreed that

TutorHome gave a positive impression of the OU but indicated some difficulty navigating

and remembering where to find resources and facilities that they had used before.

When asked to identify what activities they found most valuable, respondents placed

Getting my students’ contact information first (20% of responses). Downloading student

information (14%) and Getting my students’ course results (13%) came next. Looking

up information related to working with the OU and Claiming for expenses ranked last,

with 5% each.

The most common arrangement for working online was to use a dial-up modem at

home (42% of responses). Most other respondents used a broadband connection at

home (33%) or a computer at another workplace (23%).

When asked to compare their approach to finding information on TutorHome with

their approach to finding information on the Internet, respondents indicated a greater

tendency to hunt for specific information on TutorHome, along with a tendency to return

to sources that have helped in the past.

In making suggestions for the front page of TutorHome, respondents most often

mentioned design improvements and access to job vacancies, the eTMA system, and

course information.

Evaluation of TutorHome, Autumn 2004 -- 4

Summary -- Exploration of

TutorHome Areas

Among the overall questions for the TutorHome areas, Tutor Claims Online received the

highest rating (4.02), although it had been used by only 24% of respondents. Accessing

information related to my course and Faculty followed (3.88), having been used by 40% of

respondents. Working with my students received a rating of 3.74.

Regional TutorHome sites had been used by 38% of respondents and received the lowest

overall rating (3.32)

Responses indicated that, despite high overall satisfaction, Tutor Claims Online would

benefit from covering more types of expenses and claims. Comments also indicated that

its placement on TutorHome could be more prominent.

In answering questions about working with course resources, respondents indicated

potential room for improvement in communications with course teams, particularly in having

appropriate online connections. There was strong support for being able to access tutor

notes and marking guidelines online and for being able to access related courses’ websites

for background and ideas.

Their comments emphasised an interested in having more course information and contact

information available. While some respondents suggested improvements to FirstClass,

others preferred it over TutorHome.

In rating aspects of working with their student groups, respondents indicated that

changes to student details were clear and that information on students’ progress was

generally sufficient. There may be scope to improve pathways for support and referrals

and especially to provide more information on students’ history and circumstances along

with more information about communications being sent to students.

Comments included suggestions for improving student details, along with requests for more

assessment-related information and better support for handling queries.

Evaluation of TutorHome, Autumn 2004 -- 5

Summary -- Exploration of

TutorHome Areas (2)

In rating the University’s support for working online, respondents expressed high

confidence in responding to student e-mails and seemed relatively satisfied with the

support provided by the LTS Helpdesk. There is potential room for improvement in letting

students know what to expect from ALs and for providing stronger technical support in

using FirstClass and in developing general computing skills.

Suggestions emphasised providing more financial support for working online, with providing

equipment or support and providing more training also receiving frequent mention.

The 40% of respondents who had worked with the employment-related information on

TutorHome found it relatively easy to understand. The sub-site could do more to identify

how to get support with employment issues, and respondents do not yet seem to be relying

on it to find out about changes that may affect their appointments.

Comments clustered around improving the application process, providing more equipment

are support, and improving Tutor Claims Online.

Finally, respondents who had worked with the Regional information available through

TutorHome gave the highest rating to the sites’ role in alerting them to opportunities for

support and development. Their greatest difficulty lay in being able to tell when information

on the sites had changed.

Requests included indicating when changes had occurred, providing more contact

information, and providing more resources or facilities.

Evaluation of TutorHome, Autumn 2004 -- 6

Summary – Exploration of

Potential Developments

In rating their interest in new facilities for working with student groups, respondents

expressed greatest interest in having links for referring student queries. They expressed

least interest in opportunities to let the University know about their experiences and in

having more resources related to induction.

There was considerable interest in having expanded facilities to retrieve and submit

application forms online, with 62% of respondents giving a rating of 5, Very Interested.

There was a similar expectation that such a facility would include all tutoring roles.

Respondents indicated moderate interest in having opportunities to participate in

community. Being able to obtain more information about University research had the

greatest appeal, and being able to contribute news stories to TutorHome and other

University news channels had the least.

When asked what would help them feel valued, respondents commented that existing

discussion facilities suffice, that the University should recognise financial and time

constraints, and that improving communications opportunities would help.

When asked what would help them feel more connected to others, respondents mentioned

having more opportunities to make contact with other ALs, improving TutorHome, and

having opportunities to tell others about themselves. Others mentioned that the existing

discussion facilities suffice.

In comments identifying the biggest benefit of TutorHome, access to student information

received by far the greatest mention, with ease of use, providing a single point of access,

speed, and access to conferencing and e-mail also ranking high.

In suggesting the single most important improvement to TutorHome, respondents

identified further improvements to student facilities and information, along with FirstClass

improvements, improved technical operations, design improvements, navigational

improvements, better access to information, and facilitating contact with others.

Evaluation of TutorHome, Autumn 2004 -- 7

More magazines by this user
Similar magazines