PP654 UniSa Freney - Final Report Feb 2010.pdf - Office for ...

More documents

Recommendations

Info

Despite working towards the Shibboleth solution from an early stage in the project, itproved to be difficult to configure correctly for institutions other than UniSA as thisrequired cooperation between the lead institution and the external institutions.Fortunately, there was a project within the lead institution to implement Shibboleth forother projects, so the CAFAS project was able to capitalise on this via the expertise ofstaff working on this project 24 . Staff in the Information Strategy and TechnologyServices (ISTS), UniSA, liaised with their counterparts in external institutions to ensurethat Shibboleth was configured so that it was providing the necessary data forauthentication to CAFAS.Although access to CAFAS Online for The University of Adelaide and the QueenslandUniversity of Technology was eventually confirmed, no trials of CAFAS were conductedexternally due to many delays caused by repeated problems with Shibboleth andintermittent problems with CAFAS itself.With the benefit of hindsight, a different authentication system might have facilitatedaccess for academics from other institutions. Indeed a work-around solution wasdeveloped by the CAFAS team to bypass Shibboleth when it was clear that Shibbolethissues were going to delay a critical trial. However, in the long term, the open-sourceShibboleth system should prove to be an effective authentication system as it gainsuse throughout the tertiary education sector nationally and internationally. What hasbeen realised is that it is very useful to have the ability to offer different solutions whenone solution fails and that engagement with key ICT staff in collaborating institutions isessential and these links should be fostered early in the project to ensure earlyidentification of issues and rapid resolution thereof.9.0 Applicability to Other InstitutionsApplicability of OutcomesThe main outcome of the project, the open-source online feedback & assessmentsystem, is specifically designed to meet the needs of a variety of institutions andlocations.The functionality of CAFAS Online caters to the assessment and feedbackrequirements of a wide variety of disciplines and institutions via various options in theway it is configured for a certain institution and the options available to academics indesigning a feedback form. For example, the first action a first time user of CAFASOnline must take is to specify which institution they are from via a drop down list ofinstitutions (refer Appendix F). Accordingly, the user interface is reconfigured so thatthe terminology of the user’s institution is used, thus the term “course” would change to“unit” or “subject” as appropriate for the institution selected. More importantly, thegrade/mark scheme and graduate attributes specific to the institution are referencedfrom the CAFAS database and utilised and remembered for each user. CurrentlyCAFAS Online is configured for use in five institutions.Consultation with learning and teaching experts in 10 institutions 25 established the keyfunctionality required by the system. As a result, there are a variety of options available24 Authentication and Authorisation Infrastructurehttp://wiki.unisa.edu.au/display/AAI/Sites accessed 10/12/09.25 Australian Catholic University, Central Queensland University, Charles DarwinUniversity, James Cook University, Macquarie University, Royal Melbourne Institute ofComputer Aided Feedback & Assessment System (CAFAS) 38
to the academic when designing a feedback form, which widens the applicability of thesystem to a great variety of disciplines, assessment regimes and personal preferences.CAFAS is a criterion referenced system that is typical assessment regime in manydisciplines. It uses weighted assessment criteria to provide a summative assessment(mark/grade) in conjunction with formative feedback comments associated with eachassessment criterion. This combination of summative and formative feedbackmechanisms can be displayed as a Slider (sliding scale mechanism) or a Rubric(matrix/table) to communicate to the student their performance level/outcome in aparticular assessment criterion. This mimics the format and mechanisms used intraditional paper based feedback forms 26 but adds the functionality and efficiency of adigital system.A review of the system by Professor Geoffrey Crisp, Director Online Education, TheUniversity of Adelaide, indicates that the basic premise of the system is valid and thatdifficulties with the authentication system (Shibboleth) was the main impediment duringhis evaluation of the system:The CAFAS system developed at UniSA is a potentially useful tool for coursecoordinators, teachers and students. There is an extensive series of trainingvideos that make it very clear how one might effectively use the system. Thissystem would be particularly useful for large classes and would facilitate theprovision of timely feedback to students. It allows quite a bit of freedom forteachers to include custom Rubrics, criteria, weightings, as well as links toresources as part of the feedback to students. The system incorporates goodtools for the analysis of student responses so that teachers can make informedjudgements about the efficacy of their assessment tasks. I would recommendthis system to teachers for trials, especially where large classes are involved.The main problem I had with using the system was the Shibboleth authenticationsystem, which was not CAFAS itself, but the ability of different universitysystems to communicate and pass on authentication information. Although thiswas not CAFAS itself, it did affect use of the system as it was configured duringmy trials.Professor Geoffrey CrispDirector, CLPD; Director, Online Education2009 ALTC National Teaching Fellow; HERDSA President and HERDSA Fellow;2009 ASCILITE FellowTwo colleagues who had agreed to trial the system, but were prevented due totechnical problems, have indicated that they plan to use the system in 2010. TimWilliams, Lecturer Industrial Design, Queensland University of Technology and JoyMcEntee, Lecturer, Discipline of English, The University of Adelaide, have been inregular communication with the project leader regarding trials of the system. The keyfunctionality of the system has been communicated to these colleagues, via video clipsand written documentation, who have ascertained that it will be suitable for theirrequirements.Technology, Southern Cross University, Swinburne University of Technology, TheUniversity of Adelaide, University of Notre Dame Australia.26 Learning and Teaching Unit, UniSA, Examples of Assessment Feedback Forms,http://www.unisanet.unisa.edu.au/learn/learningconnection/?PATH=/Resources/PD-OT/Assessment+feedback+forms/&default=Introduction.htm accessed 10/12/09.Computer Aided Feedback & Assessment System (CAFAS) 39
Page 1 and 2: Report - 2010COMPUTER AIDED FEEDBAC
Page 4 and 5: 1.0 Executive SummaryThe primary ob
Page 6 and 7: 2.0 OutcomesAs stated in the origin
Page 9 and 10: more understandable to the user. Cu
Page 12 and 13: later (in conjunction with formativ
Page 14 and 15: Generic descriptors of the institut
Page 16 and 17: At the conclusion of the assessment
Page 18 and 19: Thus there is the potential to buil
Page 20 and 21: institution.Product Collate researc
Page 22 and 23: A presentation at the ALTC Assessme
Page 24 and 25: These adjustments to the approach e
Page 27 and 28: Overall there were eight different
Page 29 and 30: CAFAS 7 . The students appreciated
Page 31 and 32: difficulties described previously,
Page 33 and 34: ALTC funded project), demonstrated
Page 35 and 36: grades based on their professional
Page 37: Quantification of the Programming T
Page 41 and 42: The staged trials of the system wer
Page 43 and 44: Colleagues from the Queensland Univ
Page 45 and 46: within the timeframe and the availa
Page 47 and 48: Assessment, University of South Aus
Page 49 and 50: 3. Installation of SoftwareWith ref
Page 51 and 52: If you have any comments or suggest
Page 53 and 54: e.g. Literature Review - Assignment
Page 55 and 56: to students to inform them of their
Page 57 and 58: Computer Aided Feedback & Assessmen
Page 59 and 60: Appendix D - Staff Questionnaire Re
Page 65 and 66: Appendix E - Comments Database Scre
Page 67 and 68: Appendix G - System SpecificationsC
Page 69 and 70: Specifications - Slider & RubricImp
Page 71 and 72: possibility of including hyperlinks
Page 73 and 74: Grade and Mark for Assignment 1Grad
Page 75: Fig 4Building on the previous asses

PP654 UniSa Freney - Final Report Feb 2010.pdf - Office for ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?