PP654 UniSa Freney - Final Report Feb 2010.pdf - Office for ...

Both respondents “strongly agreed” with the statement “Now that I have experiencedCAFAS I see it as a viable option to paper-based feedback.” Likewise, bothrespondents indicated that in future they would prefer to use CAFAS as a replacementto paper-based feedback.Some criticisms included a lack of a spell checking function and some confusion aboutthe correct procedure for adding and editing comments in the database.Minor criticisms regarding functionality issues were also reported informally to theproject leader. These suggestions for functionality improvements have been logged onthe “Enhancement Request” page of the CAFAS Project website 11 and it is anticipatedand hoped that these minor problems will eventually be overcome through collaborativeefforts by the open-source programming community, or through support from anothertertiary institution.The poor response rate (sample of only two) makes it difficult to draw strongconclusions, however, the fact that all staff involved in the trials persisted with CAFASfor at least two assignments and one (not including the project leader) persisted forover a year, indicates that they could all see some merits in continuing to use it despitesome of the frustrations of some lacking functionality such as a spell checker. Alsosupporting this argument is the fact that approximately 10 staff are still using theprototype (Excel based) version of CAFAS.Discussion of ApproachThe use of a public online survey system, Survey Monkey, was chosen for theevaluation as it offered the possibility of creating a central repository for all responsesregardless of which course/unit/subject or institution was involved. This facilitated datacollection and simplified and standardised the means for administering the survey inother institutions, overcoming the need for colleagues to attend to this administrativetask (although this benefit was not realised due to lack of external trials, it is mentionedto rationalise the decision to use it). Another benefit of Survey Monkey was the manyoptions for responding to questions, no limitation on the number of questions andfreedom with scheduling the questionnaire.A low response rate from staff indicates the need to review the approach for obtainingtheir feedback. A catered meeting at which morning tea is served might improveattendance and would be an excellent forum for brainstorming innovations to thesystem. A paper based questionnaire could be handed out for completion at themeeting to ensure a high response rate. Thus funds for catering at meetings and fordata entry of paper based questionnaires should be included in budgets.A difficulty with the questionnaires was establishing a protocol for cooperating withother institutions in terms of running trials and the associated external evaluation.Initially the team’s discussions revolved around whether or not the respectivePVC/DVC Academic needed to give authorisation or whether individual academicscould be contacted directly and supplied with the approved Human Research EthicsCommittee documentation (from the lead institution) for consideration by theirrespective ethics committees. It was decided that the later approach would beacceptable and would encourage participants from other institutions to becomeinvolved due to the minimisation of paperwork. Ultimately however, due to the11 CAFAS Website http://cafas.pbworks.com/Enhancement-Requests accessed12/1/10.Computer Aided Feedback & Assessment System (CAFAS) 30