AMEE Berlin 2002 Programme

0.05 - 0.13) to moderate (kappa: 0.51; 95% CI: 0.32 -
0.70) between various pairs of examiners. Agreement
between two endocrinologists assessing a case in
endocrinology was only fair (kappa: 0.21; 95% CI:
0.02-0.40). The results suggest unreliable (inter-rater)
grading in written clinical case examinations.
8D2 Identifying threshold competence
in an undergraduate qualifying
clinical examination
John Cookson* and Stewart Petersen
Leicester Warwick Medical School, Division of Medical
Education, Faculty of Medicine, P O Box 138, Leicester LE1
9HN, UK
Undergraduate qualifying examinations seek to identify
those who are competent to proceed to house officer
training. This requires definition of the standard of
competence required of the new graduate, and its
conversion to a threshold within an examination. We
have attempted to define the concept of the ‘minimally
competent’ practitioner – one who is just able to
proceed. The final professional clinical examination
at LWMS is graded in terms of five ‘categories of
competence’, with precise grade descriptors. We asked
consultants who train house officers to define the
minimally competent new graduate in terms of the
proportions of patients where they would not perform
satisfactorily in each category of competence.
Proportions ranged from 0% to 50%. There was a clear
tendency for history and examination skills to be
considered more important. The findings have
implications for setting examination thresholds.
8D3 The validity of performance
standards on a certification
examination for occupational
therapists
Ernest Skakun
Division of Studies in Medical Education, 2 J3000 WMC, Faculty
of Medicine and Oral Health Sciences, University of Alberta,
EDMONTON, Alberta, T6G 2R7, CANADA
This study investigated the validity of passing scores
set by the Nedelsky and Angoff methods for an
occupational therapy national certification exam. Eight
judges rendered Nedelsky and Angoff judgments for
302 multiple choice items. Judges used a five-point
scale to rate the competence of examinees selecting
each alternative as an answer. Judges’ decisions were
correlated with scores of examinees near the passing
scores. Item difficulties were computed for examinees
with scores just above (higher) and just below (lower)
the passing scores for each method. The number of
alternatives rated as “significant concerns about
competence” and selected as possible answers was
compared for higher and lower scoring examinees. The
correlation between judges’ decisions and examinee
scores around the passing score was 0.34 and 0.23 for
the Angoff and Nedelsky methods respectively.
Comparisons of proportion correct for higher and lower
candidates showed more consistency for the Angoff
method.
Wednesday 5 September
- 4.88 -
8D4 Reliability of a sequential clinical
examination
Stewart Petersen* and John Cookson
Leicester Warwick Medical School, Division of Medical
Education, Faculty of Medicine, P O Box 138, Leicester LE1
9HN, UK
Clinical competence of students at Leicester Warwick
Medical School is assessed by direct observation of
consultations with a series of patients. Summative
assessments are sequential. All students are observed
with 2 or 4 patients, (depending on stage in the course),
and those whose competence remains in doubt are
observed with a further 3 or 4. In each consultation, a
different pair of examiners observes students
continuously. Examiners grade independently
performance in five categories of competence defined
by precise descriptors. There are precise grade
descriptors. All examiners are trained. In a full, final
examination, therefore, each student has 16 grades for
each of five categories of competence. Analysis of data
from three full cycles of assessment demonstrates that:
(i) there is a high degree of inter-examiner reliability;
(ii) eight cases are sufficient to mitigate case specificity;
(iii) students vary in the relationship between
performance in different categories of competence.
8D5 Considerations of Legal, Ethical,
Organizational Aspect of Clinical
Competence in a National
Licencing Examination, Evaluation
at the Medical Council of Canada
Jacques E. Des Marchais*, Dale W. Dauphinee and
David Blackmore
12420 rue Joseph-Edouard-Samson, Montreal Quebec H4K
2N9, CANADA
Organizations responsible for development and
administration of national licensing examinations are
becoming major partners in responding to evolution
of emerging new social needs, such as Considerations
of Legal, Ethical and Organizational aspects of medical
practice(CLEO).The Medical Council of Canada
(MCC) has been responsible for developing an
examination process which leads to licensure to practise
in the Canadian provinces. The study presents the
decision-making process which has resulted in the
development of the CLEO content of the Medical
Council of Canada Qualifying Examination. The
validation process of this new content is demonstrated
as well as specific objectives in each of the three CLEO
components. Results are presented from the first
mandatory implementation, in May 2000, for about
2000 candidates. Comparison with the MCQ scientific
component is presented and discussed. Integration of
CLEO disciplines into the second component (OSCE)
of the MCC Qualifying examinations is also
demonstrated.
8D6 Combining different components of
an assessment procedure
J Degryse*, J Goedhuys, A Roex and A Dermine
ACHG K U Leuven, Department of General Practice,
Kapucijnenvoer 33 Blok J, B3000 Leuven, BELGIUM