ONE, ALL OR NONE OF THE ABOVE Multiple-choice question ...

Disparate Measures in Examinations for STCW
Certificates of Competency: The Use and
Effectiveness of Multiple Choice Questions
by
Denis Drown
Robert Mercer
Gary Jeffery
Stephen Cross
St. John’s, Newfoundland,
Canada

Whereas:
What it’s all about
Because:
Then:
MCQ are widely used in MET and Assessment
Use of MC is increasing
Formal training in MC construction is the exception
Many factors influence MC scores
The more these influences are understood and
mitigated the more effective is MC Assessment
Presentation:-
2

Definitions
Public attitudes
Student attitudes
Student Mariner: a student in a second or third year of
education and training towards a STCW certificate of
competency as a Navigating or Engineer Officer.
Student Novice: a student of comparable age and level of
education and English language comprehension to the
student mariner, but with no marine training or experience,
studying a subject other than maritime technology.
4

MCQ Events
1978: Course revisions following STCW „78
1996: Employment screening
2000: Experiment with office staff (10 persons)
2002: Experiment with „focus group‟ (6 persons)
2005: First trial study (35 Middle &19 High School)
2006: Survey & Study (53 countries) (Marseille)
2007: Second trial study (157 test subjects)(Rotterdam)
2009: Full-scale study (930 test subjects)(Szczecin)
5

insufficient data
50%
NOVICES - DECK QUESTIONS
T
e
s
t
45%
40%
S
c
o
r
e
s
(
a
v
)
35%
30%
25%
20%
15%
10%
Males Females 25 1st 2nd V Good Good Not Good Primary Secondary P Sec Today
Gender Age English Comprehension First MC Experience
6

BANANAS
How are cultivated bananas propagated
A. By tissue culture
B. By plant offshoots
C. By transplanting underground stem roots
D. By seeds
Give one reason for your answer
Cultivated bananas are parthenocarpic which
makes them sterile and unable to produce viable
seeds.
7

Ongoing Research
Knowledge
Deduction
Word/Grammar Clue
Word/Concept Association
Guessing
Intuition
8

Question
Structure
Instructor Training in Item
construction
Guessing &
Chance
Testwiseness (cracking the code)
MC Test Experience
Intuition
SUMMARY OF MCQ INFLUENCES
Language Skills
Lefthandedness
Cheating
Gender & Age
Examiner preferences
Specimen examinations
General strategies
Changing responses
Student Learning Styles/Skills
Other
Computer
familiarity
Probabilities & Mind Games
impulsivi
9

Qu. Sketch a half section double
bottom A and describe the main
construction features.
Figure 2
8. In Figure 2, of a half section double bottom, A is
indicating a:
A. Bracket, used to increase strength
B. Margin plate, used to increase strength
C. Lightening hole, used to reduce weight
D. Limber hole, used to reduce weight
10

“Upon what meat doth this, our MC, feed, That it is grown
so great”
Cheap
Easy
Quick
Simple
Objective
11

Conclusion & Solution
• MCQ have a place in checking factual knowledge and are
effective assessment tools where there is dialogue between
instructor and student.
• MCQ in STCW competency examinations should be variants
and a minor element in a multi-dimensional examination
process, with effectiveness evaluated during an interview.
• “Disparity” fosters doubt and doubt compromises confidence.
The first step to a solution is to gather detailed and
investigative data about maritime examination methods. Such
a study requires an international and institutional approach.
12

Final Thoughts
29 Jul 2008, The Times: Dyslexic (medical) student to fight multiple
choice questions in Court.
21 Jan 2010, Washington Post: Medical Student’s (Court) Appeal
Highlights Challenges of Dyslexia.
23 Feb 2010: Sydney Coroners Court findings following a fatal
collision in Sydney Australia Harbour on 29 Mar 2007
13

“The school don‟t teach me proper”
My client was not properly found
competent by the training provider/
authority responsible for issuing the
certificate, because the examination by
which he was assessed was not properly
constructed, neither has it been shown to
be valid or reliable.
14

Disparate Measures in Examinations for STCW
Certificates of Competency: The Use and
Effectiveness of Multiple Choice Questions
by
Denis Drown
Robert Mercer
Gary Jeffery
Stephen Cross
THE END

End
16

Definitions
Public attitudes
Student attitudes
Student Mariner: a student in a second or third year of
education and training towards a STCW certificate of
competency as a Navigating or Engineer Officer.
Student Novice: a student of comparable age and level of
education and English language comprehension to the
student mariner, but with no marine training or experience,
studying a subject other than maritime technology.
17

What does the term 'head' mean when applied to a fire pump
A. Length of the discharge pipe
B. Height of the discharge pipe
C. Difference between the discharge and suction pressures
D. Sum of discharge and suction pressures
Answer C. I‟m thinking that because options C and D are both
dealing with pressure and there‟s a subtle difference between the
two that one of them is probably correct. Again, I think that
because discharge and suction may be working against each other
that C could be the correct answer. It sounds unlikely that you
could combine discharge and suction. But if not C, then I think D is
the correct one. But probably C again because it‟s longer than the
others – this is a dead giveaway.
18

MCQ Use – Summary
Maritime Administrations
A-II/1 16 A-II/2 19 A-III/1 15 A-III/2 16
Bangladesh 16 USA 19 USA 14 USA 15
Japan 15 Bangladesh 18 Japan 12 Bangladesh 12
USA 14 Japan 18 Bangladesh 10 Chile 9
Australia 11 Australia 14 Canada 8 Japan 9
Canada 9 Canada 11 Chile 8
METS Acting for Maritime Administrations
Philippines 16 China PR 19 China PR 15 China PR 16
China PR 15 Korea Rep 19 Korea Rep 15 Korea Rep 16
Korea Rep 15 Philippines 19 Philippines 15 Philippines 14
Hong Kong 9 Hong Kong 10
Netherlands 8
19

DECK QUESTIONS - Frequency of Correct Responses ≥50%
Test #
Study Test Question #
Basic Questions
Advanced Questions
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Percentage ≥ 50% Correct in Each Test
Novices
Mariners
92 85 62 62 54 85 69 69 54 69 77
4D 11 10
87 77 87 84 90 97 77 61 71 52
Novices: University, English Language & Literature: English Second Language.
Student Mariners: Second Year Navigation: English Second Language.
Question 10: Drinking salt water will
A. dehydrate you
B. prevent seasickness
C. be safe if mixed with fresh water
D. protect against heat cramps
Question 20: Which is an indication of reserve buoyancy
A. Metacentric height
B. Righting moment
C. Freeboard
D. Rolling period
20

MCQ Events
1978: Course revisions following STCW „78
1996: Employment screening
2000: Experiment with office staff (10 persons)
2002: Experiment with „focus group‟ (6 persons)
2005: First trial study (35 Middle &19 High School)
2006: Survey & Study on international use (Marseille)
2007: Second trial study (157 test subjects)(Rotterdam)
2009: Full-scale study (930 test subjects)(Szczecin)
23

Question 10: Drinking salt water will
A. dehydrate you
B. prevent seasickness
C. be safe if mixed with fresh water
D. protect against heat cramps
Question 20: Which is an indication of reserve buoyancy
A. Metacentric height
B. Righting moment
C. Freeboard
D. Rolling period
24

“MCQ are not used in professional marine
examinations administered by or on behalf of the
(UK) Maritime & Coastguard Agency, and
presently there is no intention of introducing such
a system. There is interest in other countries'
applications of MCQ with respect to the
robustness of the technique and the validity of its
results.”
(Permission) 7 July 2006
25

Conclusion & Solution
• MCQ have a place in checking factual knowledge and are
effective assessment tools where there is dialogue between
instructor and student.
• MCQ in STCW competency examinations should be variants
and a minor element in a multi-dimensional examination
process, with effectiveness evaluated during an interview.
• “Disparity” fosters doubt and doubt compromises confidence.
The first step to a solution is to gather detailed and
investigative data about maritime examination methods. Such
a study requires an international and institutional approach.
26

The Debate Continues
You may be right – eventually. Times (Leader) 13 Jun 2006.
Narrow testing It’s the American way. E. Jackson, TES,16 Jun 2006.
Multiple choice is not a dumb option. M. Cresswell, TES, 23 Jun 2006.
Girls much keener than boys to go to University. A. Frean, Times, 28 Aug 2007
GCSE science questions require no scientific knowledge. N, Woolcock,
Times, 6 Jul 2009.
Schools are producing a generation of illiterate scientists. E. McCracken,
Herald Scotland, 28 Mar 2010
Exams for boys, exams for girls. W. Stewart, TES, 8 Jun 2010.
Dyslexic student to fight multiple choice questions in court. J. Sugden, Times, 29
Jul 2008.
Medical Student’s Appeal Highlights Challenges of Dyslexia. Washington
Post, 21 Jan 2010.
27

A shipboard Oil Pollution Emergency Plan is required of
A. all vessels, regardless of size and commercial application
B. any barge or ship constructed or operated in such a manner
that no oil in any form can be carried aboard
C. an oil tanker of 150 gross tons and above, or other ship of
400 gross tons and above
D. an oil tanker of 400 gross tons and above, or other ships of
150 gross tons and above
B sounds garbled and doesn‟t really make sense. A doesn‟t make
sense as it would be ridiculous if this applied to canoes etc. So it
must be C or D. I‟m thinking C is the correct answer as this would
be essential for even smaller oil tankers – they can still create an
environmental hazard.
28

Tips for Taking Multiple Choice Tests
by Dr. Robert Runte
Faculty of Education, University of Lethbridge
“ even though students do not know any of the course
content, or have any grasp of the meaning of the
question, they can still score 100% on this quiz if they
are test-wise.”
3. Bogh qo‟ rar zorkrans inkle
A. gi hien
B. gis inkle
C. gish frankel
D. gishen fronks
29

7. Small dry cargo vessels are frequently framed to give
longitudinal strength in in the the deck and double bottom as
well as well as as transverse strength strength in the in ship‟s the ship‟s side. side. This
This
system of framing is known as a ________________
system of framing.
A. Combination
B. Longitudinal
C. Oblique
D. Transverse
30

Which lights are shown by a NUC
vessel, making way
Draw the lights of a NUC vessel,
starboard side, making way.
A
B
C
D
31

Education
Level
Age
Number
of
Participants
Mean
Score:
Range
0-7
Middle
School
High
School
Career
Academy
11-15 35 1.0
17-18 19 2.5
18-24 81 3.1
University 18-24 76 4.0
32

A B C
STCW
Mandatory minimum requirements for certification
A-II/1: I/C NAVIGATIONAL WATCH SHIPS ≥ 500 GT
Competence
Plan and conduct a passage and determine position
Maintain a safe navigational watch
Use radar and ARPA to maintain safety of navigation
Respond to emergencies
Respond to a distress signal at sea
Standard Marine Navigational Vocabulary (IMO)
Visual signalling
Manoeuvre the ship
Monitor cargo operations
Inspect and report defects/damage
Compliance with pollution prevention regulation
Maintain seaworthiness of the ship
Prevent, control and fight fires on board
Operating life-saving appliances
Written Examination
Scoring/Grading
Method
Essay Calc. MCQ
Other
Method
Apply medical first aid on board ship
Monitor compliance with legislative requirements
34

Essay
Calc
MCQ
Other
Essay
Calc
MCQ
Other
STCW 95
Competencies 1
A-II/1
A-II/2
16 19
Exam Method 2
Australia 8 3 0 11 14 12 4 14 16
Bangladesh 16 4 16 16 19 9 18 19
Canada 5 3 9 15 9 3 11 19
Malaysia 12 2 0 13 17 4 0 18
Netherlands 3 1 8 13 2 4 5 11
Philippines 3 0 16 12 5 3 19 12
Sri Lanka 14 3 4 15 17 5 2 13
United Kingdom 10 2 0 16 17 7 0 19
USA 0 5 14 14 0 6 19 18
Means 4 8 2 5 13 10 4 5 14
Survey Results - Maritime Administrations
35

MCQ Use – METS Acting for Maritime Administrations
A-II/1 16
Philippines 16
China PR 15
Korea Rep 15
Hong Kong 9
Netherlands 8
Netherlands 6
Ukraine 5
Sri Lanka 4
Vanuatu 4
Denmark 2
Sweden 1
Iceland 0
Japan 0
Malaysia 0
Sweden 0
Trinidad 0
UK 0
Vietnam 0
A-II/2 19
China PR 19
Korea Rep 19
Philippines 19
Ukraine 6
Netherlands 5
Netherlands 4
Sri Lanka 2
Hong Kong 1
Denmark 0
Iceland 0
Japan 0
Malaysia 0
Sweden 0
Sweden 0
Trinidad 0
UK 0
Vanuatu 0
Vietnam 0
A-III/1 15
China PR 15
Korea Rep 15
Philippines 15
Hong Kong 10
Netherlands 5
Netherlands 5
Sri Lanka 4
Malaysia 2
Denmark 1
UK 1
Iceland 0
Japan 0
Sweden 0
Sweden 0
Trinidad 0
Ukraine 0
Vietnam 0
A-III/2 16
China PR 16
Korea Rep 16
Philippines 14
Ukraine 5
Japan 4
Netherlands 4
Netherlands 3
Sri Lanka 1
UK 1
Denmark 0
Hong Kong 0
Malaysia 0
Sweden 0
Sweden 0
Trinidad 0
Vanuatu 0
Vietnam 0
36

Essay
Calc
MCQ
Other
Essay
Calc
MCQ
Other
STCW 95
Competencies 1
A-III/1
A-III/2
15 16
Exam Method 2
China PR 0 0 15 0 0 0 16 0
Denmark 6 3 1 12 9 5 0 13
Germany
Hong Kong China 0 0 10 13 14 2 0 10
Japan 15 0 0 15 15 0 4 16
Korea Republic 0 0 15 13 0 0 16 11
Sweden 1 0 0 15 3 2 0 16
Trinidad and Tobago 3 0 0 15 5 1 0 16
Ukraine 15 0 0 11 12 2 5 11
United Kingdom 6 2 1 10 10 4 1 6
Vietnam 10 0 0 9 14 0 0 6
Means 4 3 0 2 7 5 1 2 6
Survey Results - METS
37

STUDY ITEM
1. Development
What are the procedures for planning, designing and preparing tests
How much time is allocated to development
2. Determination of cognitive levels
How are learning styles, personal strengths and weaknesses recognised
Are formal ability and aptitude tests used to determine cognitive level
11. Training of MCQ constructors
What are the procedures for training test constructors
What time is allowed for test writing, standardizing and validating
12. Popularity of MCQ
What are the advantages and disadvantages of MCQ tests
13. Other items of interest
14. Experiences
Study Extract
38

Intuition
Sarnacki defines intuition as “a subject‟s capacity to utilize
the characteristics and formats of the test and/or the testtaking
situation to receive a high score”, with a definitional
addendum that “testwiseness is logically independent of the
examinee‟s knowledge of the subject matter for which the
items supposedly measure”.
39

The Study
• Background: The IMLA 14 study and survey, and the IMEC 19 exploratory
study, both described in the Paper.
• Supposition: If persons (novices) are given a MCQ test on a subject with
which they are unfamiliar then scores must reflect factors other than subject
knowledge.
• Instrument: 20-question marine technology tests drawn randomly from a
1500 MCQ pool contributed by METS and maritime administrations.
• Method: 44 individual tests completed by 930 mariner and novice study test
subjects from N America, Europe, Asia and the Subcontinent. Mariner and
novice scores are compared.
• Objective:To provide insights into the relative significance of gender, age,
English comprehension and previous test experience so that these factors
may be considered when constructing MCQ items.
40

Mean of 20 Questions
12.0
Mariner & Novices - Deck Questions
10.0
8.0
6.0
4.0
2.0
Mariners
Novices
0.0
1 2 3 4 5 6 7 8 9 10 11 12 13
Table 1D: Test 1 - 13: Questions 1 - 20 (Basic & Advanced)
41

.
Conclusions
• MCQ tests are useful in-course assessment tools.
• Factual-based MCQ are ineffective as part of an examination process leading
to a STCW certificate of competence, because MCQ cannot reliably assess
subject knowledge, and are not designed to evaluate competence.
• MCQ should be constructed clearly and explicitly with attention to the
principles of Maritime English.
• If MCQ are used in STCW competency examinations they should be variants
and a minor element in a multi-dimensional process. Variants require
resources for construction, validation, and marking. Considering the possible
consequences of incompetence, these resources are a good investment in
a safe maritime transportation industry.
42

insufficient data
T
e
s
t
50%
45%
40%
NOVICES - DECK QUESTIONS
S
c
o
r
e
s
(
a
v
)
35%
30%
25%
20%
15%
10%
Males Females 25 1st 2nd V Good Good Not Good Primary Secondary P Sec Today
Gender Age English Comprehension First MC
1 – 10
11 – 20
All
43

MCQ Use – Summary
Maritime Administrations
A-II/1 16 A-II/2 19 A-III/1 15 A-III/2 16
Bangladesh 16 USA 19 USA 14 USA 15
Japan 15 Bangladesh 18 Japan 12 Bangladesh 12
USA 14 Japan 18 Bangladesh 10 Chile 9
Australia 11 Australia 14 Canada 8 Japan 9
Canada 9 Canada 11 Chile 8
METS Acting for Maritime Administrations
Philippines 16 China PR 19 China PR 15 China PR 16
China PR 15 Korea Rep 19 Korea Rep 15 Korea Rep 16
Korea Rep 15 Philippines 19 Philippines 15 Philippines 14
Hong Kong 9 Hong Kong 10
Netherlands 8
Individuals with Maritime Interests
Philippines 16 Greece 19 Philippines 14 Philippines 14
Greece 15 Philippines 19 Greece 11
Poland 15 Poland 18
Singapore 9
44

BANANAS
How are cultivated bananas propagated
A. By tissue culture
B. By plant offshoots
C. By transplanting underground stem roots
D. By seeds
1500 Study Test MCQ - Response Count
A B C D
26% 25% 22% 27% 100%
Give one reason for your answer
Cultivated bananas are parthenocarpic which
makes them sterile and unable to produce viable
seeds.
45