awej 5 no.4 full issue 2014

AWEJ Volume.5 Number.4, 2014Fair, Reliable, Valid: Developing a Grammar Test UtilizingEfeotorMike OrrThe Item DesignThe next step in the test development process is to find a way ―to stimulate responses that canconstitute observations about the construct that the measurer wishes to measure‖ (Wilson, p.41).The aim is to operationalise dimensions of the construct into items that give an accuraterepresentation of the ability of the participant. In order for the test to be reliable it must give anaccurate indication of the ability of the student. Test reliability is discussed in detail later in thepaper. However, it is worth noting that there were several decisions that had to be made with theintention of producing a reliable test. The first of these was the test item format that would beused. As Downing (2006a) purports, ―The creation and production of effective test questions,designed to measure important content at an appropriate cognitive level, is one of the greatestchallenges for test developers‖ (p.10). There are many different formats that a test can adopt.Perhaps the two most common type of item are the open-ended item format and the fixedresponseformat (Wilson, 2005). The choice of which format to use depends largely on theconstruct. It is pertinent to choose the format that will enable operationalisation of the construct.There are various advantages and disadvantages of each format. The open-ended item format hasbeen praised (RePass,1971; Iyengar, 1996) as it enables the examinee to express his thoughts,without giving any prompts or cues. Moreover, this format allows for detailed responses tocomplex issues, as well as revealing the examinees logic and reasoning. However, open-endeditems do have some shortcomings. Firstly, they may not be responded to due to ineloquencerather than indifference; examinees may not respond to open-ended questions because they areunable to express their thoughts. Furthermore, open-ended items can be difficult to grade, whilenecessitating a certain degree of subjectivity.The fixed-response format is often used as multiple-choice items or a Likert-type response scale.In contrast to the open-ended format, the fixed-response format is easier to grade, as possibleresponses are limited. The objective nature of the test limits scoring bias. In addition, studentscan potentially respond to many items which can permit wide sampling. When conducting astudy and asking for voluntary participation, requesting long open-ended responses can deterwould-be participants, whereas they may be more willing to answer fixed-response items.There are also disadvantages of the fixed-response format. Firstly, there is the criticism thatmultiple-choice tests teach misinformation (Toppino & Brochin, 1989; Rees 1986), can suggestthe answer to the examinee, and allow for guessing. The use of item-person matrixes canhowever highlight possible occurrences of guessing. Another shortcoming of fixed-responseitems is that it is difficult to measure certain learning outcomes such as displaying thoughtprocesses and articulating explanations. The progression of some constructs may bedemonstrated by higher-order thinking skills, such as those found in Bloom‘s Taxonomy. It is alot harder to demonstrate analysis, synthesis and evaluation with fixed-response items. Despitethese shortcomings, the fixed response format was favourable for the purpose of testinggrammar. This is due to the nature of grammar itself. It would be very difficult for an openendeditem to specify to the examinee what grammatical structure was required of them. Onepossible way would be to rely on the use of grammatical terminology. For example, one may askan examinee to answer a question using the present perfect continuous tense. The problem withthis approach is that it would not be testing the construct, solely, but also their knowledge ofArab World English JournalISSN: 2229-9327www.awej.org208