October 2012 Volume 15 Number 4 - Educational Technology ...

Zualkernan, I. A., Raza, A., & Karim, A. (2012). Curriculum-guided Crowd Sourcing of Assessments in a Developing Country.
Educational Technology & Society, 15 (4), 14–26.
Curriculum-guided Crowd Sourcing of Assessments in a Developing Country
Imran A. Zualkernan *1 , Anjana Raza 2 and Asad Karim 3
1 Computer Engineering, American University of Sharjah, PO BOX 26666, Sharjah, UAE// 2 Development in
Literacy, House 15, Street 29, Sector F-10/1, Islamabad , Pakistan // 3 TeleTaleem Ltd., Plot 291, Street 3, Sector I-
9/3, Industrial Area, Islamabad, Pakistan //izualkernan@aus.edu // anjana@dil.org // asadkarim@iphonica.com
*Corresponding author
ABSTRACT
Success of Wikipedia has opened a number of possibilities for crowd sourcing learning resources. However, not
all crowd sourcing initiatives are successful. For developing countries, adoption factors like lack of
infrastructure and poor teacher training can have an impact on success of such systems. This paper presents an
exploratory study to determine if teachers in a developing country are able to create quality multiple-choice
questions for primary school students. An adoption model is developed and evaluated to ascertain if the teachers
would actually contribute to such a Wiki. Results are that, given student learning outcomes, content constraints,
and a Bloom’s assessment level, a reasonable number of teachers were able to formulate quality questions, and
that there is a strong intention to use such a system. Teachers with high intention to adopt also had a better
attitude, enjoyed making questions and found the process easy to use. However, there is no obvious relationship
between the intention to use and an ability to pose good assessments. In addition, there is no obvious predictor
of where the good question contributors came from.
Keywords
Crowd sourcing, Wiki, Developing world, Primary education, Online assessments
Introduction
According to UNESCO, 67 of the 142 countries surveyed had a moderate to severe primary teacher gap, and the
countries with gaps were primarily developing countries (The Global Demand for Primary teachers – 2010 Update.
Information Sheet no. 5, 2010). Enabling educational technologies can help obviate this teacher gap. For example,
given the success of Wikipedia, it is tempting to consider a crowd sourcing platform that provides automated
assessment support for primary school teachers. Crowd sourcing is not new to the developing world and platforms
like Ushahidi have been successfully used in developing countries for citizen journalism (Heinzelman & Waters,
2010). Similarly, in Nigeria, Wiki has been used in very large classes (> 2000 students) with a positive impact
(Aborisade, 2009). Unfortunately, not all crowd sourcing experiments are as successful as Wikipedia. For example, a
recent study showed that only a small percentage of Wikis exhibited healthy growth (Kittur & Kraut 2010). Culture
is another important variable. For example, using various measures of cultural influences, Pfeil, Zaphiris, & Ang
(2006) found differences in patterns of Wiki contributor’s behaviour across cultures. A study of teachers using Wiki
shows that teachers were frustrated by their own materials being modified by others, and were afraid of criticism
from others (Lio, Fraboni, & Leo, 2005). In addition, teachers were reluctant in giving up ownership of materials and
had a fear that they may break the whole system by their changes. Lio et al. (2005) also observed that teachers were
hesitant in changing others’ work for fear of offending, and were somewhat put off by the open-ended nature of the
Wiki; the fact that the materials they created would keep changing.
Another pertinent observation is that a crowd sourcing model entirely driven by interests of the contributors does not
necessarily guarantee adequate curriculum coverage for formal learning scenarios. For example, Halavais & Lackaff
(2008) and Zhang, Sun, Datta, Chang & Lim (2010) show that topical coverage in crowd sourced systems like
Wikipedia, is different than that of expert-created representations. Another related phenomenon is the “long tail”
behaviour of users of Wikipedia (Riedl, 2009); a small number of topics on Wikipedia are visited most often. This
small proportion of topics also has higher quality as well (Wilkinson & Huberman, 2007). Taken together, the
implication is that without adequate guidance from a curriculum, a crowd sourced Wiki may have skewed coverage
in terms of content and quality. Therefore, a crowd sourcing solution for assessments must explicitly incorporate the
curriculum. This creates an essential tension between “democratic” spirit of crowd sourcing and the curricular needs
of a particular community. Any realistic deployment of a crowd sourcing solution to learning in general must
negotiate this tension successfully.
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the
copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by
others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.
14