October 2012 Volume 15 Number 4 - Educational Technology ...

Chang, C.-C., & Wu, B.-H. (2012). Is Teacher Assessment Reliable or Valid for High School Students under a Web-based
Portfolio Environment?. Educational Technology & Society, 15 (4), 265–278.
Is Teacher Assessment Reliable or Valid for High School Students under a
Web-based Portfolio Environment?
Chi-Cheng Chang 1* and Bing-Hong Wu 2
1 Department of Technology Application and Human Resource Development, National Taiwan Normal University,
Taiwan // 2 Department of Data Processing, Taipei Municipal Sung-Shan Senior High School, Taiwan //
samchang@ntnu.edu.tw // binghong.wu@gmail.com
* Corresponding author
ABSTRACT
This study explored the reliability and validity of teacher assessment under a Web-based portfolio assessment
environment (or Web-based teacher portfolio assessment). Participants were 72 eleventh graders taking the
“Computer Application” course. The students perform portfolio creation, inspection, self- and peer-assessment
using the Web-based portfolio assessment system; meanwhile, the teachers used the assessment tool to review
students’ portfolios and evaluate their learning performances. The results indicated that: 1) the Web-based
portfolio teacher assessment achieved an acceptable level of reliability; 2) the Web-based portfolio teacher
assessment, showing a strong level of inter-rater reliability and inner-rater reliability, can be regarded as a
reliable assessment method; 3) the Web-based portfolio teacher assessment demonstrated an acceptable level of
validity; 4) the portfolio scores were highly consistent with students’ end-of-course examination scores,
implying that Web-based portfolio teacher assessment is a valid assessment method. Future studies are
recommended to gain further insight into the self-built portfolios created by digital tools (e.g. PowerPoint,
Word, or Frontpage software), for which effective rubrics and reliability or validity of the assessment may be
also provided.
Keywords
Web portfolio, e-Portfolio, Portfolio assessment, Reliability, Validity
Introduction
Despite large time commitment required from both faculty and students, learning portfolios not only have positive
influences on the quality as well as process of learning, but also offers a number of benefits not available as using
conventional assessment approaches (Zalatan, 2001). Traditional paper-based portfolios are relatively ineffective in
terms of inquiry, update, storage and management with particular difficulties in documenting various types of
information, e.g. audio and video. Nevertheless, Chang and Tseng (2009a) stated this issue related to information
storage can be settled by digital portfolios when technical support is available, such as floppy disk, hard disk, and
CD-ROM. If not, Internet technologies may come into play. The Internet has markedly contributed portfolio
assessment by establishing an excellent mechanism where learners are allowed to inspect and assess peers’ work
online ─ this may not be possible for paper-based portfolio assessments.
Although learning portfolios have shifted from paper- to Web-based model, it is unfortunate that many Web-based
portfolios, as an “assessment” tool, are not yet aligned with well-constructed scoring criteria. More than scoring
standards, assessment rubrics play a facilitating role in students’ portfolio creation (Oskay, Schallies, & Morgil,
2008). According to Lynch and Purnawarman (2004), the digital portfolio assessments adopted by 35% American
teacher preparation programs did not come along with scoring rubrics. Researchers including Johnason, Fisher,
Willeke, and Mcdaniel II (2003) and Rogers (2003) claimed that presenting a concrete rubric in Web-based portfolio
assessment is a crying need. It is desirable that rubrics are reasonably reliable and valid, since this is a key factor
allowing effective evaluation and promising evidence. To date, portfolio rubrics are mainly concerned with
conventional paper formats, while few are tailored to accommodate Web-based assessments. Although the rubrics of
the two types of assessments may have something in common, there are some features that should be taken into
account when assessing Web-based portfolios, such as multimedia design, information presentation and layout. In
addition, current emphasis of assessment rubrics is placed on the descriptions of portfolio entries instead of students’
learning achievements (Bonita & Duquence, 1999; Danielson & Abrutyn, 1997; Johnason et al., 2003; Skawinski &
Thibodeau, 2002). At this point, the fundamental purpose of an assessment ─ to measure learners’ achievements ─
cannot be served. Also, reliability or validity investigations so far have paid little attention to reliability and validity
analysis of rubrics, largely focusing on the interpretations of assessment results. If a portfolio assessment is to be
significant, rubrics that can accurately reflect learning achievements need to be developed, and the reliability as well
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the
copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by
others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.
265