Abstracts - Earli

etrospective to task performance, whereas on-line measures are registered during actual taskperformance. Apart from the off-line/on-line dimension, assessment methods will be evaluatedagainst three validity criteria: Internal consistency (reliability), concurrent validity by mutualrelations with other assessment methods, and external validity for the prediction of learningoutcomes. Although we are still in the process of gathering literature data for this review study,some tentative conclusions can be drawn. On-line measures should probably be preferred over offline-measures,as the latter seem to lack concurrent and external validity. Moreover, there is a needfor research using multi-method designs.Assessing the use of on-line trace methodologies for analyzing the deploymentRoger Azevedo, University of Memphis, Department of Psychology, USAAmy Witherspoon, University of Memphis, Department of Psychology, USAJeremiah R. Sullins, University of Memphis, Department of Psychology, USAShanna E. Baker, University of Memphis, Department of Psychology, USALearning with hypermedia environments involves the deployment of key cognitive andmetacognitive self-regulatory processes. A key to assessing these processes during learninginvolves the use of several on-line trace methodologies such as concurrent think-aloud protocols,log files and video analyses. We have accumulated an extensive knowledge base from laboratoryand classroom studies examining the role of adolescents’ and college students’ self-regulatedlearning with hypermedia-based environments (see Azevedo, 2005). Our current understanding ofthe dynamic and recursive nature of use of these processes during learning is somewhat limited bythe capabilities of current technological tools which are augmented by the use of multipleconverging on-line trace methodologies. In this presentation, we will assess the conceptual,methodological, and empirical issues related to using on-line trace methods to assess learners’understanding about complex science topics (e.g., circulatory system) with hypermedia.Conceptually, we will (1) present results illustrating the advantages and disadvantages to using onlinetrace methodologies in collecting cognitive and other non-cognitive processes during learning;(2) describe current limitations in the use of log file analyses to examine self-regulatory processes;and, (3) argue for the integration of other self-report measures to augment our theoreticalunderstanding of the complex nature of self-regulatory processes. Methodologically, we willdiscuss ways in which current learning technologies and cognitive methodologies can be combinedwith computational and other advances in computer science and AI to collect, detect, trace, andmodel learners’ self-regulatory processes during learning. Empirically, we will discuss ourexisting approaches to converging process and product and assess its advantages anddisadvantages.– 580 –