Promoting Grit, Tenacity, and Perseverance - U.S. Department of ...

Draft
Leveraging Data from Students’ Interactions with Intelligent Tutoring Systems
Persistence and “Gaming the System”
Using EDM techniques, researchers have identified concrete behaviors in online learning environments that
reflect persistence or gaming the system. Persistence in an online learning environment or a game means
spending more time on difficult problems as measured by indicators such as time on unsolved problems
(controlling for ability), and number of restarts and revisits to unsolved problems (Shute & Ventura, in press).
Time spent on problems has been positively linked to achievement outcomes in some studies, but not always.
Thus, the nature of the learning environment and kinds of questions are an important factor. In contrast to
behaviors associated with persistence, when students are gaming the system, they are “attempting to succeed
in an educational environment by exploiting properties of the system rather than by learning the material and
trying to use that knowledge to answer correctly” (Baker et al., 2006). Gaming behavior has been shown to be
detrimental to learning (Baker et al., 2004), so it is important to detect and address. Detection of gaming
behavior involves looking for “sustained and/or systematic guessing” and “repeated and rapid help requests”
(Baker, Corbett, Roll, & Koedinger, 2008; Baker, D’Mello, Rodrigo, & Graesser, 2010).
Help-Seeking Behavior
Most cognitive and intelligent tutoring systems include scaffolds and help resources and track when students
use these features. Data on help-seeking can include identification of the moment when a student asked for
help, the amount of help provided, the average number of hints requested per problem; how much time and
effort was invested when the student asked for help; the percentage of helped problems; whether help was
requested before or after making an initial attempt or after entering the correct answer (Arroyo & Woolf, 2005).
In one study on the Help Tutor, an enhanced version of Geometry Cognitive Tutor, students received
metacognitive feedback on their help-seeking behaviors (Roll, Aleven, McLaren, & Koedinger, 2011). For
example, if a student was “drilling down” to the bottom-level hint without reading the intermediate hints, the
help tutor would suggest: “No need to hurry so much. Take your time and read the hint carefully. Consider
trying to solve this step without another hint. You should be able to.” Students in the Help Tutor condition
made fewer help seeking errors and learned more.
Some example measures. Here we describe several illustrative examples of these types of
measures. The first is a set of low-tech measures that simply use wait time. The Preschool Self-
Regulation Assessment includes tasks such as the “Toy Wrap,” in which the child is asked not to
peek while a toy is being wrapped, and the “Toy Wait,” in which the child must wait before
touching the wrapped toy (Murray & Kochanska, 2002). In controlled laboratory settings, these
kinds of performances are easily observable. The amount of time children can delay their
gratification can be used as an indicator of executive functions and effortful control.
The next set of measures includes approaches to examining students’ patterns of interactions
with digital learning environments. Interaction patterns can demonstrate how tool use in an
environment relates to mindsets, learning goals, learning strategies, and help-seeking. Examining
these interaction patterns can provide insight into the relationships between a learner’s goals and
behaviors. For example, many researchers are exploring how to leverage data from students’
interactions with intelligent tutoring systems, such as how they may “game the system” or seek
help (see callout box for more information).
42