The Blue DOT 14 - Multidisciplinary Science & Evidence For Education

OPINION
Decisions are often made
on the basis of incomplete
and imperfect information,
and the uncertainty around
quantitative results is one of
the key factors at play.
quantitative results is one of the
key factors at play. As the eventual
implementation of interventions may have
positive or negative impact on learners,
understanding uncertainty of impact
estimates must be an integral part in
educational practice and policymaking. In
particular, the consistency or variability
of effect sizes across studies of similar
interventions is critical to support how
their desired outcomes generalize to
populations and contexts (Borenstein,
2019). Effect sizes 1 (with confidence
intervals) provide a better indication of
impact than significance testing (p-values)
and should be discussed in all cases. This
rationale implies moving away from a
dichotomous interpretation of statistical
significance as the means to gauge the
effectiveness of an intervention.
How do we remain faithful
to the evidence, whilst
making it fit our context?
Using the best evidence as outlined above
REFERENCES
Borenstein, M. (2019). Common Mistakes in Meta-Analysis and How
to Avoid Them. Englewood, USA: Biostat.
Brighouse, H., Ladd, H. F., Loeb, S., & Swift, A. (2018). Educational
goods: Values, evidence, and decision-making. Chicago, IL: University
of Chicago Press.
Caswell, R.J., Maidment, I., Ross, J.D.C., & Bradbury-Jones, C.
(2020). How, Why, for Whom and in What Context, Do Sexual
Health Clinics Provide an Environment for Safe and Supported
Disclosure of Sexual Violence: Protocol for a Realist Review, BMJ
Open, 10, 1–8, doi:10.1136/bmjopen-2020-037599
De Souza, D. (2016). Critical Realism and Realist Review: Analyzing
still offers no guarantee of success, because
this evidence describes what has worked,
rather than what will work. In addition,
experimental results, even aggregated
and recalibrated through meta-analysis,
are always context-dependent and
interventions are never implemented in
the same context (Smets & Struyven,
2018). Effectiveness predictions (Joyce
& Cartwright, 2020), on the contrary, is
the prediction that a given intervention,
documented as the most effective, will
work concretely within the multi-faceted
characteristics of a given context of
application. Effectiveness predictions
emphasize evidence regarding how a
particular context is likely to provide
the necessary conditions for anticipated
benefits to occur. Thus, this type of
evidence is very different from evidence
related to the generalization of
experimental studies. Yet, it focuses on the
same key aspect as evidence of relative
effectiveness, that is, causality between
interventions and outcomes. This causality
is further explored through an emphasis
on mechanisms, which formalize causal
processes (Caswell, Maidment, Ross,
& Bradbury-Jones, 2020) hinging on
structure, culture and agency (De Souza,
2016).
This last step, by focusing on adapting
the context of implementation, avoids the
circular reasoning involved in adapting
the best possible intervention to a given
context: an intervention not implemented
as tested will likely yield unknown
outcomes, because the causal link between
an intervention and outcome depends on
the permanency of the intervention and
of the outcome; you change one or the
other: the prediction does not hold.
Conclusion: how to remain
up to date in obtaining and
applying evidence?
The methodology to obtain evidence
of relative effectiveness in education
has been recently updated. Reaping
its benefits is a massive undertaking
consisting of systematic literature
search, followed by appropriate
meta-analysis followed by
systematic literature reviews for
predictions of effectiveness. Once
completed in any pertinent educational
domain, this work should be turned
into a living review, in which the initial
framework is used to integrate new studies
as they get published. This integrative
scholarly work is frequently overlooked.
Nevertheless, it seems that it is the
essential final step in using research for
policymaking and practice in education.
Governments, universities, and partner
NGOs should collaborate further to gather
the resources and expertise to pursue this
international endeavor.
Complexity in Educational Restructuring and the Limits of
Generalizing Program Theories Across Borders. American Journal of
Evaluation, 37(2), 216-237.
Joyce, K., Cartwright, N. (2020). Bridging the Gap Between Research
and Practice: Predicting What Will Work Locally. American
Educational Research Journal, 57, (3), pp. 1045–1082. DOI:
10.3102/0002831219866687
Smets, W. & Struyven, K. (2018). Realist review of literature on
catering for different instructional needs.pdf. Educational Science, 8,
113; doi:10.3390/edusci8030113
Rebooting our relationship
to technology: Insights from
psychology into how we
think about AI and EdTech
in the aftermath of Covid-19
John Sabatini is a Distinguished
Research Professor in the Institute
for Intelligent Systems and the
Department of Psychology at
the University of Memphis. He
conducts research in educational
technology, reading literacy
development and disabilities,
assessment, cognitive psychology,
the learning sciences.
Art Graesser is emeritus professor
in the Department of Psychology
and the Institute of Intelligent
Systems at the University of
Memphis. He conducts research
in discourse processing, cognitive
science, educational psychology,
computational linguistics, and
artificial intelligent in education.
He developed AutoTutor, a system
that helps people learn with
conversational agents.
JOHN SABATINI,
ARTHUR GRAESSER
Abstract
In this article, we explore whether it
is time we reconsider our relationship
to technologies in education, both
as individuals and societies. Multiple
generations have now experienced
different forms of digital devices and the
functions and roles they play in society
and education, layered upon traditional
models of the role and function of
education institutions. Over the same time
period, we have learned a great deal from
psychology about how human’s reason,
make decisions, and learn. In this article,
we review recent and emerging AI and
1 An effect size is an indication of the magnitude of differences between groups or the intensity of the relation between variables.
ISSUE • 14
3 9