Free Will as an Obstacle to Randomize Control Trials

Authors: Jason Bryer, Angela Lui, Piet Wesling, and Heidi Andrade.

AERA Paper
AERA Slides

Abstract

Randomized controlled trials (RCTs) are considered the gold standard for estimating causality. However, conducting research in educational settings using RCTs can result in unnatural intervention implementations due to participant agency — that is, their free will. This study explores the prevalence of RCTs in education using What Works Clearinghouse (WWC) data and examines challenges from implementing seven RCTs across five institutions. We found that RCTs were less likely than non-RCTs to report positive effects and faced issues of noncompliance. Propensity score analysis (PSA) was used to estimate treatment effects and revealed significant differences not evident in intent-to-treat analysis. These findings suggest that free will complicates RCT assumptions and that PSA may better capture the heterogeneous effects of educational interventions.

Setup

# Install required R packages
install.packages(c('tinytex', 'ggplot2', 'dplyr', 'papaja', 'knitr', 'readxl'))
# Install LaTeX so we can convert the manuscript to PDF
tinytex::install_tinytex()

---

DAACS was developed under grants P116F150077 and R305A210269from the U.S. Department of Education. However, the contents do not necessarily represent the policy of the U.S. Department of Education, and you should not assume endorsement by the Federal Government.