Assessing the Threat Posed by Small Inaccuracies in Evaluations That Seek to Detect Small Effects

The recent trend to design studies that detect smaller impacts may create the challenge of guarding against smaller inaccuracies or biases.

A new report from the Institute of Education Sciences (IES) National Center for Education Evaluation (NCEE) finds that small inaccuracies can pose a real threat in evaluations that need to detect small impacts. The NCEE recommends strategies researchers can use to avoid or reduce these inaccuracies.

The NCEE report examines the potential for small biases to increase the risk of making false inferences, a phenomenon the report calls asymdystopia. The report examines this potential for both randomized controlled trials (RCTs) and studies using regression discontinuity designs (RDDs).

See the IES NCEE website to view the complete report Asymdystopia: The threat of small biases in evaluations of education interventions that need to be powered to detect small impacts.