This article in The Economist entitled “Removing Phones from Classrooms Improves Academic Performance” (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5370727) has been cited by Chris Ferguson as an example of how ignoring effect sizes can lead to false conclusions. Here is the abstract of the paper, followed by Ferguson’s critique:

Widespread smartphone bans are being implemented in classrooms worldwide, yet their causal effects on student outcomes remain unclear. In a randomized controlled trial involving nearly 17,000 students, we find that mandatory in-class phone collection led to higher grades — particularly among lower-performing, first-year, and non-STEM students — with an average increase of 0.086 standard deviations. Importantly, students exposed to the ban were substantially more supportive of phone-use restrictions, perceiving greater benefits from these policies and displaying reduced preferences for unrestricted access. This enhanced student receptivity to restrictive digital policies may create a self-reinforcing cycle, where positive firsthand experiences strengthen support for continued implementation. Despite a mild rise in reported fear of missing out, there were no significant changes in overall student well-being, academic motivation, digital usage, or experiences of online harassment. Random classroom spot checks revealed fewer instances of student chatter and disruptive behaviors, along with reduced phone usage and increased engagement among teachers in phone-ban classrooms, suggesting a classroom environment more conducive to learning. Spot checks also revealed that students appear more distracted, possibly due to withdrawal from habitual phone checking, yet, students did not report being more distracted. These results suggest that in-class phone bans represent a low-cost, effective policy to modestly improve academic outcomes, especially for vulnerable student groups, while enhancing student receptivity to digital policy interventions.

Ferguson says: This is a great example of why study authors relying on p-values and ignoring the likelihood of weak effect sizes being statistical noise can do great harm. The study looked at 17000 students in a study of cellphone bans in schools…still found null effects for most outcomes, but focus on one positive outcome with a d = .086 (about r = .043). Aside from the likelihood the one finding (against the other null findings) is due to chance, this effect size is well within the range of those indistinguishable from statistical noise (Ferguson & Heene, 2021). It should never have been interpreted as hypothesis supportive. Yet the Economist (falsely) portrays it as a “ringing endorsement.” It is, in fact, far better evidence against the use of cellphone bans than for it.