This article in The Economist entitled “Removing Phones from Classrooms Improves Academic Performance” (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5370727) has been cited by Chris Ferguson as an example of how ignoring effect sizes can lead to false conclusions. Here is the abstract of the paper, followed by Ferguson’s critique:

Widespread smartphone bans are being implemented in classrooms worldwide, yet their causal effects on student outcomes remain unclear. In a randomized controlled trial involving nearly 17,000 students, we find that mandatory in-class phone collection led to higher grades — particularly among lower-performing, first-year, and non-STEM students — with an average increase of 0.086 standard deviations. Importantly, students exposed to the ban were substantially more supportive of phone-use restrictions, perceiving greater benefits from these policies and displaying reduced preferences for unrestricted access. This enhanced student receptivity to restrictive digital policies may create a self-reinforcing cycle, where positive firsthand experiences strengthen support for continued implementation. Despite a mild rise in reported fear of missing out, there were no significant changes in overall student well-being, academic motivation, digital usage, or experiences of online harassment. Random classroom spot checks revealed fewer instances of student chatter and disruptive behaviors, along with reduced phone usage and increased engagement among teachers in phone-ban classrooms, suggesting a classroom environment more conducive to learning. Spot checks also revealed that students appear more distracted, possibly due to withdrawal from habitual phone checking, yet, students did not report being more distracted. These results suggest that in-class phone bans represent a low-cost, effective policy to modestly improve academic outcomes, especially for vulnerable student groups, while enhancing student receptivity to digital policy interventions.

Ferguson says: This is a great example of why study authors relying on p-values and ignoring the likelihood of weak effect sizes being statistical noise can do great harm. The study looked at 17000 students in a study of cellphone bans in schools…still found null effects for most outcomes, but focus on one positive outcome with a d = .086 (about r = .043). Aside from the likelihood the one finding (against the other null findings) is due to chance, this effect size is well within the range of those indistinguishable from statistical noise (Ferguson & Heene, 2021). It should never have been interpreted as hypothesis supportive. Yet the Economist (falsely) portrays it as a “ringing endorsement.” It is, in fact, far better evidence against the use of cellphone bans than for it.

As described in this article, a video game called EdeavorOTC received FDA clearance as an ADHD treatment in August of 2024. You can find the game here. The assertions made about the game’s effects on ADHD should provide your students with a good opportunity for honing their critical thinking skills.

This article from the UK’s Daily Mail provides an example of a rather questionable stress reduction device as well as an excellent opportunity for your students to think critically about assertions that are likely to carry extra weight because of their association with an international celebrity, Meghan Markle.

A recent JAMA study, found here describes a strong correlation between hearing loss and dementia in older adults. A commentator interpreted this correlation to mean that people should do all they can to mitigate hearing loss as soon as it appears so as to reduce the risk of developing dementia. You may want to use this study, and particularly the commentator’s assertion as an example of how easy it is to over-interpret the meaning of a correlation.

• Research Methods & Statistics
• Biological Aspects of Psychology
• Sensation and Perception
• Learning
• Memory
• Thinking and Cognitive Abilities
• Consciousness
• Motivation and Emotion
• Development
• Health, Stress, and Coping
• Personality
• Psychological Disorders
• Treatment of Psychological Disorders
• Social Psychology
• Industrial/Organizational Psychology

An article found here describes a study of middle-aged and older adults showing that men who have nightmares at least once a week are significantly more likely than other men to develop dementia. This correlation did not reach statistical significance for women. Speculating on the basis for this association, the researchers suggest that when dementia is in its undetectable early stages, it creates negative emotions that may be expressed as depression while awake and as nightmares while asleep. They hypothesize that medication that can reduce nightmares might also help to slow cognitive decline. Is the association between nightmares and development of dementia causal or correlational? If the latter, what third factor(s) might be responsible for it and why did it only appear in men? The study on which this article is based provides a good example of research methods whose interpretation depends on critical thinking about correlation and causation. It should provide a useful target for classroom discussion as well as a topic for student writing assignments aimed at promoting critical thinking.

This chapter emphasizes the importance of helping students to think critically, including by alerting them to these ten warning signs of pseudoscientific assertions:

Lack of falsifiability and overuse of ad hoc hypotheses

Lack of self-correction

Emphasis on confirmation

Evasion of peer review

Overreliance on testimonial and anecdotal evidence

Absence of connectivity

Extraordinary claims

Ad antequitem fallacy (an appeal to tradition as an argument for validity)

Use of hypertechnical language

Absence of boundary conditions

Source: Lilienfeld, S. O., Ammirati, R., & David, M. (2012). Distinguishing science from pseudoscience in school psychology: Science and scientific thinking as safeguards against human error. Journal of School Psychology, 50(1), 7-36.