Penn researchers find nearly half of observational social science studies overstate causality

Penn researchers found that overstating cause-and-effect relationships is a growing problem in the social sciences.

In an August study that analyzed nearly 200,000 papers, researchers at Penn’s Computational Social Science Lab found that nearly half of the surveyed articles made causal overstatements. Their research showed that these overstatements have tripled since the 1980s, and that artificial intelligence models tend to amplify them.

Lead author and fourth-year Ph.D. student at the Annenberg School for Communication Calvin Isch said in an interview that correlation refers to when “two measures go together,” but one might not directly cause another.

For example, he explained, an increase in shark attacks might be correlated with an increase in ice cream sales, but shark attacks do not cause ice cream sales, or vice versa.  

“These are two things that might go together because whenever it’s warm out, people go to the beach and they buy more ice cream, but they also are at the beach and there can be more shark attacks,” he added. “These are two things that might be correlated, but they’re not in any way causally related.” 

The research team at the Computational Social Science Lab, which is directed by Annenberg School professor Duncan Watts, looked at 194,631 observational, cross-sectional studies — research that collects data only at a single time point.

The group showed that 46% of these studies had some form of causal overstatement in their abstracts or titles. According to the study, the percentage rose from roughly 20% in the 1980s to 60% in 2024. 

Isch argued that overstating causal relationships in scientific articles is a form of “narrative license” that departs from “evidence to create a more compelling story.”

“If I put a lot of money into a thing that I think is causally connected, but it’s not, it’s just wasted money, and often it can come with other consequences,” he said, adding that “that’s really why it matters in the social sciences.”

“Most of these domains are investigating these things with societal relevance, and people will be looking to them for practical advice about how we can do better policies,” Isch added.

The researchers also experimented with AI large language model summaries of articles using various chatbots, and found that specific prompting strategies can increase causal overstatements.

According to Isch, when researchers asked an LLM to explain hard-to-understand concepts “at an eighth-grade level,” the model was “more likely to make a causal claim by a great deal.” 

He explained that adding a “guardrail prompt,” such as “make sure that all the claims you have are proportional to the evidence,” can help reduce this issue.

While Isch acknowledged that the researchers could not determine exactly what drove the increase in causal overstatements over time, he speculated that it may be explained by how “in the late 1990s, early 2000s, the NSF and the NIH changed the way that they rewarded funding to include impact-oriented science.”

“If you want to improve the world, it’s good that we’re doing science that is going to improve the world,” he added. “It adds an incentive for scholars to state that their findings are causing impact, even if they might not be, right?”

In its most recent strategic plan, the National Science Foundation also stated that it “will review and revise its funding opportunities to enhance impact” and phase out lower-priority opportunities.

Isch added that University incentives based on “the number of publications that you get and the prestige of the journals they get in” might also contribute to this increase in overclaiming. 

Now, Isch’s team plans to broaden their investigations to encompasses other study designs and research fields such as medicine, where it might be that an “article itself looked at rats and mice, but the abstract does not say that — it looks like it’s applying to humans.”

Based on his findings, Isch emphasized that readers of social research should always “check their own priors.”

“I would just say as you’re approaching scholarship, be really willing to interrogate the methods as best you can, or even ask language models to interrogate the methods for you,” he said.

---