A UBC Okanagan professor is suggesting government policy makers and advisors need to do a rethink when it comes to giving validity to reports coming across their desks.
Carey Doberstein, an assistant professor of political science a the Kelowna campus, recently published an experimental study of public sector workers and determined that many give a written report or study purported to be from a university more credibility than one from a think-tank or advocacy group.
Doberstein conducted a randomized controlled survey experiment involving British Columbia public service staff, asking them to read and assess the credibility of various policy studies. For half of the respondents, the authorship of the studies was randomly switched but the content remained the same.
Doberstein then compared the average credibility assessments between the control and experimental groups.
“There were systematic and at times extraordinarily large differences between the credibility assessments provided by these policy professionals on precisely the same policy studies, when the only part I changed was the label of who wrote it,” said Doberstein.
“Irrespective of the content and just by virtue of presenting it as written by an academic, the report suddenly becomes more credible in the eyes of bureaucrats.”
The results surprised him, in part due to the magnitude of the differences observed. For one report, originally authored by the Fraser Institute, the credibility skyrocketed among study participants when they read the same document thinking it came from a university academic.
Another policy study, this time written by a university economist, received very high credibility assessments in the control group. But when authorship was changed to be purportedly written by the Canadian Centre for Policy Alternatives think-tank, its credibility plummeted dramatically.
“Put simply, the think-tank affiliation was a significant drag on the perceived credibility of their report and analysis,” said Doberstein.
The same was true for reports said to be written by research-based advocacy groups.
“Some may interpret this finding positively,” he said. “That analysts in government are skeptical of reports or studies that emerge from think tanks or advocacy organizations offering analysis and conclusions that tend to align with the organization’s obvious ideological position.”
Yet Doberstein says having a report’s credibility increase simply by changing the name of the source is concerning as it can appear that policy-relevant research contained within its pages is being ignored by government policy advisors.
“We expect public servants to objectively examine the research evidence available to them,” he said. “However, it seems many are taking shortcuts, and in essence giving academics a free pass.”
And while this study examined the biases among policymakers in BC, Doberstein notes similar results were observed his subsequent replication experiment involving provincial policy analysts in Ontario, Saskatchewan and Newfoundland.
Doberstein’s study was recently published in Policy Studies Journal.
