Jump to Navigation

Feature

When It Comes To Polls, Read The Fine Print

When It Comes To Polls, Read The Fine Print Polling Report Survey Lilly Foundation Funding Grants Insights into Religion News

Even lay people can, and should, ask tough questions about polls to see if their claims match their findings. And when it comes to poll data we should all be informed consumers.

A recent story touted the results of a new poll: “Best place to meet friends?” the headline read, “Bars, restaurants top churches.”

But a careful look at the survey numbers might have produced a different headline. “Best place to meet friends?” the story should have read, “Bars, restaurants and churches.”

Journalists aren’t always careful readers of polls, and the same goes for pastors and other church leaders. In this particular poll by Group Publishing, a nondenominational Protestant publishing house in Colorado, the size of the poll and the margin of error pointed to a different conclusion from that drawn by the pollsters. Specifically, the survey found that 18 percent of Americans go to restaurants, bars and pubs to meet new friends while 16 percent go to churches.

Some 800 were polled, and the survey’s margin of error was four percentage points.

“Given the size of sample, there’s no statistically significant difference between the 18 percent for restaurants and bars and the 16 percent for churches,” says Deborah Bruce, research manager for the Presbyterian Church (U.S.A.) and project manager for the U.S. Congregational Life Survey.

Bruce and colleague Scott Thumma, a sociologist of religion at Hartford Seminary, say the religious community ought to be more savvy about polls if it wants to better understand people’s spiritual behaviors.

“Americans seldom think of polls as products they should question,” says Thumma. “Our assumptions are the experts are telling us the truth. What we’re suggesting is: Don’t just accept this stuff at face value.”

Polls and surveys take on an even more important role when it comes to religion because the U.S. Census is forbidden by law from asking questions about people’s religious identities. But Bruce and Thumma both agree, a person need not have taken an “Introduction to Statistics” class in college to make informed conclusions about a particular poll.

When a poll is released, the first question people should ask is “Who did the poll?” Although polls are scientific measuring instruments, they are devised by human beings with their own particular agendas and biases.

“There isn’t any research that’s completely neutral,” says Thumma. “You have to look at who’s making the claims and who’s paying for the research.”

In any survey, readers ought to know the size and scope of the poll: Was it a national poll or a congregational poll? Were its questions asked in person, on the phone, or online?

More specifically, readers should examine how questions were framed. Was the wording clear? Were the questions balanced?  Bruce suggests people should be wary of leading questions, such as “Most people experience God’s presence in worship; do you?” (Most survey takers wouldn’t want to be put in the category of those incapable of experiencing God’s presence.)

The order in which questions were asked is important too. A poll that asked people to identify their religious affiliation before asking “Do you support abortion?” might give survey takers an unconscious prompt to take their faith’s teachings into account before answering.

Then there are vague questions, such as “How often do you read the Bible?” A better question might be “How often did you read the Bible this past week?”

 When writing on polls, news accounts don’t always list the questions asked or offer important details about the size of the survey or how it was conducted. But any reputable survey ought to provide those answers in its “methodology” section, or by making its investigators available to readers.

“If it’s good research,” says Bruce, “they’ll be more than willing to give you whatever details you need to have.”

Learning to ask questions about surveys is not that different from the kinds of consumer research people do every day when they buy a washer and dryer or a lawn mower.

“Folks should treat polls as they would any other product they’re purchasing,” says Thumma. “Explore the claims; see if they are worth buying.”

 

Search

News

News

Religious ‘Nones’ are Gaining Ground in America

Thirty-four percent of Americans surveyed said they were atheist, agnostic, or “nothing in particular,” according to...

$10 million grant from the Lilly Endowment for Charleston's International African American Museum

The museum has announced several major donations this year, including a $10 million grant from the Lilly Endowment last...

Yolanda Johnson-Moton Honored

Yolanda Johnson-Moton is the Director for external relations at Eli Lilly and Company where she drives the strategy and...

Calendar

Thu, Jan 25, 2018 - 08:00 am
The annual Calvin Symposium on Worship is a three-day conference held in January and sponsored

Popular Tools

Browse dozens of topics and see responses from a major national survey of congregations.

 

 

Widget provided by: