A tale of two polls: So what is the real difference?

Recent polling commissioned by White Ribbon uncovered shocking attitudes by men on domestic violence. But questions were asked as to why the anti-domestic violence group didn’t just use the research that was already out there. We take a look.   

This is a story of two polls, both about attitudes on domestic violence but with starkly different results. 

One, released in 2017 by Australia’s National Research Organisation for Women’s Safety (ANROWS), found that 97 per cent of males aged 16 to 24 said “slapping or pushing to cause harm or fear” was domestic violence. 

The other, released by White Ribbon at the end of October, found 42 per cent of men aged 18 to 34 did not consider “hitting, punching or restraining” another person to be “a type of domestic violence”. 

This difference in results has sparked discussion. While some believe bringing any attention to the issue is important to shine a light on the problem, others believe by confusing the message, the latest poll undermines the previous research that already exists on the issue. 

One critic even labelled the move as more of a “publicity stunt than a genuine contribution” by White Ribbon to ending male violence against women.

So how did they end up so different?

A difference in methodologies 

White Ribbon’s research was originally part of a campaign run by White Ribbon and a number of other advocacy groups to criminalise coercive control. 

The organisation commissioned Essential Research to run the online poll from 30 September to 5 October 2020, with the final report based on 1,074 respondents aged over 18. 

The respondents, who opted-in to complete the survey, were provided a list of behaviours that experts regard as violence and asked to identify which they would consider domestic violence.

John Remington, Essential Research’s senior research manager, told Pro Bono News that participants may be asked for opinions on a range of social, policy, political and personal issues, and that the margin for error for full sample survey results is three per cent. 

This margin of error may be slightly larger for more specific demographics as those sample sizes are smaller. 

When compared with how ANROWS conducted its National Community Attitudes Survey (NCAS) however, there are a number of key differences. 

The NCAS is published every three years, measuring change in knowledge and attitudes towards violence against women and gender equality with a sample size of 17,500 people that are randomly selected so that they’re representative of the Australian population.  

The survey is conducted via phone interviews (a method found to be more reliable than online surveys because researchers are able to clear up any confusion around questioning immediately). Researchers then group responses to individual questions using scales developed to more accurately measure attitudes or knowledge towards violence against women.  

While results are only released every three years, Heather Nancarrow, the CEO of ANROWS, said it was important to note the rigour and time that goes into developing and carrying out the survey. 

“It’s a three year project, and it’s complex,” Nancarrow said. 

“It takes a year to develop and test the survey to make sure that the questions are understood by participants in the way the research is intended, and then to get the 17,500 samples it took over 250,000 phone calls.”

This goes some way to explaining why the results are so different. The bigger question is does that matter.

Crowding out the field

Nancarrow said that a key part of conducting research is being able to prove it is scientifically rigorous and reliable, because if it’s not, policymakers won’t take it on board.  

“There needs to be confidence in the findings by policymakers and practitioners who use the results,” she said.  

“That means that the research must be done well to build trust and to inform policy and practice.” 

She said that research carried out without the same rigour or accuracy risked “spoiling the field”. 

“If [the research] is not done well, then it can undermine sound research because you’ve got a number of people competing for research participants or confusing results because one survey showed this and those results show that,” she said.  

But the CEO of White Ribbon, Brad Chilcott, told Pro Bono News the organisation had made the decision to commission its own data because it didn’t make sense to “release media about someone else’s survey from three years ago”.

He said that while he was surprised the results of the latest poll were so different to research that was already out there, the focus should be on what action is now taken to address these concerning attitudes among young men.  

“Obviously there are differences between polling methodologies and other research that has taken place,” Chilcott said. 

“But at the same time, these were literally young men answering these polling questions…

We can spend a lot of time arguing about the research, or we can think about why some young men are answering the questions in this way and try to address it.” 

Working alongside one another

Nancarrow said she’s not out to criticise the work of White Ribbon, and noted that ANROWS always encouraged advocacy groups doing primary prevention work to walk alongside them. 

“Our work is to inform, and we work with policy owners to support policy and practice design and I can understand why White Ribbon and others want to do their own research to inform their work, but I don’t think doing that gives them a strong evidence base to guide their work,” she said. 

“And we definitely encourage organisations like White Ribbon and Our Watch and Respect Victoria and others which are doing primary prevention work to use the results of the NCAS.” 

For Hayley Foster, the CEO of NSW Women’s Safety, the White Ribbon survey wasn’t an attempt to “undermine the importance or the credibility of the NCAS” but more so about keeping the issue of domestic and family violence on the agenda.      

“There was nothing dishonest about the White Ribbon survey, but it just wasn’t as vigorous or nuanced,” Foster told Pro Bono News. 

“I really think any attempt to try to highlight how severe this [family and domestic violence] problem is in our society culturally, should be welcomed.”