More or Less is the BBC’s flagship programme on the use and misuse of statistics. On 1 February it led on on digging down into alarmist reports of a survey on the extent of Holocaust denial in the UK. It discovered poor survey design that led to gross over-statement of the extent of the problem. All holocaust denial must be challenged but exaggerating its extent both normalises it, aiding antisemites; and gives apparent credence to those who claim that Israel is the only safe haven for Jews.
Despite the flaws in the methodology the survey results were widely reported in lurid terms, including on the BBC itself Holocaust Memorial Day: ‘Shocking’ levels of denial remain
Transcript
Tim Harford: … Last Sunday was Holocaust Memorial Day; a day of solemn remembrance, but it was also a day of appalled surprise because a poll was published claiming that “as many as one in 20 adults in Britain don’t believe the Holocaust took place and 1 in 12 believe its scale had been exaggerated.”
One in 20 Britons; that would be about three million people not believing that the Holocaust happened. The survey said that many others were confused about the details.
So, to clear up any uncertainty, at least here, The Holocaust is a name given to the genocidal murder of around 6 million Jews led by the German State under Adolf Hitler’s Nazi government and part of an even bigger policy of systematic murder of a variety of targets including the Roma, disabled people, political prisoners and many others.
We’ve always known that a few people love to claim that this never happened or happened on a dramatically smaller scale; but as many as one in twenty?
Loyal listeners were quick to ask questions. George Gallant perhaps best summed up the mood in our Inbox. “This statistic paints a particularly bleak picture of a population I don’t recognise. Is it really possible that one in twenty of us don’t believe that the Holocaust occurred?
A number of listeners had tried to do the fact checking themselves and, got as far as the website of the organisation that had commissioned the poll – the Holocaust Memorial Day Trust – but they didn’t find much detail to go on.
So Ruth Alexander is here, “Hello Ruth” “Hi Tim”. “and you’ve been following up”.
Yes the Holocaust Memorial Day Trust provided only basic information on its Website Press Release.
This was a survey of more than 2,000 people, it said; carried out by Market Research Company called Opinion Matters and these, the Holocaust Memorial Day Trust said, were the views of a representative sample of UK adults. But we and many of our listeners wanted to know more.
Tim: Well first of all, we need to know precisely what people were asked and how.
Ruth: Yes, so I asked for and was given the actual survey questions.
Tim: And then what we need is a Professor of Survey Methodology.
Ruth: coming up
Peter Lynn: I’m Peter Lynn, I’m Professor of Survey Methodology at the University of Essex
Tim: That’ll do.
Ruth: Now when Professor Lynn heard about the results of this survey, he raised an eyebrow.
The results were unlikely
Peter Lynn: Yes, I was immediately sceptical that this sounded a bit (umm) unlikely.
Ruth: The number sounds too big?
Peter Lynn: Yes
Ruth: In fact, I’ve spoken to three other survey design experts – they all agree, there are some serious flaws with this study.
Now it’s true that 5% surveyed agreed or strongly agreed with the statement that the Holocaust never really happened. However, Professor Lynn thinks they may not all have done so deliberately
Peter Lynn: I guess the first thing that struck me was that the wording of the question about believing that the Holocaust happened seems to me to have some serious shortcomings and I think that may have caused some people to appear to agree that the Holocaust had never really happened when that wasn’t what they intended.
Tim: So the issue here is that people taking part in an online survey like this; they’re ticking boxes, maybe they’re not looking too closely; maybe they’re in a hurry, they’re distracted by what’s on TV; they’re thinking more about the shopping vouchers they might receive for doing the survey than what they’re actually agreeing or disagreeing with and some of them might make outrageous claims just for the fun of it.
Ruth: After all, according to the latest census there are 177,000 people living in the UK who follow the Jedi religion – the worship of the mythology of Star Wars.
Tim: Which is a more good-natured piece of trolling but it’s trolling none the less. And the point is that when you are talking about a small percentage of people giving an outrageous response, the results are easily skewed by a few wind-up merchants, drunks and people randomly ticking boxes because they just don’t care. So what did the survey do to weed out this sort of thing?
Holocaust not explained
Ruth: Well, not as much as we might hope, First of all the term ‘Holocaust’ wasn’t explained, and Professor Lynn says that’s poor practice; it would have been better if the survey designers had explained the term and said it refers to the systematic murder of millions of people by the Nazi State.
Now to the questions in the survey:
Q1 asked respondents how many Jewish people did they think were murdered in the Holocaust and people’s answers range from 0 through to 6 Billion.
Tim: which is more than the population of the planet at the time
Ruth: … mmmm – Now Q2 asked ‘when roughly do you think the Holocaust ended’ and you could choose a date from 1863 through to 1981 or say ‘none of these’ or ‘don’t know’.
And the third question presented 10 statements and respondents had to pick an answer from a response scale from ‘Strongly agree’ to ‘Strongly disagree’ or don’t know
Here are some of the statements:
- It is important to know about the Holocaust in today’s world
- More needs to be done to educate people about what happened during the Holocaust
- The Holocaust never really happened
- Something like the Holocaust couldn’t happen again
And it was the third statement in that list “The Holocaust never really happened” which 5% of respondents said they agreed or strongly agreed with and which created all the media shock headlines. But Professor Lynn says many of them could easily have agreed with this statement by mistake.
Peter Lynn: there is always a significant minority of respondents who take short cuts and I think that that could be the case here because the respondents here are presented with a series of statements.
Now the first two items in this scale are firstly: ‘It is important to know about the Holocaust in today’s world’ and secondly: ‘More needs to be done to educate people about what happened during the Holocaust’.
So, at that point you might be beginning to think, ah I can see a battery of statements about the Holocaust, seems like I’m the kind of person who tends to agree with them; I’ll just agree to the next few and assume that that represents my position.
Reversed items confuse
But the item we’re talking about here, the third item ‘the Holocaust never really happened’ is worded the other way round; it’s what we call a reversed item, where if you believe that the Holocaust happened, you should now be disagreeing with the item. So you could easily fall into the trap of just assuming you agreed with all these items and, therefore, not giving a response to this third item that actually represents your true view.
Ruth: And, ideally you’d cross check all the responses to make sure there was no inconsistency. (Now) Opinion Matters says that it has done that with the Holocaust deniers but it wouldn’t give us the full data on individual responses so that we could check ourselves, and from the data we do have there are clues that there may be some inconsistency in people’s responses.
Peter Lynn: One thing I actually noticed in the data is there is this question about how many Jewish people do you think were murdered during the Holocaust: only 5 people in the entire survey gave an answer of zero; 5 people out of just over 2,000, so that’s one quarter of 1%.
My guess is that’s probably something closer to the reality of what proportion of people genuinely don’t think the Holocaust happened.
Tim: So what do the Holocaust Memorial Day Trust and Opinion Matters have to say about this?
Ruth: Well they stand by the results of the survey and Opinion Matters says that it’s been carried out to the Market Research industry standards. Now Holocaust Memorial Day Trust admit to being surprised by the results and they say they’ve made it clear that they don’t think 5% of people are actively and maliciously working to deny the Holocaust occurrence and they do welcome further research.
Tim: It seems like there is a lot of ignorance out there and clearly Holocaust denial is a real thing and it would be worth trying to measure how prevalent it is. So do we know of any other, perhaps more reliable, research that can give us a better sense of the true numbers?
Holocaust denial exaggerated in US survey
Ruth: well there’s an interesting story from the United States that tells us quite a lot. There was a study there, in the early 1990s, which put the number of Holocaust deniers at more than 20% of the US population, so you can imagine the horror and the headlines that caused. But survey experts didn’t believe it and did further studies to test the finding. It turns out that the issue was with the wording of the question. So, listener, I’ll give you the question now, see if you can answer it get ready:
“Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?”
So the two choices for your answer are ‘possible’ or ‘impossible’: choose now:
OK, so I’ll give you the answer.
If you said possible, you are a Holocaust Denier.
Tim: it is one of those confusing multiple negatives like ‘must not fail to miss’.
Ruth: mm – I have to say, even though I knew the issue with the question, every time I read it, I got totally confused about which way I should answer and I’m not a Holocaust denier!
Tim: and did the survey respondents also get confused?
Ruth: They did, many of them were. Once studies were done with clearer questions and follow up questions asking for detail of the respondents’ beliefs about the Holocaust, it turned out that the correct number of Holocaust Deniers in the US was more like 2% of the population. And experts have told me that studies in Europe have tended to give lower numbers still.
Tim: Well thank you Ruth…
