655,000 War Dead? - NOT!

[red-beard]

From the WSJ online this morning

655,000 War Dead?

By STEVEN E. MOORE
October 18, 2006; Page A20

After doing survey research in Iraq for nearly two years, I was surprised to read that a study by a group from Johns Hopkins University claims that 655,000 Iraqis have died as a result of the war. Don't get me wrong, there have been far too many deaths in Iraq by anyone's measure; some of them have been friends of mine. But the Johns Hopkins tally is wildly at odds with any numbers I have seen in that country. Survey results frequently have a margin of error of plus or minus 3% or 5% -- not 1200%.

The group -- associated with the Johns Hopkins Bloomberg School of Public Health -- employed cluster sampling for in-person interviews, which is the methodology that I and most researchers use in developing countries. Here, in the U.S., opinion surveys often use telephone polls, selecting individuals at random. But for a country lacking in telephone penetration, door-to-door interviews are required: Neighborhoods are selected at random, and then individuals are selected at random in "clusters" within each neighborhood for door-to-door interviews. Without cluster sampling, the expense and time associated with travel would make in-person interviewing virtually impossible.


However, the key to the validity of cluster sampling is to use enough cluster points. In their 2006 report, "Mortality after the 2003 invasion of Iraq: a cross-sectional sample survey," the Johns Hopkins team says it used 47 cluster points for their sample of 1,849 interviews. This is astonishing: I wouldn't survey a junior high school, no less an entire country, using only 47 cluster points.

Neither would anyone else. For its 2004 survey of Iraq, the United Nations Development Program (UNDP) used 2,200 cluster points of 10 interviews each for a total sample of 21,688. True, interviews are expensive and not everyone has the U.N.'s bank account. However, even for a similarly sized sample, that is an extraordinarily small number of cluster points. A 2005 survey conducted by ABC News, Time magazine, the BBC, NHK and Der Spiegel used 135 cluster points with a sample size of 1,711 -- almost three times that of the Johns Hopkins team for 93% of the sample size.

What happens when you don't use enough cluster points in a survey? You get crazy results when compared to a known quantity, or a survey with more cluster points. There was a perfect example of this two years ago. The UNDP's survey, in April and May 2004, estimated between 18,000 and 29,000 Iraqi civilian deaths due to the war. This survey was conducted four months prior to another, earlier study by the Johns Hopkins team, which used 33 cluster points and estimated between 69,000 and 155,000 civilian deaths -- four to five times as high as the UNDP survey, which used 66 times the cluster points.

The 2004 survey by the Johns Hopkins group was itself methodologically suspect -- and the one they just published even more so.

Curious about the kind of people who would have the chutzpah to claim to a national audience that this kind of research was methodologically sound, I contacted Johns Hopkins University and was referred to Les Roberts, one of the primary authors of the study. Dr. Roberts defended his 47 cluster points, saying that this was standard. I'm not sure whose standards these are.

Appendix A of the Johns Hopkins survey, for example, cites several other studies of mortality in war zones, and uses the citations to validate the group's use of cluster sampling. One study is by the International Rescue Committee in the Democratic Republic of Congo, which used 750 cluster points. Harvard's School of Public Health, in a 1992 survey of Iraq, used 271 cluster points. Another study in Kosovo cites the use of 50 cluster points, but this was for a population of just 1.6 million, compared to Iraq's 27 million.

When I pointed out these numbers to Dr. Roberts, he said that the appendices were written by a student and should be ignored. Which led me to wonder what other sections of the survey should be ignored.

With so few cluster points, it is highly unlikely the Johns Hopkins survey is representative of the population in Iraq. However, there is a definitive method of establishing if it is. Recording the gender, age, education and other demographic characteristics of the respondents allows a researcher to compare his survey results to a known demographic instrument, such as a census.

Dr. Roberts said that his team's surveyors did not ask demographic questions. I was so surprised to hear this that I emailed him later in the day to ask a second time if his team asked demographic questions and compared the results to the 1997 Iraqi census. Dr. Roberts replied that he had not even looked at the Iraqi census.

And so, while the gender and the age of the deceased were recorded in the 2006 Johns Hopkins study, nobody, according to Dr. Roberts, recorded demographic information for the living survey respondents. This would be the first survey I have looked at in my 15 years of looking that did not ask demographic questions of its respondents. But don't take my word for it -- try using Google to find a survey that does not ask demographic questions.

Without demographic information to assure a representative sample, there is no way anyone can prove -- or disprove -- that the Johns Hopkins estimate of Iraqi civilian deaths is accurate.

Public-policy decisions based on this survey will impact millions of Iraqis and hundreds of thousands of Americans. It's important that voters and policy makers have accurate information. When the question matters this much, it is worth taking the time to get the answer right.

Mr. Moore, a political consultant with Gorton Moore International, trained Iraqi researchers for the International Republican Institute from 2003 to 2004 and conducted survey research for the Coalition Forces from 2005 to 2006.

[Rearden]

From the left-leaning, anti-Bush 'Iraq Body Count'

http://www.iraqbodycount.org/press/pr14.php

[red-beard]

When the right and the left agree on something...it has to be wrong!

hmmmmmmmm

Two wrongs don't make a right, but three lefts do!

[legion]

I wondered if their clusters were picked this way:

"This neighborhood looks pretty bombed-out...seems like a good cluster to me!"

[Jeff Higgins]

This is really going to dissapoint some of the liberals around here.

[techweenie]

Gosh, the WSJ Opinion people found someone to disagree with a peer-reviewed scientific report that reflects badly on the Bush administration.

A guy who worked for the impeccably qualified "International Republican Institute" no less. I'm sure he has no bias.

Who could imagine such a turn of events?

[Rodeo]

So how many Iraqis has Bush's war actually killed?

[Tim Hancock]

Apparently not enough as there are still quite a few causing havoc over there.

[tabs]

I wouldn't worry about how many Iraqis we have wasted until we get down to two or three left.

[Jim Richards]

why so many, tabs?

[jorian]

Whatever the number really is, Iraq is a clusterfuch by any survey.

[red-beard]

The 655,000 Fraud
WSJ - October 19, 2006; Page A18 (Paper Edition)

"A team of American and Iraqi epidemiologists estimates that 655,000 more people have died in Iraq since coalition forces arrived in March 2003 than would have died if the invasion had not occurred."

So read a story in last week's Washington Post on the new John Hopkins-led study -- published in the British medical journal Lancet -- purporting to document "excess deaths" in Iraq. "We have no reason to question the findings," the Post quoted a Human Rights Watch official as saying. The article was fairly typical of reporting on the Lancet study, which has also been all over television and radio, as well as Internet sites such as Google and Yahoo! news.

All of which leaves us wondering if reporters and editors have enough sense anymore to ask basic questions about such enormous numbers, or whether they are simply too biased against the Bush Administration and its Iraq policy to do so. The 655,000 figure is more than 10 times higher than previous estimates of violent deaths in Iraq since the U.S. invasion, and it is larger than the number of Germans killed by allied bombing during all of World War II and larger than the number of Americans who died during our own Civil War.

While it's obvious that Iraq has a terrible problem with sectarian violence at the moment, we find it hard to believe killing on the scale of Antietam or Gettysburg has been going on without anybody having noticed until the statistical wizards from Johns Hopkins showed up.

The 655,000 figure turns out to be an extrapolation based on a very inadequate sampling process. Pollster Steven E. Moore, who has worked extensively in Iraq, pointed out in an op-ed on this page yesterday that the Lancet study is based on information from a mere 47 "cluster points" around Iraq and 1,849 total interviews.

By contrast, a 2004 U.N. survey of Iraq used 2,200 cluster points for more than 21,000 interviews. The Johns Hopkins researchers also appear to have collected no demographic data on their subjects, so the group cannot be compared to census data to check if it is representative. "I wouldn't survey a junior high school, no less an entire country, using only 47 cluster points," Mr. Moore wrote.

Iraq Body Count -- a nonpartisan outfit that keeps track of Iraqi mortality figures -- has also issued a devastating critique of the Lancet/Johns Hopkins survey. It points out that the study implies that a thousand Iraqis died violently every day in the first half of 2006, with fewer than a tenth of them being noticed by "public surveillance mechanisms" and the press, as well as "incompetence and/or fraud on a truly massive scale by Iraqi officials in hospitals and ministries." It adds that death totals of the Lancet magnitude "are unnecessary to brand the invasion and occupation of Iraq a human and strategic tragedy."

Of course, the latter is precisely the agenda of the majority of those trumpeting the Lancet findings. Their goal isn't merely to nail the Bush Administration for incompetence in failing to achieve a sustainable victory in Iraq. They also, and perversely, want to discredit the war as a moral enterprise by suggesting there's no difference between Saddam Hussein's now well documented mass murders and the violence taking place today.

Omar Fadil, who with his brother writes from troubled Baghdad at IraqTheModel.com, has no doubt that the Lancet figure is a gross exaggeration. "All they want is to prove that our struggle for freedom was the wrong thing to do," he writes. "This fake research is an insult to every man, woman and child who lost their lives."

[techweenie]

655,000 Iraq War Deaths: Watch Ideologues Slander Good Science

Last week the esteemed medical journal The Lancet released an epidemiological study concluding that 655,000 Iraqis died from war-related injury and disease from March 2003 to July 2006. This shockingly high figure has drawn attacks from the Bush administration and right-wing pundits.

Speaking as a medical doctor, I wish to set the record straight.
The Lancet study is sound science. The study followed a strict, widely accepted methodology to arrive at its sobering conclusion. The study is being attacked not on scientific grounds, but for ideological reasons.

Full blog:
http://www.huffingtonpost.com/curren-w-warf-md-/655000-iraq-war-deaths-_b_31843.html

Author:
Curren W. Warf, M.D.
National Board member of Physicians for Social Responsibility.

[red-beard]

Iraq Body count is hardly a Bush controlled group!

[Rearden]

Originally posted by techweenie
655,000 Iraq War Deaths: Watch Ideologues Slander Good Science
Speaking as a medical doctor, I wish to set the record straight.
The Lancet study is sound science. The study followed a strict, widely accepted methodology to arrive at its sobering conclusion.

Since when is a pediatrician an expert on conducting statistically-significant population studies?

[Rodeo]

So how many Iraqis has Bush's war killed?

I suppose it was worth it, though. I mean, look at Iraq today, a model democracy for the Middle East, a check on Iraq and Syria, and an example of what Middle Eastern secular leaders can accomplish with America's backing.

[jyl]

Sounds like a bunch of laymen arguing about a particle physics experiment. Lack the data, the methodology, and the background knowledge.

I don't blindly accept the Johns Hopkins survey numbers as exact, nor do I accept the low-end estimates out there, nor do I think that Op-Ed writers can be believed. I basically split the difference and figure that most likely, somewhere between 100,000 and 400,000 Iraqis have died.

[red-beard]

John, being a Statistician is part of my Job. I am certified by the ASQ (American Society of Quality). I understand exactly what they did and the mistakes they made. To get to 95% confidence with the data they collected, they had to open the confidence interval wide. And even then, they did not collect the information to check against other population norms to make sure they had a representative sample, or to check if they are double counting.

Cluster sampling is used to minimize the work by the samplers. 47 data points are not exactly exhaustive.

What would be the best way to confirm this study? Do another sample with 50 other clusters. A 2 sample T test will easily tell if the samples are the same or different.

[techweenie]

Originally posted by Rodeo
So how many Iraqis has Bush's war killed?

Likely, more than the "madman" Saddam killed in the last 20 years.

[Jeff Higgins]

It seems to me that statisticians that have reviewed the methodology employed in this study have pretty much discounted it accross the board. It has not passed any sort of peer review. I'm no statistician, so I will go with what these peers have to say. Funny how a "medical doctor" feels he can "set the record straight" on statistical analysis of a population like this, and then go on to claim the numbers have been attacked on ideological, not scientific, grounds. He sounds very much like the ideolog here.

As an admitted "layman" in this arena, it also makes sense to me when these peers point out the alarming rate of death needed to support these numbers. A thousand per day? Yet we only hear of, on the average, a few per day? Not to trivialize their suffering and death, but it seems several dozen deaths on any given day make for very big news. Can anyone show verification of any day in the last year where there were at least a thousand deaths reported? I can't remember one, much less several, much less daily. Maybe I'm just not paying attention.