Much to my regret, I find myself forced to revisit old territory on the subject of civilian casualties in Iraq and the questionable means being used to estimate them. In 2004 a study by researchers from Johns Hopkins University was published in The Lancet. It purported to be a statistical analysis of civilian deaths in Iraq during the first year after the invasion. It was immediately latched onto by the left and they began talking about 200,000 civilian casualties and other alarming numbers despite the fact that the study was poorly conceived, shoddily executed and a classic example of bad methodology.
The study was widely debunked and analyzed and deconstructed and proven to have almost no validity, but academic ideologues tend to stick by their guns in the face of superior analysis, obvious logic and widespread criticism. Opposition just makes them get out their Excel spreadsheets and try harder. The problem is that despite great determination they don't learn from their mistakes, so this Thursday we find in The Lancet another attempt by the same group to apply what is basically the same flawed methodology to the first three years of coalition occupation in Iraq, operating on the bizarre assumption that if you do something incompetently twice that somehow makes the results more convincing, especially if you do it on a grander scale.
Before I go any further I want to point out that I deplore any civilian casualties and think that the post-invasion period in Iraq has been extremely badly managed. Even one civilian death is tragic, and the ongoing nature of conflict there is certainly cause for alarm. That said, I feel that it is still necessary to address the problems with this new study in The Lancet in the interest of truth, because there is every reason to believe that it will be promoted far out of proportion to its accuracy and grossly misrepresented for partisan purposes just as the 2004 study was. By all means let's stop the deaths in Iraq, but let's do it for legitimate reasons and not based on questionable research.
As with the earlier study, the media and the left are going to jump on this one despite the flaws, because their dramatic new number for excess civilian deaths is 654,965 over the course of three years, which is an awful lot of deaths. It's even more per year than the largest potential estimate in their old study, presumably on the theory that if no one believed your first whopper you should tell an even bigger lie with a straight face and eventually people will believe you.
Article comments
— go to most recent comments1 - Ruvy in Jerusalem
Dave,
Never forget what Samuel Clemens said many years ago.
"There's lies, there's damned lies, and there's stastistics."
2 - Alec
Dave " I think you go overboard in debunking the Lancet’s figures, but I largely agree with you. On the other hand, the most thorough mainstream debunking of these figures that I recently heard was on a public radio station, supposedly the heart of the left, so I don’t think that you can simplistically proclaim that the left will universally embrace this dis-information. Rather, there is a pseudo-progressive core that foolishly believes that if you can demonstrate that a war kills some magically big number of innocents or civilians, then everyone will equally magically agree that war is bad and end all hostilities.
On the other hand, working through the data to conclude that the number of firearm deaths of men suggests “active involvement in combat” is not necessarily correct by any means. For example, it is clear that there are factions that target males who attempt to volunteer for the army or the police force, and that Sunni or Shiite males are targeted by one side or the other in an attempt to eliminate any potential future opposition, whether or not these people ever actively take up arms as insurgents. Kurds have more often been victims of bomb attacks since there is already a degree of Kurdish separation from areas that have mixed Sunni and Shiite enclaves. It is not quite ethnic cleansing, since women and children are not yet equally targeted, but still, any attempt to draw simple distinctions between civilian and combatant just plays the same game as the Lancet.
3 - Dave Nalle
Dave - I think you go overboard in debunking the Lancet's figures, but I largely agree with you. On the other hand, the most thorough mainstream debunking of these figures that I recently heard was on a public radio station, supposedly the heart of the left, so I don't think that you can simplistically proclaim that the left will universally embrace this dis-information.
I have some confidence in the honesty of public radio. But was this a debunking of these figures or the 2004 study? They're already crowing about these new numbers at DailyKos and DU, but I haven't heard anyone criticising them yet.
Rather, there is a pseudo-progressive core that foolishly believes that if you can demonstrate that a war kills some magically big number of innocents or civilians, then everyone will equally magically agree that war is bad and end all hostilities.
That appears to be the state of mind which is motivating the authors of the study who seem to believe that if they can throw out a really, really big number as a possible outcome of the study then it will make everything better.
What I didn't point out specifically in my analysis is that my reduced amount based on their research is almost exactly the same as the low-end version of their conclusions adjusted for their (unstated) margin of variation. Despite the fact that their conclusion states that 655,000 people died, their data suggests that what they really mean is that somewhere between 131,000 and 655,000 died, but they aren't promoting that lower number and it's not the one that's' going to get all the press.
Dave
4 - Dave Nalle
It seems the media is already picking up the magic number after only 24 hours. It's showing up on the BBC, CNN, ABC and most of the newspapers as well, pretty much uncontested and unqualified.
My favorite so far in the media is this editorial which refers to this pile of claptrap as a 'highly credible study' and compares Bush to Pol Pot.
Dave
5 - Dave Nalle
Today's International Herald Tribune has an article discussing the study with a quote from the Iraqi government rejecting the study an saying that it's 10 times higher than any realistic casualty estimate.
Dave
6 - troll
Dave - love the transformations and adjustments...keep up the analysis and you soon might be able to 'show' that no one died in Iraq today
the improvement in your work this time out is that you don't rely on Iraq Body Count methodology for your 'corrected' estimate
...but please let us know on what studies/evidence you base your claims for example that the Iraq countryside has been significantly 'less violent' than the cities or that underrepresented areas have experienced significantly fewer excess deaths than overrepresented ones - I've heard that information about much of the country is inaccessible to observers and government employees...that there is no central agency visiting morgues and counting death certificates
and why should anyone believe such convenient claims in the absence of data - ?
this brings us back to the problem of how to improve on cluster study data collection methodologies in violent environments which is what these researchers have been working on around the world for years
rather than accusing them of being political hacks and liars I suggest that you offer them your ideas on methodological improvements for their third time out in Iraq
7 - Georgio
Dave even though I agree with your article on what it says I couldn't help but notice that the Bush team works the same formula..
IF YOU TELL THE SAME LIE OVER AND OVER PPL WILL BELIEVE IT ....this formula has worked for Bush for the last four years and the ppl bought it..So if the Dems want to jump on this and use it to their gain I will give them credit for it..who cares whats right or wrong anymore, Bush has lied so much already that Dems would be smart to take this flawed info and go with the big number and then say it over and over again even run ads with the big number HELL it will take four years for the American ppl to figure it out.
8 - Dr. Kurt
Dave, I am echoing what Troll said. In the US, rural and urban crime/murder rates are generally pretty equal. I need data to support your assertion that the rates are different in Iraq; do you have any? I am willing to bet that most foreign reporters hang out in cities, and that the Iraqi media and government pay more attention to urban events than to rural ones. Today, fourteen bodies were discovered in an orchard outside Baghdad... would it have made the news at all if they were farther from the big city?
BTW everyone, "methodology" is the study of methods. It is pure academic pretension to tack the "ology" on when we are really just discussing methods. Thank you, and stay skeptical.
9 - McNab
"By all means let's stop the deaths in Iraq, but let's do it for legitimate reasons and not based on questionable research."
Because if the murder of innocent Iraqi men, women and children by the US was stopped while it was permissible it would be a fucking horrific tragedy, right Dave!
10 - Engineer
We've dropped 240,000 cluster bombs, 3 dead for each bomb seems like a decent average to me.
No matter what you think about statistics, this survey team visited 1,840 homes and found 547 post invasion deaths. 1 dead for every 4 houses randomly selected is sad no matter how you look at it.
11 - Dave Nalle
this brings us back to the problem of how to improve on cluster study data collection methodologies in violent environments which is what these researchers have been working on around the world for years
IMO cluster studies should probably be abandonned alltogether. I also think it's significant to note that none of the groups they had gathering data were attacked or had any significant problems collecting their data.
But even putting the cluster system aside, how about as a working principle - you don't count the dead with a fucking opinion poll.
Hell, huge numbers of Iraqis have cell phones now - why not just call them all on their cell phones and question them. That sounds pretty damned safe.
But for a start, if you're going to do cluster studies, you have to break the sources down proportional to where the population lives. You can't just poll the cities and ignore the villages and rural areas.
rather than accusing them of being political hacks and liars I suggest that you offer them your ideas on methodological improvements for their third time out in Iraq
As someone earlier suggested, why not just count the dead? This time out they went to great pains to make sure that almost all the people they questioned provided death certificates for the people they claimed had died - and about 80% had them. If those death certificates exist, why not go to the issuing body and get a total and then add 20% to it as a 'fudge factor'? That would be a hell of a lot more accurate than this effort.
Dave
12 - Dave Nalle
Dave even though I agree with your article on what it says I couldn't help but notice that the Bush team works the same formula..
IF YOU TELL THE SAME LIE OVER AND OVER PPL WILL BELIEVE IT
Georgio. Did I say one word about Bush in this article?
Dave
13 - MCH
CANDIDATE-VETERAN ATTACKS BUSH ON IRAQ
By DENNIS CONRAD, Associated Press Writer
Sep 30, 2006
WASHINGTON - An Illinois congressional candidate who lost both her legs during combat in Iraq said Saturday that President Bush has no real strategy for securing the war-ravaged nation, just political talk designed to appeal to voters.
"Instead of a plan or a strategy, we get shallow slogans like 'mission accomplished' and 'stay the course,'" former U.S. Army Captain Tammy Duckworth said in the Democrats' weekly radio address. "Those slogans are calculated to win an election. But they won't help us accomplish our mission in Iraq."
Duckworth, who co-piloted a Black Hawk helicopter that crashed while under a rocket grenade attack almost two years ago, also criticized Bush and others in his administration for accusing anyone who challenges the president's policies of "cutting and running."
"Well, I didn't cut and run, Mr. President. Like so many others, I proudly fought and sacrificed," Duckworth said. "My helicopter was shot down long after you proclaimed 'mission accomplished.'"
At a GOP fundraiser Thursday in Alabama, Bush said, "The party of FDR and the party of Harry Truman has become the party of cut and run."
Duckworth is seeking the suburban Chicago seat against Illinois state Sen. Peter Roskam.
In her address, Duckworth, now a major in the Illinois National Guard, also lashed out at the GOP-led Congress for refusing to do its job of holding the Bush administration accountable for its flawed Iraq policy.
"We need a Congress that will ask the tough questions and work together for solutions rather than attacking the patriotism of those who disagree," she said. "It is time to encourage Iraqi leaders to take control of their own county and make the tough choices that will stop the civil war and stabilize the country."
14 - Bliffle
The estimate of 600k seems extraordinary on the face of it, but the article has failed to point out failings in the methods. The Lancet report has not been refuted, merely had curses thrown at it.
15 - John Bishop
Your analysis is incorrect in many ways. At this time I'd like to point out two ways.
You suggest that high violence areas were oversampled by 20% because of rounding error. I've assembled the data from Table 1 of both the Lancet articles to attempt to determine the over- and under-sampling. I assigned each governate to High, Medium and Low violence categories according to Figure 3 of the Lancet article. These assignments are congruent with the ones you made, except you mentioned only the overrepresented high violence areas and the underrepresented low violence areas. There also happen to be under-represented high violence areas and over-represented low violence areas. I then calculated the under/over representation for each of the 3 categories. I did this with and without weighting the governates by their proportion of the total population, but the results are nearly identical. Here are the results (2006 sample only):
High violence: overrepresented by 4.1%
Medium violence: underrepresented by 2.6%
Low violence: underrepresented by 1.5%
I note that although you and I both include the low violence areas Dahuk and Muthanna that Burnham et al actually omit from their analyses because they did not sample it (that is, the ~1 million people in those provinces were essentially subtracted from the total population. If we drop these two provinces from consideration, as the study does, and weight each province by its proportion of the total population, then we obtain:
High violence: over-represented by 3.2%
Medium violence: under-represented by 4.3%
Low violence: *over*-represented by 1.1%
Net result: its unclear that there is any over-representation at all of violent areas.
Even if you're adjustment percentages were correct, you applied them incorrectly. Your calculation was:
654,965 * (-.3 + -.2 + -.3) = 130,993
In fact you should multiply:
654,665 * .7 * .8 *.7 = 256,746
If we correct for your error regarding over-representation of high violence areas by changing the -.2 to -.04, then you have
654,665 * .7 * .96 *.7 = 308,000
Three hundred and eight thousand excess civilian casualties!
And that's assuming your other adjustments are justified, which as others have pointed out, they are not.
16 - Alec
Dave - RE: I have some confidence in the honesty of public radio. But was this a debunking of these figures or the 2004 study? They're already crowing about these new numbers at DailyKos and DU, but I haven't heard anyone criticising them yet.
The public radio report very clearly debunked the recent Lancet study, and also noted the deficient methodology of the earlier study. I didn't pay much attention to the report at the time, didn't even note which public radio program it was, because I under-estimated the degree to which people would gloam onto this stuff and pass it off as valid analysis.
17 - RJ Elliott
OUTSTANDING article, Dave. This should not only be a "Blogcritic pick of the week," but it should also be picked up and echoed by other fair-minded bloggers.
Great job!
18 - Bliffle
"The Lancet" is peer reviewed, isn't it?
19 - Dave Nalle
MCH, #13 is off-topic and a violation of copyright. Take it somewhere else, perhaps.
Dave
20 - Dave Nalle
You suggest that high violence areas were oversampled by 20% because of rounding error.
Keep in mind that just as the study conclusions are an 'estimate', I'm making an estimate as well.
I've assembled the data from Table 1 of both the Lancet articles to attempt to determine the over- and under-sampling. I assigned each governate to High, Medium and Low violence categories according to Figure 3 of the Lancet article.
That's more or less what I did as well, except I gave the medium regions a pass.
These assignments are congruent with the ones you made, except you mentioned only the overrepresented high violence areas and the underrepresented low violence areas.
Actually, I only mentioned those which were over or underrepresented by at least 25%.
There also happen to be under-represented high violence areas and over-represented low violence areas.
Yes, but under or overrepresented at a much smaller percentage than the ones I noted.
I then calculated the under/over representation for each of the 3 categories. I did this with and without weighting the governates by their proportion of the total population, but the results are nearly identical. Here are the results (2006 sample only):
You have to weight the governorates by population or at the very least by number of clusters as well.
High violence: overrepresented by 4.1%
Medium violence: underrepresented by 2.6%
Low violence: underrepresented by 1.5%
I note that although you and I both include the low violence areas Dahuk and Muthanna
I actually didn't factor those areas in, I merely note that they were not reported in the study. With no data at all to go on it didn't seem fair to include them.
And as I said, my final number was merely an estimate. I don't see how you could possibly come up with a more accurate number working only with the tables and the data in the report, because the report as published does not break casualties down by governorate. To get an accurate count you'd need to have that data.
that Burnham et al actually omit from their analyses because they did not sample it (that is, the ~1 million people in those provinces were essentially subtracted from the total population. If we drop these two provinces from consideration, as the study does, and weight each province by its proportion of the total population, then we obtain:
High violence: over-represented by 3.2%
Medium violence: under-represented by 4.3%
Low violence: *over*-represented by 1.1%
Net result: its unclear that there is any over-representation at all of violent areas.
I'd have to go through and do all the math and come up with somethinng more precise than my original estimate, but I don't see how this result can possibly be correct when some of the largest provinces were under or overrepresented by as much as 30% because of rounding.
Even if you're adjustment percentages were correct, you applied them incorrectly. Your calculation was:
654,965 * (-.3 + -.2 + -.3) = 130,993
In fact you should multiply:
654,665 * .7 * .8 *.7 = 256,746
A different way of looking at it, certainly. Not necessarily more valid unless you assume that the geographical underrepresentation and the rounding somehow apply more to military age men than to women. At the very least you have to do the removal of military age men as a separate calculation.
If we correct for your error regarding over-representation of high violence areas by changing the -.2 to -.04, then you have
654,665 * .7 * .96 *.7 = 308,000
So, if we accept your approach, the study's estimate is only off bya factor of 100%? Wow, that's great news.
Three hundred and eight thousand excess civilian casualties!
In the course of three years, of course.
And that's assuming your other adjustments are justified, which as others have pointed out, they are not.
Maybe I should point out again that even my 130,000 casualties estimate is double any of the figures reached by more verifiable methods. If anything my adjustments were super-conservative.
And even you don't have the temerity here to try to claim the the study is in any way accurate or anything but horribly misrepresentative.
Dave
21 - Dave Nalle
"The Lancet" is peer reviewed, isn't it?
From what I can tell their medical articles are peer reviewed, but some of their other material is not. There is no indication that this study has been peer reviewed in any way. They seem to publish two sorts of articles. Straight science or medicine articles are peer reviewed and appear primarily in their print edition. Articles of a mostly political or policy nature appear in their online edition without peer review.
Dave
22 - Dave Nalle
I read some very good comments on the study over at pajamas media in response to an emotionall critique from Iraq by Omar Fadil.
A couple of points were raised there which didn't even occur to me, including the idea that if there are 650,000 dead, where are the 2 million corresponding wounded and where are the graves for all these people?
But my favorite comment is from one fellow who points out that if he'd done a paper in college where the results had a margin of error of +/- 40% he'd have been laughed out of class.
Dave
23 - KillKitten
I'm not sure about the methods of the study, but the number they reached is obviously wrong. If there were that many dead people in Iraq there would be no way to minimize it or cover it up.
I'm most bothered by how the media is just running with it and not thinking twice. I did a search for 'iraq casualties' on google news and got over 1000 entries for news stories in the past 24 hours, adn they're all the same thing just repeating that big number and wringing their hands.
24 - McNab
Dave, you haven't been paying attention. Just turn on your television and start surfing US news channels if you want to see dead and wounded Iraqis. There are scores of news crews all over Iraq filming US military personnel and US contracted death squads butchering innocent Iraqi men, women and children and Eye Witless News can't wait to beam it into the living rooms of the nation.
25 - Dave Nalle
McNab, you live in a truly bizarre alternate reality. I watch the news from time to time. And yes, I've seen reports direct from Iraq. But I certainly haven't seen mountains of bodies or anything resembling 'death squads'.
In three years we have a handful of incidents where coalition soldiers stepped off the reservation - and this while faced with endless terrorist attacks which violate every one of the rules of war. What's shocking about Iraq is the forbearance which our forces have shown.
Dave