The Lancet study saying that there are as many as 655,000 extra civilian deaths in Iraq as a result of the war and occupation has attracted considerable interest. Some say the study is right, others point to the disparity between it and other analyses.
There is a small point in the methodology of the Lancet study which I believe may offer a quick and simple facial test of the study’s validity.
The Lancet authors report that they attempted to confirm the death reports given to them by their interview subjects. They report being able to confirm 80% of the death reports by finding official death certificates filed by local and provincial governments. This augments the credibility of this part of their methodology – if, in a war zone, 80% of your self-reports are verified by government records, the reporting you’re getting is probably of good quality.
However, if 80% of the sample’s death reports are verifiable, then a similar fraction of the total population should show similar levels of documentation. There should be 500,000 death certificates on file through Iraq. If there are not, then the study authors have destroyed the presumption of the validity of their sample, because there’s a huge disparity between the sample and the population. From where I sit, there are either a half million death certificates on file somewhere (in which case the Lancet study is noncontroversial, and we have to start asking what is wrong with Iraq Body Count, Brookings Institute, and other non-Pentagon estimates that put the civilian toll in the 50 to 100K range), or we have to ask why the study sample shows such an enormous differentiation on the documentation question.
In other words, show me the death certificates.
If death certificates are maintained locally, there is probably nobody bothered to compile national statistics in the face of more pressing business.
Indeed, the Lancet study would presumably not have been done at all if there were good national statistics.
Comment by ohwilleke — October 12, 2006 @ 3:26 pm |
If death certificates are maintained locally, and it’s possible to find them in the places where the Lancet did their survey but not elsewhere, it raises a huge flag about their sample.
Comment by bobhayes — October 12, 2006 @ 3:38 pm |
The Iraq Body Count consists only of deaths reported in major newspapers. In New York City, perhaps 20-30 deaths are reported in the NY Times most days, while at least 200 deaths occur in NYC each day.
Iraq Body Count is a minimum – a small fraction of actual deaths. There’s no inconsistency between a low minimum number and a much larger estimate of the total number.
Brookings data says right in it, “NOTE ON “IRAQI CIVILIANS KILLED” TABLES:
Information for May 2003-December 2005 is based upon data from Iraq Body Count.”
So Brookings also, is a minimum.
Why does anyone think there’s any inconsistency here? 650,000 civilians have died as a result of this war, about 600,000 from violence, and about 50,000 of which have been reported in the newspapers.
The Lancet is a peer-reviewed journal. It’s 180 years old. They put their reputation behind every paper they publish. While there is still some chance of error or fraud, it’s quite unlikely — and merely saying “Gee, that seems like a lot” is no basis on which to challenge the study.
Comment by Anon — October 12, 2006 @ 9:31 pm |
Doesn’t the filing of death certificates require that there first be a functioning government?
Comment by dr sardonicus — October 12, 2006 @ 10:23 pm |
Robert,
You are comparing two very different senses of access to death certificates.
In the Lancet study, groups of surveyors go house to house and ask about deaths and ask for the death certificates for those deaths. If the families of the dead get death certificates from some local authority and keep those certificates, then you would expect that most deaths will be documented by a death certificate (apparently, you need a death certificate to get a body past the check points in order to have a funeral).
In a national survey of death certificates issued, a central authority asks a list of scattered and disparate officials for the number of death certificates issued in their region (maybe those officials are the numerous individuals who write out and sign those death certificates, more likely they are higher level officials who are asked to ask their scattered underlings to give them a report of death certificates issued). If there are break downs in communication (if officials in Anbar province aren’t speaking to the officials from Baghdad, if the Anbar officials who are supposed to report are also under pressure to bring the death rate down, if the local morgue official doesn’t want to look like a collaborator by talking to the provincial authority, if the notary is too busy writing up death certificates to fill out the reporting form, if the notary is ill trained and doesn’t know they are supposed to send in a report, if the local notaries aren’t being paid and therefore feel no great duty to waste an hour writing up a list to send to someone who hasn’t done a damn thing for them, if the local official gets shot for collaboration, if the phone service is down, etc etc), then the local lists can be an undercount or the central list can be an undercount. Add to that that the government department publishing the list is part of a government that is trying to claim that order is being restored (and is now denying reporters direct access to the local morgue officials) and is therefore certainly under pressure to under count the numbers.
Just because someone has a legitimately issued death certificate doesn’t mean that the pink copy of that certificate made it to the archive, nor does it mean that anyone compiled the number of pink copies in the archive, nor that that number got passed on to central processing, nor that central processing didn’t massage their numbers in order to look better, nor that the administrator didn’t step in to massage the final report to make the final number look better. The Lancet study looks at the white copy someone’s relative needed to hold a funeral, the official numbers are a long game of telephone with the number of pink copies. It just isn’t the same thing.
Comment by Charles S — October 12, 2006 @ 10:37 pm |
You know, after the first study, the authors basically said: “this is a preliminary result, our error bars are huge, no small group of researchers can do this adequately. Someone needs to set up a serious survey of the dead.” If anyone had done that then, maybe you’d have a better answer on exactly how many death certificates have been written in Iraq in the past 3 1/2 years. No one did that, so we have another similar but bigger study, confirming the previous study results and going on to say there are now another 3 to 700,000 dead since the previous study. It would be nice if the Iraqi government were a well oiled machine that had good record keeping and efficient and reliable tabulation of national data that could confirm or disprove these results. It would be nice if they weren’t deep in a bloody collapse into a civil war.
Comment by Charles S — October 12, 2006 @ 10:52 pm |
And I see dr sardonicus beat me to my response by managing to say it in a single question.
Comment by Charles S — October 12, 2006 @ 10:53 pm |
Oh, and Daniel at Crooked Timber also rebuts your argument.
Comment by Charles S — October 12, 2006 @ 11:00 pm |
Dr. Sardonicus, your point is valid. But it doesn’t undermine the argument; it points out that there could be a very large differential between sampled and unsampled areas. If the people who were surveyed could get death certificates, why couldn’t others? Do the reasons point to other possible differences between the sample and the population?
Charles, your point concerning custody of records is also valid. I suppose my test would have to include checking the state of official records in the archives in the areas sampled as well as randomly elsewhere to see if there was a variation.
The Crooked Timber argument is interesting, thanks for the link.
Comment by Robert — October 12, 2006 @ 11:52 pm |
I don’t understand what you’re claiming here. Is there another survey of Iraqi households which showed that most Iraqi households can not get death certificates? If not, then what “others” are you referring to here?
Comment by Ampersand — October 14, 2006 @ 2:56 pm |
Amp, according to the Lancet team, some high percentage of their interviewees had access to the death certificate for the household member(s) who were dead.
Accordingly, if this sample is truly representative, then lots of other people in Iraq should also have the death certificates of their casualties. Where are they? (In other words, fill a couple soccer stadiums with people waving death certificates, and the objections to the study disappear into mist.)
I don’t expect the Lancet folks to suddenly materialize with a mile-high stack of death certificates; I’m merely noting that that stack should be somewhere (even if it’s distributed all across the country). If the stack doesn’t exist, then it raises questions about the representativeness of the sample.
Comment by Robert — October 14, 2006 @ 5:48 pm |
You think?
Comment by Daran — October 14, 2006 @ 6:02 pm |
Just noticed this passage, from Bob’s post:
This may be part of the reason we’re talking past each other. Do you understand that it was the people interviewed – not local government agencies – who had the death certificates checked by the Lancet interviewers?
You’re saying that if most Iraqi households in which a household member had died had possession of death certificates, and if causalities had been high, we could rationally expect Iraqis to spontaneously fill soccer stadiums in order to wave the death certificates in the air? And the fact that they haven’t done so disproves the Lancet survey?
No, that’s obviously not what you’re saying. But what are you saying? Please tell me, because you’re not making sense. What, specifically and realistically, do you expect to see happen if casualties are high, and if most Iraqi households in which a household member had died have possession of death certificates; that is not happening?
If I wanted to find out how many Iraqi households had had a member die, and could prove it because they possess a death certificate, here’s what I’d do: I’d randomly select one or two thousand Iraqi households to interview. I’d have trained Iraqi interviewers go knocking on doors and asking if any resident of this address had died since January 2002; and then after they had told me, I’d ask them to show me the death certificate. Then I’d compile the statistics, and report the answer.
But it’s obvious that you don’t consider this method a reasonable way of finding out information. So what alternative method do you suggest, that you think is more reliable?
Comment by Ampersand — October 15, 2006 @ 4:14 am |
I don’t “expect” to see anything happening. I suggested this test as a logical exercise; I don’t know what the answer is. I think you’re overestimating my hostiity to the study; it could be on the right lines; I don’t know. If I knew how many Iraqis had death certificates, then I would have more information about the study’s validity.
I have some problems with the methodology which I’ll detail later in a post, but those problems don’t arise to the level of me saying presumptively that the study is rubbish. They’re just ways that it could be wrong.
Comment by Robert — October 15, 2006 @ 4:43 am |
What test? I don’t intend this as hostile, but I honestly don’t understand what the test you’re proposing is, Bob. Please spell it out for me.
Comment by Ampersand — October 15, 2006 @ 6:45 am |
There seems to be a disconnect between the inferred number of death certificates held by family members, and the number of casualties recognised by the central government. A survey of local registry offices would perhaps throw some light on where that disconnect lies.
Comment by Daran — October 15, 2006 @ 7:10 am |
Expecting that the government will count all deaths accurately, is expecting that the government will be more efficient and accurate now than it was before the war began. From the companion paper:
Les Roberts, one of the study’s authors, made a similar point in an interview:
Les Roberts’ claims seem plausible to me. Is there any reason to think that government officials, during a war, are going to be keeping accurate mortality statistics?
The question is whether or not “passive” methods – that is, depending on the press or the government to have accurately collected death counts, and then totaling up what those other agencies have collected – can reasonably be expected to be as accurate as active methods – that is, a representative random survey. I don’t know of any theory under which suggests that passive data collection is more accurate than active data collection (unless you can demonstrate that the active data collection’s methodology is horribly flawed).
Comment by Ampersand — October 15, 2006 @ 9:56 am |
Assuming that you’re replying to my last remark, it certainly isn’t my premiss that “the government will count all deaths accurately”. Quite the contrary, they do not appear to be able even to count the bodies in a single city morgue. I merely remarked that such a survey would allow us to map out the disconnect more precisely.
Comment by Daran — October 15, 2006 @ 12:02 pm |
My comment was intended as a reply to one of the arguments I think Robert may be making, not as a response to you. Sorry for the mix-up.
Comment by Ampersand — October 15, 2006 @ 12:31 pm |
What Robert is suggesting, I think, is that we see how many of the violent deaths that were reported in the survey are in the official files. If, say, only 10% of the death certificates in question are on file, then the total number of certificates on file for violent deaths should be on the order of 10% of the number of violent deaths estimated in the Lancet survey. If not, then this suggests that the survey sample was not representative.
It seems reasonable in theory, though I’m not sure that the number of death certificates available on file is sufficient to guarantee accuracy.
Comment by Brandon Berg — October 15, 2006 @ 2:41 pm |
The test I intend is broadly along the lines of what Brandon says, although I’d compare apples to apples. Show up in a town, say “everyone who has lost a household member, come over here”, and then ask them for death certificates. If 80% of them say “here you go”, the Lancet study is strengthened. If 10% of them say “here you go” and 90% say “are you nuts, where would I get a death certificate, this is a war zone”, then the Lancet study is weakened; how come all its respondents had access to this paperwork?
Comment by Robert — October 15, 2006 @ 3:04 pm |
So you’re proposing that someone go and do to a whole town, what the Lancet researchers did to individual households in random clusters, because what? A survey of one whole town is more representative of Iraq than random clusters?
Comment by Daran — October 15, 2006 @ 3:34 pm |
You may be overestimating the degree to which we’re wedded to the study findings. I’m certainly not to any great degree, and I doubt Amp is either. Nothing would delight me more, than to find out that the casualties are much lower than appears to be the case.
However, this discussion has taken place against a persistant backdrop of trashing of any scientific results which go contrary to the Republican political agenda, not to mention personal attacks on the scientists themselves. It’s hardly surprising then, that we take a skeptical view of right-wing skepticism.
Comment by Daran — October 15, 2006 @ 3:49 pm |
Robert’s objection is wacky. The Lancet authors were shown the death certificates for 80% of the cases they looked at. Of course, a bigger study will narrow the confidence interval to some extent — but that has no impact on the validity of this study. If it’s as accurate as the sample size allows, that’s all you can ask.
Comment by Gene Callahan — October 17, 2006 @ 7:07 am |