Creative Destruction

October 12, 2006

Lancet Study Methodology Gives Validation Test

Filed under: Current Events,Iraq — Robert @ 2:40 pm

The Lancet study saying that there are as many as 655,000 extra civilian deaths in Iraq as a result of the war and occupation has attracted considerable interest. Some say the study is right, others point to the disparity between it and other analyses.

There is a small point in the methodology of the Lancet study which I believe may offer a quick and simple facial test of the study’s validity.

The Lancet authors report that they attempted to confirm the death reports given to them by their interview subjects. They report being able to confirm 80% of the death reports by finding official death certificates filed by local and provincial governments. This augments the credibility of this part of their methodology – if, in a war zone, 80% of your self-reports are verified by government records, the reporting you’re getting is probably of good quality.

However, if 80% of the sample’s death reports are verifiable, then a similar fraction of the total population should show similar levels of documentation. There should be 500,000 death certificates on file through Iraq. If there are not, then the study authors have destroyed the presumption of the validity of their sample, because there’s a huge disparity between the sample and the population. From where I sit, there are either a half million death certificates on file somewhere (in which case the Lancet study is noncontroversial, and we have to start asking what is wrong with Iraq Body Count, Brookings Institute, and other non-Pentagon estimates that put the civilian toll in the 50 to 100K range), or we have to ask why the study sample shows such an enormous differentiation on the documentation question.

In other words, show me the death certificates.

About these ads

24 Comments »

  1. If death certificates are maintained locally, there is probably nobody bothered to compile national statistics in the face of more pressing business.

    Indeed, the Lancet study would presumably not have been done at all if there were good national statistics.

    Comment by ohwilleke — October 12, 2006 @ 3:26 pm | Reply

  2. If death certificates are maintained locally, and it’s possible to find them in the places where the Lancet did their survey but not elsewhere, it raises a huge flag about their sample.

    Comment by bobhayes — October 12, 2006 @ 3:38 pm | Reply

  3. The Iraq Body Count consists only of deaths reported in major newspapers. In New York City, perhaps 20-30 deaths are reported in the NY Times most days, while at least 200 deaths occur in NYC each day.

    Iraq Body Count is a minimum – a small fraction of actual deaths. There’s no inconsistency between a low minimum number and a much larger estimate of the total number.

    Brookings data says right in it, “NOTE ON “IRAQI CIVILIANS KILLED” TABLES:
    Information for May 2003-December 2005 is based upon data from Iraq Body Count.”

    So Brookings also, is a minimum.

    Why does anyone think there’s any inconsistency here? 650,000 civilians have died as a result of this war, about 600,000 from violence, and about 50,000 of which have been reported in the newspapers.

    The Lancet is a peer-reviewed journal. It’s 180 years old. They put their reputation behind every paper they publish. While there is still some chance of error or fraud, it’s quite unlikely — and merely saying “Gee, that seems like a lot” is no basis on which to challenge the study.

    Comment by Anon — October 12, 2006 @ 9:31 pm | Reply

  4. Doesn’t the filing of death certificates require that there first be a functioning government?

    Comment by dr sardonicus — October 12, 2006 @ 10:23 pm | Reply

  5. Robert,

    You are comparing two very different senses of access to death certificates.

    In the Lancet study, groups of surveyors go house to house and ask about deaths and ask for the death certificates for those deaths. If the families of the dead get death certificates from some local authority and keep those certificates, then you would expect that most deaths will be documented by a death certificate (apparently, you need a death certificate to get a body past the check points in order to have a funeral).

    In a national survey of death certificates issued, a central authority asks a list of scattered and disparate officials for the number of death certificates issued in their region (maybe those officials are the numerous individuals who write out and sign those death certificates, more likely they are higher level officials who are asked to ask their scattered underlings to give them a report of death certificates issued). If there are break downs in communication (if officials in Anbar province aren’t speaking to the officials from Baghdad, if the Anbar officials who are supposed to report are also under pressure to bring the death rate down, if the local morgue official doesn’t want to look like a collaborator by talking to the provincial authority, if the notary is too busy writing up death certificates to fill out the reporting form, if the notary is ill trained and doesn’t know they are supposed to send in a report, if the local notaries aren’t being paid and therefore feel no great duty to waste an hour writing up a list to send to someone who hasn’t done a damn thing for them, if the local official gets shot for collaboration, if the phone service is down, etc etc), then the local lists can be an undercount or the central list can be an undercount. Add to that that the government department publishing the list is part of a government that is trying to claim that order is being restored (and is now denying reporters direct access to the local morgue officials) and is therefore certainly under pressure to under count the numbers.

    Just because someone has a legitimately issued death certificate doesn’t mean that the pink copy of that certificate made it to the archive, nor does it mean that anyone compiled the number of pink copies in the archive, nor that that number got passed on to central processing, nor that central processing didn’t massage their numbers in order to look better, nor that the administrator didn’t step in to massage the final report to make the final number look better. The Lancet study looks at the white copy someone’s relative needed to hold a funeral, the official numbers are a long game of telephone with the number of pink copies. It just isn’t the same thing.

    Comment by Charles S — October 12, 2006 @ 10:37 pm | Reply

  6. You know, after the first study, the authors basically said: “this is a preliminary result, our error bars are huge, no small group of researchers can do this adequately. Someone needs to set up a serious survey of the dead.” If anyone had done that then, maybe you’d have a better answer on exactly how many death certificates have been written in Iraq in the past 3 1/2 years. No one did that, so we have another similar but bigger study, confirming the previous study results and going on to say there are now another 3 to 700,000 dead since the previous study. It would be nice if the Iraqi government were a well oiled machine that had good record keeping and efficient and reliable tabulation of national data that could confirm or disprove these results. It would be nice if they weren’t deep in a bloody collapse into a civil war.

    Comment by Charles S — October 12, 2006 @ 10:52 pm | Reply

  7. And I see dr sardonicus beat me to my response by managing to say it in a single question.

    Comment by Charles S — October 12, 2006 @ 10:53 pm | Reply

  8. Oh, and Daniel at Crooked Timber also rebuts your argument.

    Comment by Charles S — October 12, 2006 @ 11:00 pm | Reply

  9. Dr. Sardonicus, your point is valid. But it doesn’t undermine the argument; it points out that there could be a very large differential between sampled and unsampled areas. If the people who were surveyed could get death certificates, why couldn’t others? Do the reasons point to other possible differences between the sample and the population?

    Charles, your point concerning custody of records is also valid. I suppose my test would have to include checking the state of official records in the archives in the areas sampled as well as randomly elsewhere to see if there was a variation.

    The Crooked Timber argument is interesting, thanks for the link.

    Comment by Robert — October 12, 2006 @ 11:52 pm | Reply

  10. If the people who were surveyed could get death certificates, why couldn’t others? Do the reasons point to other possible differences between the sample and the population?

    I don’t understand what you’re claiming here. Is there another survey of Iraqi households which showed that most Iraqi households can not get death certificates? If not, then what “others” are you referring to here?

    Comment by Ampersand — October 14, 2006 @ 2:56 pm | Reply

  11. Amp, according to the Lancet team, some high percentage of their interviewees had access to the death certificate for the household member(s) who were dead.

    Accordingly, if this sample is truly representative, then lots of other people in Iraq should also have the death certificates of their casualties. Where are they? (In other words, fill a couple soccer stadiums with people waving death certificates, and the objections to the study disappear into mist.)

    I don’t expect the Lancet folks to suddenly materialize with a mile-high stack of death certificates; I’m merely noting that that stack should be somewhere (even if it’s distributed all across the country). If the stack doesn’t exist, then it raises questions about the representativeness of the sample.

    Comment by Robert — October 14, 2006 @ 5:48 pm | Reply

  12. fill a couple soccer stadiums with people waving death certificates, and the objections to the study disappear into mist.

    You think?

    Comment by Daran — October 14, 2006 @ 6:02 pm | Reply

  13. Just noticed this passage, from Bob’s post:

    The Lancet authors report that they attempted to confirm the death reports given to them by their interview subjects. They report being able to confirm 80% of the death reports by finding official death certificates filed by local and provincial governments.

    This may be part of the reason we’re talking past each other. Do you understand that it was the people interviewed – not local government agencies – who had the death certificates checked by the Lancet interviewers?

    Accordingly, if this sample is truly representative, then lots of other people in Iraq should also have the death certificates of their casualties. Where are they? (In other words, fill a couple soccer stadiums with people waving death certificates, and the objections to the study disappear into mist.)

    You’re saying that if most Iraqi households in which a household member had died had possession of death certificates, and if causalities had been high, we could rationally expect Iraqis to spontaneously fill soccer stadiums in order to wave the death certificates in the air? And the fact that they haven’t done so disproves the Lancet survey?

    No, that’s obviously not what you’re saying. But what are you saying? Please tell me, because you’re not making sense. What, specifically and realistically, do you expect to see happen if casualties are high, and if most Iraqi households in which a household member had died have possession of death certificates; that is not happening?

    If I wanted to find out how many Iraqi households had had a member die, and could prove it because they possess a death certificate, here’s what I’d do: I’d randomly select one or two thousand Iraqi households to interview. I’d have trained Iraqi interviewers go knocking on doors and asking if any resident of this address had died since January 2002; and then after they had told me, I’d ask them to show me the death certificate. Then I’d compile the statistics, and report the answer.

    But it’s obvious that you don’t consider this method a reasonable way of finding out information. So what alternative method do you suggest, that you think is more reliable?

    Comment by Ampersand — October 15, 2006 @ 4:14 am | Reply

  14. I don’t “expect” to see anything happening. I suggested this test as a logical exercise; I don’t know what the answer is. I think you’re overestimating my hostiity to the study; it could be on the right lines; I don’t know. If I knew how many Iraqis had death certificates, then I would have more information about the study’s validity.

    I have some problems with the methodology which I’ll detail later in a post, but those problems don’t arise to the level of me saying presumptively that the study is rubbish. They’re just ways that it could be wrong.

    Comment by Robert — October 15, 2006 @ 4:43 am | Reply

  15. I suggested this test as a logical exercise; I don’t know what the answer is.

    What test? I don’t intend this as hostile, but I honestly don’t understand what the test you’re proposing is, Bob. Please spell it out for me.

    Comment by Ampersand — October 15, 2006 @ 6:45 am | Reply

  16. There seems to be a disconnect between the inferred number of death certificates held by family members, and the number of casualties recognised by the central government. A survey of local registry offices would perhaps throw some light on where that disconnect lies.

    Comment by Daran — October 15, 2006 @ 7:10 am | Reply

  17. Expecting that the government will count all deaths accurately, is expecting that the government will be more efficient and accurate now than it was before the war began. From the companion paper:

    Even with the death certificate system, only about one-third of deaths were captured by the government’s surveillance system in the years before the current war, according to informed sources in Iraq. At a death rate of 5/1,000/year, in a population of 24 million, the government should have reported 120,000 deaths annually. In 2002, the government documented less than 40,000 from all sources. The ministry’s numbers are not likely to be more complete or accurate today.

    Les Roberts, one of the study’s authors, made a similar point in an interview:

    We have gone and looked at every recent war we can find, and only in Bosnia did all governmental statistics add up to even one-fifth of the true death toll. And in Bosnia, the rate was 30 or 40 percent, with huge support for surveillance activities from the UN. So it’s normal in times of war that communications systems break down, systems for registering events break down.

    Les Roberts’ claims seem plausible to me. Is there any reason to think that government officials, during a war, are going to be keeping accurate mortality statistics?

    The question is whether or not “passive” methods – that is, depending on the press or the government to have accurately collected death counts, and then totaling up what those other agencies have collected – can reasonably be expected to be as accurate as active methods – that is, a representative random survey. I don’t know of any theory under which suggests that passive data collection is more accurate than active data collection (unless you can demonstrate that the active data collection’s methodology is horribly flawed).

    Comment by Ampersand — October 15, 2006 @ 9:56 am | Reply

  18. Expecting that the government will count all deaths accurately, is expecting that the government will be more efficient and accurate now than it was before the war began. From the companion paper:

    Assuming that you’re replying to my last remark, it certainly isn’t my premiss that “the government will count all deaths accurately”. Quite the contrary, they do not appear to be able even to count the bodies in a single city morgue. I merely remarked that such a survey would allow us to map out the disconnect more precisely.

    Comment by Daran — October 15, 2006 @ 12:02 pm | Reply

  19. My comment was intended as a reply to one of the arguments I think Robert may be making, not as a response to you. Sorry for the mix-up.

    Comment by Ampersand — October 15, 2006 @ 12:31 pm | Reply

  20. What Robert is suggesting, I think, is that we see how many of the violent deaths that were reported in the survey are in the official files. If, say, only 10% of the death certificates in question are on file, then the total number of certificates on file for violent deaths should be on the order of 10% of the number of violent deaths estimated in the Lancet survey. If not, then this suggests that the survey sample was not representative.

    It seems reasonable in theory, though I’m not sure that the number of death certificates available on file is sufficient to guarantee accuracy.

    Comment by Brandon Berg — October 15, 2006 @ 2:41 pm | Reply

  21. The test I intend is broadly along the lines of what Brandon says, although I’d compare apples to apples. Show up in a town, say “everyone who has lost a household member, come over here”, and then ask them for death certificates. If 80% of them say “here you go”, the Lancet study is strengthened. If 10% of them say “here you go” and 90% say “are you nuts, where would I get a death certificate, this is a war zone”, then the Lancet study is weakened; how come all its respondents had access to this paperwork?

    Comment by Robert — October 15, 2006 @ 3:04 pm | Reply

  22. So you’re proposing that someone go and do to a whole town, what the Lancet researchers did to individual households in random clusters, because what? A survey of one whole town is more representative of Iraq than random clusters?

    Comment by Daran — October 15, 2006 @ 3:34 pm | Reply

  23. I think you’re overestimating my hostiity to the study; it could be on the right lines; I don’t know.

    You may be overestimating the degree to which we’re wedded to the study findings. I’m certainly not to any great degree, and I doubt Amp is either. Nothing would delight me more, than to find out that the casualties are much lower than appears to be the case.

    However, this discussion has taken place against a persistant backdrop of trashing of any scientific results which go contrary to the Republican political agenda, not to mention personal attacks on the scientists themselves. It’s hardly surprising then, that we take a skeptical view of right-wing skepticism.

    Comment by Daran — October 15, 2006 @ 3:49 pm | Reply

  24. Robert’s objection is wacky. The Lancet authors were shown the death certificates for 80% of the cases they looked at. Of course, a bigger study will narrow the confidence interval to some extent — but that has no impact on the validity of this study. If it’s as accurate as the sample size allows, that’s all you can ask.

    Comment by Gene Callahan — October 17, 2006 @ 7:07 am | Reply


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Rubric Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: