Creative Destruction

September 19, 2006

Poor Methodology In Anti-Divorce Study

Filed under: Statistical Method — Ampersand @ 3:38 pm

Last year, on CNN’s “Anderson Cooper 360” show ((“Anderson Cooper 360,” November 22 2005.)), Elizabeth Marquardt, author of Between Two Worlds – which is being re-released in a trade paperback edition this month – had this exchange with Cooper:

Anderson Cooper: Elizabeth, one of the questions that you asked the participants is whether or not they would describe their family as stressful. Fifty-one percent of children of — of what might be termed good divorces agreed with that statement, as compared to only 35 percent of those in unhappy marriages. What do you make of that?

Elizabeth Marquardt: That’s right. One of the striking findings to come out of our study was that children of good divorces often fare worse than those from unhappy marriages, so long as the marriage is low-conflict. And most marriages that end in divorce now are low-conflict.

This is fairly typical of how the Between Two Worlds study was reported – compared to the control group (young adults raised in intact marriages), young adults whose parents divorced fare worse. Elizabeth relies on this comparison often; in the Anderson Cooper interview, she criticizes The Good Divorce author Constance Abram’s research for having no control group of nondivorced parents.

Cover art for Between Two WorldsHere’s another example, from a New York Times article about the Between Two Worlds study: ((“Poll Says Even Quiet Divorces Affect Children’s Paths,” New York Times, November 5 2005, p. A13.))

The new survey, based on the first nationally representative sample of young adults, highlights the many ways that divorce shapes the emotional tenor of childhood.

For example, those who grew up in divorced families were far more likely than those with married parents to say that they felt like a different person with each parent, that they sometimes felt like outsiders in their own home and that they had been alone a lot as a child.

Those with married parents, however, were far more likely to say that children were at the center of their family and that they generally felt emotionally safe.

Unfortunately, the methodology of the Between Two Worlds study is fatally flawed, and the research cannot support any of these comparisons. Why? Because respondents were asked different questions depending on if their parents had divorced or not.

For instance, take the finding, reported in the Times, that adults raised by married parents were far more likely to sat “that they generally felt emotionally safe” when they were growing up. This finding is also reported in Between Two Worlds (page 59):

In our national survey, most young adults from intact families strongly agree that when they were grouping up, “I generally felt emotionally safe.” But fewer than half of those from divorced families say the same thing. Young people who grew up in “bad divorces” are less likely to say they felt emotionally safe, but I was surprised to find that even those of us from “good divorces” felt significantly less safe than our peers from intact families with unhappy but low-conflict marriages.”

This finding is based on asking young adults whose parents stayed married to agree or disagree with the statement “I generally felt emotionally safe.” In contrast, those whose parents divorced were instead asked “After my parents’ divorce, I generally felt emotionally safe.” ((Source: Appendix “B” of Between Two Worlds, page 117, available online as a pdf file.))

So the respondents in the control group were asked about their childhood in general. In contrast, the respondents from divorced families were asked to focus specifically on a major family trauma. Given the biased questioning, it would have been a miracle if Elizabeth didn’t find major differences between the two groups. But it doesn’t tell us anything about outcomes, or about long-range trauma. The study design cannot distinguish between those who were unhappy for a while in the post-divorce period, but on the whole recovered; and those who were left with long-term, ongoing trauma due to their parents’ divorce.

In the Times article, Robert Emory (of the University of Virginia’s “Center for Children, Families and the Law”) said ”The key is to separate pain from pathology.” This is the distinction that the Between Two Worlds study failed to make.

To see why this matters, imagine doing a similar study, this time dividing respondents into those who had pets who died, and those who didn’t. If you asked the “no pet” group if they felt emotionally safe as kids, but asked the “pet died” group if they felt emotionally safe after their pet’s death, probably many more of the latter group would say they didn’t feel emotionally safe. But would that tell us anything about the long-range outcomes for these two groups? Would a direct comparison of the two groups, as if they had been asked the same question, be appropriate? Of course not. A question about a particular, traumatic period in childhood cannot be used to characterize a respondent’s upbringing on the whole.

According to Elizabeth Marquardt, the respondents from divorced families were “periodically” reminded to focus on the period after their parents’ divorce throughout the survey. This means that almost none of the comparisons between divorced and non-divorced families in Between Two Worlds have any validity. For instance, the finding that Anderson Cooper focused on, that those from divorced families are more likely to “describe their family as stressful,” seems unremarkable when you keep in mind that the respondents were asked only about stress “after my parents divorce.” What child wouldn’t find that time stressful? But it doesn’t prove that their upbringing was more stressfel as a whole.

Even results from questions that didn’t include a “after my parents’ divorce” provision are dubious, in my eyes, because the context of the survey as a whole is changed by constantly asking those with divorced parents – and only those with divorced parents – to focus on a painful memory. The poor methodology taints the entire survey.

One finding from Between Two Worlds that Elizabeth emphasized is that “grown children from unhappy, low-conflict marriages generally fare better than those from ‘good’ divorces.” I doubt this finding would have been as strong — or existed at all — if this survey had asked the control group the same questions. Even the current, flawed results show some cases in which those raised by “good divorce” parents appear to be better off than those who parents stayed together. For instance, contrary to the New York Times’ reporting, those raised in “good divorce” families were slightly less likely to feel “like an outsider in my home,” than those whose parents stuck with unhappy marriages.

How much “extra” unhappiness, doubt and misery did Between Two Worlds measure by asking those with divorced parents, over and over, to think of “after my parents divorce?” There’s no way of knowing, but it could have been a large effect. Even a small effect could have seriously changed the finding that “grown children from unhappy, low-conflict marriages generally fare better than those from ‘good’ divorces,” because in many cases the statistical difference between those two groups was fairly small.

* * *

When I asked her about the difference in questions on her blog, Elizabeth responded:

Since we were studying post-divorce childhood, we asked those from divorced families to answer the question in regard to “after your parents’ divorce” and reminded them of that periodically through the interview. …If you want to study post-divorce childhood you have to ask people about, well, post-divorce childhood.

I thought they were studying the long-term effects of divorce. If it’s true that a person whose parents divorced when they were a child is significantly more likely to recall their upbringing as emotionally unsafe, stressful, etc., then that effect should show up without loading the question by asking subjects to focus specifically on the divorce.

If Elizabeth fears that subjects whose parents divorced when they were older – say, 10 or 15 – would have put too much weight on their pre-divorce time, then she should have asked her questions bounded by period (“When you were between 10 and 15 years old….” and so on). Then all respondents could have been asked the same questions, and the results examined for differences pre- and post- divorce. Failing to ask the control group the same questions is not a small design flaw; it’s a catastrophic error that invalidates all results based on comparisons to the control group. ((It’s notable that this study was put together by an anti-divorce think tank, and has never been subjected to peer review. It’s natural to wonder if the peer review process could have forced the study’s authors, Elizabeth Marquardt and Professor Norval Glenn, to improve their methodology.))

Between Two Worlds proves that many young adults whose parents divorced recall the period “after my parents divorce” as painful and confusing. However, nearly all ((I say “nearly all” because some of the findings — such as the greater likelihood of divorce among children of divorced parents — would not have been effected by the bad study design)) of the study’s conclusions — including the conclusion that children are better off if their “low conflict” unhappy parents stay together, rather amicably divorcing — are based on an invalid study design. In particular, most findings from Between Two Worlds based on comparisons to the control group should be regarded as invalid and unproved.

(Related material: See the STATS critique of Between Two Worlds, including a response by Marquardt and Glenn, here and here.)

3 Comments »

  1. There may also be little in the way of a valid control group in existence.

    Low conflict unhappy marriages would, one would suspect, still be better marriages, on average, than unhappy marriages where either party exercised a unilateral right to divorce that exists in almost every state, even if some marriages that end in divorce are better than others.

    Unhappy but tolerable marriages would be expected to stay together, unhappy and intolerable marriages would be expected to fall apart.

    Surely, there is overlap, where a marriage could have gone either way. But, it is virtually impossible in a post-hoc survey to compare.

    Until you have get a survey group and meaningfully predict happiness in advance, and then see who gets divorced and who doesn’t and how that compares with predictions, it is hard to make a valid comparison.

    Where 51% of one group say they are happy, and 35% of another do, that is a sufficiently subtle difference to capture the tolerable v. intolerable distinction.

    To do a really valid study, one ought to study, for example, families that delayed a divorce in Ireland until it was legal, with comparable families in England, who divorced much earlier. Even then, I’m aware of no studies that show that seperation from bed and board is better than divorce for children.

    Comment by ohwilleke — September 20, 2006 @ 3:39 pm | Reply

  2. This is Elizabeth Marquardt’s reply, posted at Alas a Blog:

    Hi Amp —

    Thanks for all the time you’ve devoted to my work.

    As you already noted, my opinion is that if you want to study people’s childhoods after their parents divorced, you have to ask them questions about, well, after their parents’ divorce. Perhaps this is a problem or even a limitation, as you suggest, but I think you’re hard-pressed to claim that it “invalidates” the whole study.

    And while the children of good divorce *generally* fare worse than those from unhappy, low-conflict marriages, on some selected items they fared about the same or better. But in general (i.e., on more items) they fared worse and that is the only claim I’ve ever made in that regard.

    This study does not claim to examine “outcomes” or “long-term effects.” It studies the quality of childhood itself by asking adults from divorced and intact families to reflect, retrospectively, on their childhoods.

    The problem I have with making a distinction between pain and pathology is that is carries an assumption that pain doesn’t matter so long as it doesn’t result in pathology. I’m interested in both. We have a lot of studies documenting pathology among children of divorce. I’m interested in going deeper and looking at pain too. And I think most parents are interested not just in what might cause pathology in their children but also what might cause pain.

    In response to Rachel S., yes, longitudinal studies are the gold-standard. Our study is cross-sectional. It is the first to take on the “good” divorce, the first to examine the moral and spiritual experiences of children of divorce, and the first national survey in the U.S. of grown children of divorce. It is also a beginning. There is much more work to be done. I sincerely hope that other researchers will also take on these questions, in particular the moral and spiritual development of children of divorce. If they are seeking funding for a longitudinal study I would be happy to write a letter of support.

    Again, many thanks,

    Elizabeth Marquardt

    Comment by Elizabeth Marquardt — September 20, 2006 @ 5:25 pm | Reply

  3. Rather than reproduce my response to Elizabeth here, I’ll just paste in a link to the thread over at “Alas.”

    Comment by Ampersand — September 21, 2006 @ 8:27 am | Reply


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: