Whitaker vs Torrey: Crunching Numbers

This is the fourth in our series of pieces dealing with Fuller Torrey’s response to Robert Whitaker’s 2010 “Anatomy of an Epidemic.” In his review, Dr Torrey asserts that on matters of schizophrenia and antipsychotic drugs, “Whitaker got it mostly wrong.” The same day that Torrey published his review, Whitaker in a blog post issued an angry rebuttal and his own counter-attack.

In our most recent installment, we discussed Whitaker’s heavy reliance on a 2007 publication of a Harrow-Jobe 15-year longitudinal study which found that about 40 percent of schizophrenia patients did well when taken off their antipsychotic medication. What the authors of the study found of greater significance, however, was the fact that the patients who did well were those with a “good prognosis” to begin with.

In his book, Whitaker made the barest passing reference to Harrow’s real findings, neither reporting on the primary results of the study nor how this information can be used to help patients with schizophrenia achieve better outcomes. Rather, Whitaker interpreted the study to mean that “that the drugs worsened long-term outcomes” (p 118).

Had this been a long-term trial of antipsychotics as Whitaker would have us believe, the authors would have conducted an entirely different study. There would have been at least two evenly-matched groups of patients, one on antipsychotic meds and one not on antipsychotic meds, if not at the very beginning of the study then at a key stage further along.

The catch is long-term studies of this type are impossible to conduct. The costs are prohibitive and no review board would dare permit such an enterprise. But the real world affords untold opportunities to observe the natural course of schizophrenia without meds. As Torrey scathingly notes:

[Whitaker] fails to focus any attention on the fact that on any given day in the United States half of all individuals with schizophrenia, or about one million people, are not being treated. This is a huge natural experiment to test his thesis. Many of these individuals are found in public shelters, sleeping under bridges, in jails, and in prisons. If Whitaker had spent more time in these settings observing the outcome of this natural experiment, instead of delivering lectures on his vision of the impending antipsychotic apocalypse, he would have written a very different book.

In this regard, Whitaker’s highly selective use and non-use of information becomes a major issue, akin to writing “Gone with the Wind” without noting that there is a Civil War going on.

So, is there any legitimacy to Whitaker interpreting the Harrow study his own way? Yes. It’s called “secondary analysis,” a fairly common practice amongst researchers and journalists. Basically, one mines other people’s data in search of new - and often startling - insights. And Whitaker certainly had more than enough data to work with.

The 2007 Harrow-Jobe article tells us that 12 of 64 patients with schizophrenia (19 percent) experienced a period of recovery over 15 years. Of these: Eight of 12 (40 percent) were no longer on any meds and two of five were on meds but no longer on antipsychotics.

In contrast, only two of 39 patients (5 percent) on antipsychotics experienced a period of recovery. Moreover, 19 of 23 patients (83 percent) with uniformly poor outcomes after 15 years were on antipsychotics. Tellingly, 64 percent of these patients had psychotic activity at the 15-year point vs only 28 percent not on antipsychotics.

Harrow’s numbers back up Whitaker, right? Wrong. Recall, the Harrow study involved an apples-to-oranges comparison - good prognosis patients vs bad prognosis patients. The good prognosis patients, identified at the beginning of the study, were the ones most likely to get better in the first place and therefore were in a position to go off their meds.

For Whitaker’s secondary analysis to work, he would have had to show that the “good prognosis” patients who stayed on antipsychotics did worse than the good prognosis patients who went off antipsychotics. (It would have been useful to compare results in the bad prognosis group, as well, but this finding wouldn’t have had the same significance, as we don’t have high expectations for this group.)

So - we’re looking for an apples-to-apples comparison. Easy to show, right? We just pull up the relevant number and ...

No number. Whitaker doesn’t cite one. No problem. We’ll find the number in the 2007 Harrow-Jobe study that Whitaker refers to. Wait, this is weird. The number isn’t in the study, either. The authors slice and dice the data in a multiplicity of ways, but the closest they come to what we’re looking for is a finding that 17 percent of the good prognosis patients were on antipsychotics after 4.5 years and 13 percent after 15 years.

No mention of how these particular patients actually fared. Why? The answer is simple (okay, complicated for me). Let’s assume one-third of the patients in the study were good prognosis patients. Let’s make the number 20. If only 13 percent of these patients were on antipsychotics at the 15-year mark, we are looking at a study sample of two, at most three, patients.

A three-patient sample? Okay, let’s be generous and double it. A six-patient sample?

In his blog, Whitaker insists that “in every subgroup of patients (by prognostic type), those off medication had better long-term outcomes (in the aggregate).” In making his claim, Whitaker relies upon this paragraph from Harrow-Jobe:

In addition, global outcome for the group of patients with schizophrenia who were on antipsychotics was compared with that for the off-medication schizophrenia patients with similar prognostic status. Starting with the 4.5-year follow-ups and extending to the 15-year follow-ups the off medication subgroup tended to show better global outcomes at each follow-up.

Had Whitaker actually asked, “how much? how many?” he would have realized the absurdity of his assertion.

Conclusion: Whitaker’s secondary analysis fails. Totally, absolutely, completely. Case closed, right? Not so fast:

When the dust settles, we still have two key pieces of data that simply will not go away: Eight in 12 patients not on meds (plus two of five on meds but no antipsychotics) experiencing periods of recovery vs only two of 39 on antipsychotics.

Still a very small sample size. Still apples-to-oranges. Still other factors in play such as the type of life experience and personal make-up that separates good prognosis from bad prognosis patients.

But when all is said and done, Whitaker is perfectly justified in saying: “Hey, hold on a minute, have a look at these figures.” And we need to be listening.

Likewise, Torrey is perfectly correct in insisting that Whitaker got it wrong. Again, we need to be listening.

More to come ...

Previous Whitaker vs Torrey pieces:

At Last, a Conversation

Collision Course

Digging Deeper