Friday, February 11, 2011

The Whitaker Aftermath - Assessing the Quality of Information

Over the last several days, I have been commenting on the commenters to Robert Whitaker’s commentary,  “Anatomy of an Epidemic.” Much of the discussion - actually all of it - focuses  on the quality of the studies Whitaker relied on to support his controversial thesis that psychiatric disabilities are on the rise because of the widespread use of psychiatric meds, not despite them.

As a general rule, the more startling an assertion, the higher the level of proof we demand. Thus, if I say, “I have my driver’s license,” you are not likely to ask me to pull it out of my wallet and show it to you. (Ironically, three years ago I did not have a driver’s license, so the joke would have been on you.) You get the picture.

Accordingly, any spirited debate you run across concerning the studies Whitaker relies on are not mere academic quibbles. If his sources are faulty, or if he is making conclusions unsupported by the data, then we can easily dismiss Whitaker.

I’ve been reporting on mental health studies since 1999. My approach to looking at information as a journalist is quite a bit different from that of an academic researcher or a clinician looking at the same information. In many ways, my criteria for reporting is far more strict. And in other key ways, far more loose.

Basically, my standard is that we work from the best set of facts we have for the purpose of improving our understanding right now, and I suspect this is where Whitaker is coming from. We don’t sit around waiting for the ultimate gold standard study that is never likely to materialize. An example:

In 2009, I ran a series of reader polls on Knowledge is Necessity that found: 1) Four in five rated their meds as either the single most important tool in managing their wellness or as just as important as their other wellness tools; 2) Only 14 percent reported their meds worked “very well”; 3) Only 14 percent reported they were back to where they wanted to be.

Something clearly is very wrong with this picture. My series of polls indicate we dangerously over-rely on our meds and one result is we don’t get well. (You can read all about it on mcmanweb.)

But this is a reader poll, not a scientific study. A scientific study of this sort would probably cost in the neighborhood of several hundred thousand to get done. Maybe more. I would probably have to spend two years hustling for the same grant money six zillion other researchers are competing for, a year or more working with a team of experts designing and implementing the study and processing the data, and another year or two actually getting the thing published, assuming I was lucky enough to find an editor interested in reading it.

In all likelihood, my submission would be reviewed by an academic “referee” in the pocket of the drug industry who would find a million reasons to reject it.

But assuming my submission were accepted, there would be six to 12 months of revisions before my research saw the light of day. Nine-tenths of the study article would involve me explaining my methodology. The other one-tenth would be divided between the facts we need to know and my interpretation of what the facts mean. A journal editor would have me on a very short leash. My really interesting observations would likely be edited out, to the point that the study yielded little of value to anyone.

So - five or six or more years, several hundred thousand dollars, a team of experts, and endless aggravation for a by-now nearly meaningless study that maybe 30 people would read. If I were lucky, Robert Whitaker would find out about my study, mention it in his next book, make me famous, and draw all the conclusions that I was not allowed to make.

Then enraged academics like Andrew Nierenberg would attack him for relying on a low-quality study that really said nothing of the sort and moreover came from a nobody like me.

See what we’re up against?

***

This is the first in a series of pieces that examines the quality of mental health information, from different types of academic studies to clinical trials to media reports and blogs to informal surveys to personal experience. We will look at the tricks of the trade employed by the drug industry, academic researchers, journalists, and bloggers. We will look at why a so-called gold-standard scientific study may not be worth the paper it is printed on and why an unscientific reader poll may yield quality information.

Stay tuned ...

2 comments:

Tony Previte said...

Thanks for this John, you bring up a valid point about the realities of research, funding and the loopholes, red-tape and bureaucracy.

You've heard my story and know as someone who served in the Navy for 10 years post diagnosis and did it well...only to be medically retired when my diagnosis came to light, I KNOW bureaucracy VERY WELL. Nobody could believe I did the things I did with the diagnosis I had. In fact, I was nearly court-marshaled for it. That's just another of my incredible stories of the past 24 years, it's a doozey and I'll spare details for another day.

What the internet age had brought us, is a voice, and that voice has gotten stronger and gained lot's of traction. It has brought a voice to those of us that have been marginalized by a system that doesn't really know what to do with us.

You see lot's of us are our own best researchers, we've lived it, we know what works, what doesn't work and how best to treat ourselves. Not all of us get there, but more and more do as we all share our information and ask powerful questions.

Who knew after my brief hospitalization in 2006 and I picked up your book that I'd be having a conversation with you nearly five years later. That you would be part of a private Facebook group that contains you, Robert Whitaker, people from the recovery movement, and friends and family that I live with, have served with and went to school with who all know about my history and how it's unfolded.

I've been making these connections for years, in fact it was one of your posts about Recovery Innovations that prompted me to contact them a few years ago about attending their training. Well... the timing finally worked out, and I'm proud to announce that I'll be attending training to become a peer specialist and actually start doing "on the ground work" with fresh patients either just out of the hospital, in a crisis center, conducting training... it's a limitless opportunity. It's been my passion to educate patients for nearly a quarter decade now. I'm finally on that road. Pat yourself on the back John, because of you, someone started to see the path they were looking for!

Cheers!

John McManamy said...

Hey, that's great news, Tony! Say hey to the people at RI for me. :)