At the beginning of this year I wrote a post, 12-Step Statistics, detailing some of the issues in the way that 12-step programs present their track records. The fact is that, in this country, “rehab” and other facilities are subject to very little oversight. Any facility providing drug or alcohol treatment requires a state license, but they are not subject to nearly the level of scrutiny that is routine for doctors and hospital healthcare providers. And, as I discussed in the post linked above, their reports of success are often dubious at best and misleading at worst.
Of course, we can’t single out those who run rehab centers above all other professionals. Just this week, a post on LinkedIn raised the question of whether healthcare professionals were submitting deliberately fraudulent papers to PubMed, an online database of medical research. It was discovered that from 2000-2010, 788 papers were removed from PubMed for either errors or fraud, with 243 determined to be fraudulent and 545 simply wrong. These numbers seem very worrying, especially when the linked article uses phrases like “one-third of these were for fraud”. It’s worth keeping in mind that that phrase is referring to one-third of the 788 removed, not one-third of all papers. In fact, between 2000 and 2010, about 5 million papers were submitted to PubMed. 788 is far less than 1% of the total, meaning that the proportion of papers not retracted was about 99.98%.
The framing of statistics is a game, one that many different people and organizations are constantly playing for very different reasons. A 99.98% accuracy rate on PubMed does not mean all doctors are right all of the time, and a dubious success rate does not mean that a rehab center has never had a single success. In my opinion, the difference is in the attitudes of the two communities towards themselves. The scientific community, by opening up all research and data to the researcher’s peers, encourages a climate in which fraud is uncovered and errors corrected as soon as possible, out of competition if nothing else. It isn’t a perfect system, but it places a very high premium on objective reality and applicable, repeatable data. On the other hand, the systems used and made by 12-step rehab centers are not very self-critical; it is not in their interest to be self-critical, because if they made decisions based purely on success rate, the model would not have persisted as long as it has. Statistical framing is a useful way to avoid the need for self-criticism, and also to deflect the criticism of outside sources. And, unlike within the medical community, no particular rehab center is in very much danger of being challenged by its peers. They do not hold each other accountable.
I do not know of any online database rivaling PubMed for 12-step based rehab centers, probably because not much “research” is being done into their largely unscientific and non-standardized methods. But if there were such a database, I wonder how many articles would be retracted due to fraud — whether any would be checked for it, and how the checking would proceed, without an objective standard to measure against. I am not accusing all 12-step rehab staff to be frauds, just pointing out that the philosophy and business model of most 12-step programs builds an environment in which it would be unusually easy for fraud to pass undetected. There will be fraudulent and dishonest practitioners in any population; it then becomes our responsibility to build communities in which they cannot pass undetected in order to inflict harm on others with bad information. I am therefore very pleased to see that PubMed has removed articles in considers harmful, and I hope that in the future 12-step centers will have the courage to rid themselves of their own tenants not fully proven to do more good than harm.