On the ethics of evidence in Bio-ethics

As I reported yesterday, on the Leiter report bio-ethicists are debating conflicts of interest policies in the running of the American Journal of Bioethics. The source of the controversy is a mean-spirited article by McCollough (et al) in response to a (rather polemical) Letter of Concern (LOC) co-signed by a number of prominent bio-ethicists and philosophers (including our very own Mark Lance). (Both pieces are here.) Such controversy is an occasion to explore the standards of argument in a neighborly field.

Yesterday, I focused on three dubious elements in McCollough et. al.'s argument. One of these, my concern over the abuse of so-called "statistical significance" prompted a (somewhat technical) exchange with Edouard Machery. While I stand by my criticism on that score, I missed a more important evidential-ethical problem.

McCollough and his colleagues accuse the authors of the LOC that it "fails to meet the fundamental standard of evidence-based reasoning" and this is described as a "a fundamental error" (38). Now, let me grant -- for the sake of argument -- that the evidential errors identified By McCollough in the LOC are correct. McCollough's crucial claim is that "data cited by the authors of the LoC do not support the LoC’s claim that “dexamethasone treatment cannot responsibly be characterized as benign.” Indeed, the preponderance of evidence suggests that the therapy is benign." (40)

Below I argue that the evidence for this claim is extremely limited. In fact, it appears to be based primarily on a single empirical study. Before I describe the limitations of this particular study, it should be clear, I hope, that no evidential-ethical claim should rest on a single study without independent, third party replication.

In particular, let me quote from the discussion in the paper that is at the heart of McCollough's argument: "This study has four significant limitations: 1) It was not a randomized clinical trial; 2) we did not test the children with psychological or neuropsychological tests but used mother administered screening instruments, albeit well standardized; 3) the study response rate of 75.5% compares favorably with the response rates of typical postal questionnaire surveys, largely due to our repeated follow-up contacts with the families, but approximately a quarter of the eligible children with a known address did not participate; and 4) subsample sizes were often small and the corresponding analyses thereby of low statistical power." (313-314).

Now, this list gives a nice window in the messy details of real life medical research, when large-scale, randomized (double-blind) clinical trials are not available (as is often the case in clinical research). In fact, there probably was nothing 'blind' about this study; one of the authors (Dr. Maria New, who is at the center of the controversy) has a prior connection with the patients in the study. There are two further statistical problems (both noted in passing in the paper): the overall population is small and extremely varied in the kinds of treatments that they received (and other social factors).

More important, the authors of the study excluded a considerable number of "ineligible" children, including a few who had died! (611) If one is studying the health impact of a treatment, it is strange to exclude the worst hit from a population. This skews the result considerably, and ought to raise serious concerns about the claims that are built on this study. Limitation 2 and 3 strengthen each other. In particular, what cannot be ruled out at all is that the 24.5% that did not respond are precisely the mothers who are, say, short of time because of health problems with their daughters; either way, it would have been nice to know why the 49 mothers that explicitly declined to respond. Were they upset with Dr. New? (Leaving aside well known problems with self-reporting, rather than neurological and psychological examination.)

Finally, the smallness of the subsample sizes can easily hide serious problems (especially because the article does not provide the raw scores). For example, buried in the article is the following remark: "for the 20 children with such data for the KIDS, none of the partial correlations reached 0.05 significance (although all five subscales and the full scale uniformly showed an association of longer DEX duration with lower developmental scores)." (613) This suggests that problems were found. But because of the limitations of this particular study, it would be very hard to draw any conclusions one way or another. So, rather than claiming vindication, a more honest assessment would probably entail that we know little with much confidence about the possible harms of this therapy. (One strange fact is that the authors of the study mention that a follow-up study is in progress, but it seems McCollough does not cite its results.)

So, the burden of evidence is not on the critics, but on the defenders of the therapeutic practice. McCollough's argument reads like a lawyerly brief, rather than a dispassionate examination of the empirical evidence. I hope this practice is not the norm in bio-ethics. Either way, I find it strange that McCollough charges his critics with "unethical" behavior, while he engages in overstating the positive evidence for his side of the argument.