Recently, a feature appeared with  the title ‘How researchers dupe the public with a sneaky practice called “outcome switching”‘ accompanied by a truly scary graphic shown below. The article went on to claim only 9 of 62 clinical trials analyzed had faithfully reported the outcome of the trial.

Yes, there have been abuses, but before we all run out and say 53 out of 62 clinical trials are trying to mislead us, let’s slow down and think about things. Every clinical trial must specify a primary outcome. The FDA very rarely allows a trial to change the primary outcome, and in the few instances when this does occur, it is only after extensive and critical review. If the primary outcome does not achieve statistical significance, the FDA will reject the trial. Period, end of the game.

Trials also include secondary outcomes. Maybe the primary outcome was improvement in overall survival. Secondary outcomes might be improvement in time to disease progression, improvement in time to next treatment, improvement in patient life style, is the treatment more effective in one subset of patients or another, etc. Often these secondary outcome measures are somewhat redundant, and may not all be included in a publication. Or maybe they just did not achieve P<0.05 statistical significance even though the trend in the data supports the hypothesis. Lots of reasons a publication may not present every last one of the secondary analyses listed in the original protocol.

More importantly, it typically is years, often many years, between the time a clinical trial protocol is approved and the time when the trial is complete and the results are ready for publication. During that time a lot can happen. Science advances, new technologies and new drugs come on the scene. The investigators may notice patterns in the data that they did not anticipate when the protocol was submitted. Should they ignore all of this and limit the publication strictly to the outcomes listed in the original protocol? No, that would slow science and be a disservice to the participants of the trial. Are there multiple hypothesis testing issues associated with looking for interesting patterns in data, absolutely. The issue is not were analyses not listed on the original protocol included in a publication but rather how these new analyses are presented.

Bottom line, clinical research is hard, takes a long time and is very expensive. We need to get the most out of every study which means looking at the data in as many ways as possible and being critical in our thinking when we do.