Here’s a very interesting overview of the clinical trials funded (55 total) by the National Heart, Lung, and Blood Institute (NHLBI) over the years 1970-2012. Breaking them into pre-2000 and post-2000 shows something dramatic: in the 30 older trials, 57% of them reported a positive, beneficial outcome. Of the 25 newer ones, only 8% came back positive. Comparisons to placebo, trial design, and so on were quite similar between the two sets – so what the heck? Has cardiovascular disease gotten a lot harder (the “low-hanging fruit” effect)? Have clinical investigators, or the people providing them the money, become a lot less competent (let’s hope not)?
Neither one, most likely (or at least not enough to skew the numbers that wildly). No, the authors picked the year 2000 because that’s the year that NHLBI trials starting having to be registered on Clinicaltrials.gov, and thus to declare up front what the goals of the trial were.
Prospective declaration of the primary outcome variable is important because it eliminates the possibility of selecting for reporting an outcome among many different measures included in the study. In order to investigate this issue, we looked at the statistical significance of other variables not declared as the primary outcomes for preregistered studies. Among the 25 preregistered trials published in 2000 or later, 12 reported significant, positive effects for cardiovascular-related variables other than the primary outcome. Importantly, almost half of the trials might have been able to report a positive result if they had not declared a primary outcome in advance. Had the prospective declaration of a primary outcome not have been required, it is possible that the number of positive studies post-2000 would have looked very similar to the pre-2000 period.
That seems to be it. You can look at this from a glass-half-empty perspective (maybe a lot of those earlier “successes” were cherry-picked illusions), or a glass-half-full one (this whole effect is just an artifact, and trials are finding good results just as much as always). I lean a bit more towards the first, myself – not all the way, but enough to make the call without too much difficulty. This is a stark illustration of what it means to call your shots in the clinic. It affects the design of the trial at the beginning, of course, but it also keeps you from kidding yourself at the end.