Don't Know What We Don't Know: Cancer Studies Edition

A pretty remarkable story came out yesterday:

During a decade as head of global cancer research at Amgen, C. Glenn Begley identified 53 "landmark" publications -- papers in top journals, from reputable labs -- for his team to reproduce. Begley sought to double-check the findings before trying to build on them for drug development.

Result: 47 of the 53 could not be replicated. He described his findings in a commentary piece published on Wednesday in the journal Nature.

The result was corroborated by another study from Bayer. It’s again another example of not knowing what we don’t know—and, with the article speculating that the scientific bias towards accepting the hypothesis (especially if interesting) rather than rejecting it, one might imagine similar things are true of results in other fields.

The typical hope here is for comparative effectiveness research, but one can take an extreme skeptical view and wonder why the same biases that skew the original work wouldn’t skew the comparative work also.

It’s also unfortunate that the results of which studies were unable to be reproduced wasn’t released. After all, as the article notes, many different pharmaceutical companies are making investment decisions based on this research—and it suggests some of the potential benefits of making sure as much knowledge is free as possible.