Promoting Replication Studies
February 14, 2019
Goals & Rationale
The replication crisis has become a major issue in science. This is particularly true in biological fields, as biological growth can be notoriously difficult to standardize. As synthetic biology is a very new discipline, current research will impact its future greatly. It is fundamentally important that the studies produced have reliable and replicable results, for both practical and ethical reasons. We cannot ensure the safety of later experimentation if its basis is invalid, and therefore must guarantee replicability for non-maleficence.
The Center for Open Science started the Reproducibility Project in psychology and cancer biology. This project aims to reproduce important studies in both fields to confirm their results. However, there appears to be little incentive for researchers to participate in these studies, as it does not appear to provide funding.
I propose beginning a reproducibility project for synthetic biology; however, instead of it being done via an external organization, I believe it should be done through a governing agency itself to allow for funding for this type of research.
Design & Requirements
This proposal would require the creation of a government committee within NIH to allocate some of its research funding for synthetic biology exclusively for replication studies. While this could in theory be done by an outside group (such as the Center for Open Science), this would be seen as more legitimate (and would have a clearer source of funding) coming from the government. A financial incentive is definitely necessary, as these studies would be competing with scientists' more flashy/glamourous research topics. Alternatively, there could be a government mandate on replication studies; something like "5% of your government funded research must be on replication studies."
This committee would need to decide whether it would define the experiments to be replicated based on a review of current research or allow researchers to propose their own replication studies. I believe a combination of both would provide guidelines for research but allow enough flexibility for innovation.
Should funding not be available for this committee, an alternative to increasing reproducibility might be requiring preregistration of all projects. The preregistration process produced by the Open Science Foundation is mostly effective in getting researchers to publicly commit to their experimental goals prior to doing the experiments, reducing the amount of "fudging" that can happen in science.
Assumptions & Risks of Success
Should this be implemented, its biggest risk is diverting resources from projects that could advance the field faster. Another major risk is a replication study messing up, and claiming that real (and potentially important) results from an experiment are wrong. However, this would likely be mitigated by the original authors fighting against these results, as they would have high motivation to do so. This proposal could also cause scientists to be less likely to publish highly innovative/experimental data that they aren't fully confident on, as they may be scared of bad press if their results are found to be inaccurate.
I am currently assuming that this is indeed a problem in synthetic biology experiments, and that there exist enough studies with questionable reliability to deserve testing. I'm also assuming that it is feasible to replicate these experiments-COS mentioned that they had difficulties in getting enough information from the original authors about the methods in order to replicate experiments. This would be doubly difficult in synthetic biology as it might require specific strains or genetic manipulations that are difficult or impossible to obtain now.