Nathan S. Blow

A Simple Question of Reproducibility

O of the things that we pride ourselves on at BioTechniques is providing simple methods developments that can be implemented in the lab immediately. Maybe this makes us throwbacks—“old school” so to speak. But it seems that life science research is becoming so complex at the moment that it is now difficult to reproduce major scientific results between labs. At least this is better than the alternative explanation, that fabricating or “massaging” data have become common practices amoung researchers. Either way, the lack of reproducibility is a serious problem at the moment that needs to be addressed by life scientists. The troubling trend of irreproducibility, which has been brewing for some time now, came to a head for many with the publication of a commentary in the journal Nature by Begley and Ellis finding that the results of 47 out of 53 studies could not be replicated. These preclinical studies formed the basis for other research studies and in some instances were the starting points for costly drug studies. Begley and Ellis are not alone in their findings— other reports have surfaced in recent months highlighting the problem of irreproducible studies. Recently, a group called the Global Biological Standards Institute (GBSI) presented a report making a case for biological standards. In interviews with 60 key figures in the life science community, nearly 75% of those interviewed described having to deal with irreproducible data and/or results. The conclusion of the GBSI was that there is a need for more well-defined and consistently used standards, both material (reference reagents and chemicals) and written (optimal practices and methodologies). The Begley and Ellis commentary also brings to light another fundamental problem in life science today: researchers tackling projects beyond their fields of expertise. Preclinical and translational research have become the buzzwords of the moment, leading many investigators to search out ways in which to focus their grants on these types of studies. But what is the impact of this trend? Are these researchers sufficiently trained and equipped to perform the necessary experiments and analyze their findings in a meaningful way when it comes to clinical significance? Or are attempts to include the “right” buzzwords in grant applications in order to obtain funding dictating experimental design? The impact of irreproducible studies is particularly troubling for me as the editor of a methods-oriented journal. The manner in which methods are reported and detailed is critical to reproducing the findings of an article— researchers need to be able to replicate experiments 1, 10, or even 20 years after an article has been published. While it is a great to be able to go to another lab or exchange reagents to verify an experiment or study in the short-term, what happens over time? Researchers need to understand the methods they are using and document the manner in which they are being performed so that future generations can build upon the work that is being done today. In the end, science continues to grow more complex and more interdisciplinary. This is an exciting development, but it creates the need to modify traditional ways of thinking about grants, peer review, publishing, and experimental design. Specific steps, chemicals, and processes need to be documented. At BioTechniques, we will be taking additional steps during the peer review process to ensure that all of the information needed to replicate a method is presented in every report. In addition, we encourage authors to submit articles identifying “challenging” reagents—including the robust characterization of antibodies and or cell lines— as well as other potential roadblocks standing in the way of reproducibility. The creation of new standards in life science research is an important endeavor that requires care and thought. Researchers need to be cautious in designing their experiments and also report full experimental procedures and results. Journals should require complete methods descriptions, even if they appear as online Supplementary Materials, as this will provide a great first step towards reproducibility. It is interesting to note that at this moment of greater irreproducibility in life science, journals continue to minimize the space given to Materials and Methods sections in articles. Reporting the way in which experiments are performed should not be an afterthought. While the goal is simple—eliminating irreproducibility—getting on the right path could prove tough.

Right cell, wrong cell.

Could the problem of cell line contamination soon be coming to an end? Although contaminated cell lines have been reported in the scientific literature since the 1960s, the challenge for the research community has always been figuring out the best way to stem the tide of these misidentified or contaminated cells. What is the best procedure to authenticate the cell lines used in research articles, grants, and cell banks? And when should authentication be required rather than only suggested? Answers to some of these questions appear to be on the horizon: the ATCC has formed a committee tasked with evaluating various authentication approaches, which will be announcing their findings, along with proposed authentication guidelines, at the end of 2011. (For more on this effort and on cell line validation in general, see this month’s Tech News article by Jeffrey Perkel.) With a uniform set of authentication standards in place, a significant hurdle towards implementing ‘best practices’ in cell line usage for research articles and grant applications will have been removed. With this in mind, the editorial staff at BioTechniques will begin requiring all cell lines used in any manuscript submitted to BioTechniques be validated starting in January 2012. Our new requirements for cell line authentication will be based in part on the suggestions given by the ATCC and its committee. Although this move will place additional financial and time burdens on our authors, the process of validating all the cell lines used in every BioTechniques article in the future will benefit not only our readers, who can be more confident in the results and new methods being described, but also our authors, whose research and conclusions will even better withstand possible challenge or critique. Several people have suggested that funding agencies should step up to the plate at this stage and provide additional funding to support grantees in their authentication efforts. While this would represent an additional cost in a time of smaller budgets, the overall amount necessary should not be very high. Providing this money as a component of a researcher’s grant could significantly decrease the numbers of mischaracterized cell line, thereby reducing the financial and time cost to all involved. In the end, decreasing the number of misidentified cell lines will take a concerted effort from various groups. Journal publishers will need to require identification data for the cell lines presented in the articles they decide to publish, cell banks should require some level of validation information prior to banking (as well as perform additional, comprehensive validation studies themselves prior to distributing cell lines), and finally, funding agencies must financially support validation efforts and require grant submission be accompanied by cell line validation data. With BioTechniques’ first step starting in January 2012, we hope that both our authors and readers appreciate the rationale behind making this move now. During November and December, you will see a new Instruction to Authors in the pages of this journal and on our web site explaining the new authentication requirements for cell lines used in BioTechniques publications. If you have any questions or comments—either now or after the announcement of our requirements—please feel free to contact us. And as always, share your thoughts by posting at our Molecular Biology Forums under “To the Editor” (http:// molecularbiology.forums.biotechniques.com) or sending an email directly to the editors (bioeditor@biotechniques.com).

To edit or not: the NgAgo story

New genome-editing approaches always receive widespread attention. But in the case of a novel Argonaute-based technique published last spring, attention has been particularly intense.

Stories in ancient dna

Technology developments are leading to rapid advances in ancient DNA analysis. Nathan Blow talks to researchers peering into the past through ancient DNA.

Pcr's next wave

New applications are driving the need for faster, higher fidelity PCR. Nathan Blow looks at some clever modifications taking PCR to the next level.

Fighting for the future of antibodies

Nathan Blow looks at how efforts to create guidelines and scoring systems could change the way you buy antibodies.

Going with the flow

From bacteria to circulating tumor cells, advances in flow cytometry technology are pushing the boundaries of cell biology research.

BIG LABS AND TINY INSTRUMENTS

At many research institutions, lab space is more valuable than gold. Developers are taking note by designing smaller instruments with enhanced capabilities. Nathan Blow looks inside todays tiny lab.

CRISPR: modifying the life science landscape

From lampreys to human stem cells, the CRISPR/Cas9 system is challenging our notions of what is possible with genome editing. Nathan Blow talks to researchers pushing the boundaries of CRISPR/Cas9 technology to expand our understanding of biology.

Sounds of molecular biology

Engineers and biologists are using sound to mix fluids, move cells, and dispense liquids. Nathan Blow looks at how acoustic science is changing molecular biology and high-throughput screening.