Development and Multicenter Assessment of a Reference Panel for Clinical Shotgun Metagenomics for Pathogen Detection

Abstract

\n Background Clinical shotgun metagenomics for pathogen detection has been used in diagnosing infectious diseases. However, its technical assessments have been limited to reference standards by individual labs, single experimental workflow and laboratory.Results Here we reported the design and development of a set of reference reagents and reporting metrics dedicated for clinical metagenomics by the National Institutes for Food and Drug Control (NIFDC) of China, and a joint evaluation study including 17 independent laboratories in cross-workflow and cross-platform settings. Our results showed that the performance of metagenomic assays was significantly impacted by the factors of microbial types, host context, and read depth, thus highlighting the importance to take these factors into consideration when designing reference reagents and benchmarking assays. Through this study, we found false positives to be a common challenge across centers, and considerable site and library effects to limited the assay’s quantitative value. Our multicenter study also provided practical guidance of performance that laboratory-developed shotgun pathogen metagenomics tests should aim to detect microbes at 500 CFU/mL (or copies/mL) in clinically relevant host context (10^5 human cells/mL) within a 24h turn-around time, and with a read depth of 20M reads or lower. This collaboration work provided a unique resource comprising nearly 600 billion reads (>5Tb) for technical evaluation in clinical and regulatory settings.Conclusion We demonstrated that the performance of metagenomic assays was significantly impacted by the microbial type, the host context and read depth, which emphasizes the importance to consider these factors when designing reference reagents and benchmarking studies. Across sites, workflows and platforms, false positive reporting and considerable site/library effects were common challenges to the assay’s accuracy and quantifiability. Our study also suggested practical guidance of performance that laboratory-developed shotgun pathogen metagenomics tests should aim to detect microbes at 500 CFU/mL (or copies/mL) in clinically relevant host context (10^5 human cells/mL) within a 24h turn-around time, and with a read depth of 20M reads or lower. This collaboration work provided a unique resource comprising nearly 600 billion reads (>5Tb) for technical evaluation in clinical and regulatory settings.