Joshua C. Black

The Gold Standard Paradox in Digital Image Analysis: Manual Versus Automated Scoring as Ground Truth

CONTEXT - Novel therapeutics often target complex cellular mechanisms. Increasingly, quantitative methods like digital tissue image analysis (tIA) are required to evaluate correspondingly complex biomarkers to elucidate subtle phenotypes that can inform treatment decisions with these targeted therapies. These tIA systems need a gold standard, or reference method, to establish analytical validity. Conventional, subjective histopathologic scores assigned by an experienced pathologist are the gold standard in anatomic pathology and are an attractive reference method. The pathologists score can establish the ground truth to assess a tIA solutions analytical performance. The paradox of this validation strategy, however, is that tIA is often used to assist pathologists to score complex biomarkers because it is more objective and reproducible than manual evaluation alone by overcoming known biases in a humans visual evaluation of tissue, and because it can generate endpoints that cannot be generated by a human observer. OBJECTIVE - To discuss common visual and cognitive traps known in traditional pathology-based scoring paradigms that may impact characterization of tIA-assisted scoring accuracy, sensitivity, and specificity. DATA SOURCES - This manuscript reviews the current literature from the past decades available for traditional subjective pathology scoring paradigms and known cognitive and visual traps relevant to these scoring paradigms. CONCLUSIONS - Awareness of the gold standard paradox is necessary when using traditional pathologist scores to analytically validate a tIA tool because image analysis is used specifically to overcome known sources of bias in visual assessment of tissue sections.

Abstract 4582: Evaluating benefits of PD-L1 image analysis for the clinical setting

Tissue-based investigations can prove challenging due to complex tissue architecture and heterogeneous biomarker expression, visual and cognitive “traps” that affect interpretive precision, and subjective assessments that affect reproducibility. A major concern is that these challenges could increase the risk of failure for therapeutic/diagnostic co-development and clinical use, as the biomarker measurements continue to increase in complexity and require increasingly precise diagnostic cut-points. Image analysis tools have been developed to overcome some of the challenges for conventional anatomic pathology practices, capitalizing on the objectivity and computational power of a digital platform. A computer, however, lacks the cognitive ability and experience of a human to interpret tissue architecture and context. Flagship Biosciences’ computational Tissue Analysis (cTA™) platform integrates the power of our tissue Image Analysis (tIA™) technology with the contextual experience of an anatomic pathologist to produce robust, precise, quantitative results that demonstrate biomarker content in the tissue context. Flagship Biosciences envisions the integration of our cTA™ technology into a computer-aided clinical pathology workflow as a method to improve the precision of scoring for even some of the most challenging tissue-based biomarker measurements. In a proof-of-concept study, we evaluated the performance of manual versus digital scoring approaches in a cohort of non-small cell lung carcinoma (NSCLC) samples stained with the IHC protocol for the PD-L1 PharmDx 28-8 complementary diagnostic. A comparison of the 2 modalities demonstrated that in nearly all cases, the within sample standard deviation of the cTA™ digital score results was less than the manual score (median inter-pathologist %CVs were reduced from 124.9% to 7.8% and intra-pathologists from 65.4% to 7.6% for manual and digital scores, respectively). As an additional exploratory examination, the effect of heterogeneity on PD-L1 interpretation was also investigated. Pathologists evaluated the same whole tissue slides within 5 high powered fields (HPFs) using both manual and cTA™ -derived scoring. Results demonstrated that the use of cTA™ provides improves agreement between HPF and whole slide assessments (absolute difference between the manual scores from HPF to whole slide were larger than the absolute differences for the digitally derived scores, at 3.14% and 8.27%, respectively). Taken together, these studies demonstrate that the use of cTA™ can significantly reduce variability in PD-L1 scoring, as compared to a manual scoring approach. Citation Format: Staci Kearney, Joshua Black, Famke Aeffner, Joshua Black, Luke Pratte, Joseph Krueger. Evaluating benefits of PD-L1 image analysis for the clinical setting [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2017; 2017 Apr 1-5; Washington, DC. Philadelphia (PA): AACR; Cancer Res 2017;77(13 Suppl):Abstract nr 4582. doi:10.1158/1538-7445.AM2017-4582

Quantitative assessment of pancreatic cancer precursor lesions in IHC-stained tissue with a tissue image analysis platform

Tissue image analysis (tIA) is emerging as a powerful tool for quantifying biomarker expression and distribution in complex diseases and tissues. Pancreatic ductal adenocarcinoma (PDAC) develops in a highly complex and heterogeneous tissue environment and, generally, has a very poor prognosis. Early detection of PDAC is confounded by limited knowledge of the pre-neoplastic disease stages and limited methods to quantitatively assess disease heterogeneity. We sought to develop a tIA approach to assess the most common PDAC precursor lesions, pancreatic intraepithelial neoplasia (PanIN), in tissues from KrasLSL-G12D/+; Trp53LSL-R172H/+; Pdx-Cre (KPC) mice, a validated model of PDAC development. tIA profiling of training regions of PanIN and tumor microenvironment (TME) cells was utilized to guide identification of PanIN/TME tissue compartment stratification criteria. A custom CellMap algorithm implementing these criteria was applied to whole-slide images of KPC mice pancreata sections to quantify p53 and Ki-67 biomarker staining in each tissue compartment as a proof-of-concept for the algorithm platform. The algorithm robustly identified a higher percentage of p53-positive cells in PanIN lesions relative to the TME, whereas no difference was observed for Ki-67. Ki-67 expression was also quantified in a human pancreatic tissue sample available to demonstrate the translatability of the CellMap algorithm to human samples. Together, our data demonstrated the utility of CellMap to enable objective and quantitative assessments, across entire tissue sections, of PDAC precursor lesions in preclinical and clinical models of this disease to support efforts leading to novel insights into disease progression, diagnostic markers, and potential therapeutic targets.

Abstract 763: Analytical validation of Ki67/CD8 duplex IHC assay using computational tissue analysis (cTATM)

Chromogenic multiplex immunohistochemistry (IHC) assays enable investigation of the spatial relationships between tumor and immune cells, which is thought to be important for understanding and predicting therapeutic response. Development and analytical validation of multiplex IHC assays enables the use of such assays to simultaneously investigate multiple biomarkers as predictors of clinical response. In this study, we analytically validated a chromogenic duplex IHC assay that quantifies Ki67 and CD8 in formalin-fixed, paraffin-embedded non-small cell lung cancer tissues. Five performance criteria were selected and evaluated based on Clinical Laboratory Standards Institute guidelines: reportable range, analytical sensitivity, analytical specificity, accuracy, and precision. Similar to analytical validation studies for monoplex IHC assays, this study utilized a reference method and multiple days of staining. The percentage of cells positive for Ki67 nuclear staining and/or CD8 membrane staining were quantified using our computational Tissue Analysis (cTATM) platform. Performance of the Ki67/CD8 chromogenic duplex IHC assay was considered acceptable for the five criteria evaluated. Once the performance of the assay was established, additional exploratory cTA-based endpoints were examined, including the quantification of each biomarker in the tumor compartment and the tumor microenvironment, and analysis of the spatial arrangement of immune cells relative to tumor cells. In conclusion, Flagship’s cTA platform allows for more consistent quantification of individual analytes on dual-stained tissue sections, enabling investigation of complex biological questions that cannot be achieved with traditional tissue-based manual endpoints. Citation Format: Staci J. Kearney, Joshua C. Black, Benjamin J. Landis, Sally Koegler, Brooke Hirsch, Roberto Gianani. Analytical validation of Ki67/CD8 duplex IHC assay using computational tissue analysis (cTATM) [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2017; 2017 Apr 1-5; Washington, DC. Philadelphia (PA): AACR; Cancer Res 2017;77(13 Suppl):Abstract nr 763. doi:10.1158/1538-7445.AM2017-763

Abstract 4042: Quantifying immune oncology markers across multiple tissue sections with digital image analysis

Immuno-oncology (IO) approaches requires an understanding of the types of immune cells present in the tumor microenvironment (TME) as well as the precise localization of these immune cells. However, the acquisition of spatial IO information is technically challenging, due to the requirement for multiplex labeling of immune cells and the need to categorize their location and biomarker content simultaneously. Additionally, the multiplex biomarker panel must be engineered in advance based on a priori assumptions about the correct marker combinations and their location (such as the tumor epithelial nests, TME, or specifically the tumor-stroma interface). This limits the ability to implement novel, scientifically driven assessments into an existing clinical trial which already has defined immunohistochemistry endpoints. To meet the needs of IO clinical trials, Flagship Biosciences has developed a proprietary image analysis platform, FACTS (Feature Analysis on Complementary Tissue Sections), to deduce both the necessary multiplex staining information and the spatial context of IO markers based on the utilization of existing monochrome or multiplex stained slides from a clinical trial. In this study, we demonstrate the utility of this approach to deliver biologically relevant endpoints important for IO clinical trials. First, we performed single IO biomarker staining for CD4, CD8 and FoxP3 in colorectal cancer patient samples and developed a novel image analysis approach that allowed for the accurate quantification of multiple IO markers within specific margins of the tumor microenvironment across multiple tissue sections. For each biomarker the positive cell populations were binned based on distance from the tumor-TME boundary (i.e.% positive cells within 100μM, 200μM or 500μM of the tumor-TME boundary). Next, we used computational alignment for evaluating the co-localization of multiple immune cell biomarkers from serial sections with single stains for CD4, CD8 or FoxP3. We demonstrate that the information extracted from multiple single stained serial sections can be used to generate co-localization information for multiple IO markers in a given patient sample. Lastly, we developed multiplex chromogenic assays for these same markers, analyzed the multiplex-stained slides with image analysis, and derived the same analysis endpoints for the multiplex slides to the digitally aligned individually labeled sections. The study found that similar interpretation of the inflammatory landscape was possible with multiplex-stained slides and digital alignment of monoplex-stained slides. In summary, this study demonstrated a novel image analysis approach that allows for quantification of IO markers utilizing clinically relevant wet assay technologies and existing IHC stained slides derived from an immune-oncology clinical trial. Citation Format: Ciara A. Martin, Joshua Black, Nathan T. Martin, Logan Cerkovnik, Jasmeet Bajwa, Kristin Wilson, Daniel Rudmann, Carsten Schnatwinkel, AJ Milici. Quantifying immune oncology markers across multiple tissue sections with digital image analysis. [abstract]. In: Proceedings of the 107th Annual Meeting of the American Association for Cancer Research; 2016 Apr 16-20; New Orleans, LA. Philadelphia (PA): AACR; Cancer Res 2016;76(14 Suppl):Abstract nr 4042.

Abstract 4162: Identifying T lymphocytes in IHC-stained tissues independently of CD3+ staining using morphometric features extracted by image analysis

Assessments of leukocyte populations in the context of cancer tissues are typically determined by staining for leukocyte subtype markers in formalin fixed tissues. This requires the identification, categorization, and localization of multiple leukocytes within tissue context utilizing multiple markers. Meeting these demands are challenging due to technical constraints on the number of individual markers which can be visualized or scored in a single slide, and the complexity of staining observed. The use of multispectral imaging with fluorescent multiplexed markers has better enabled the assessment of multiple markers in a single tissue section. However, these wet chemistry and image capture technologies are very complex, and can only be successfully implemented with significant laboratory infrastructure, specialty equipment, and experienced resources. For these reasons, this approach is not widely implementable in the clinical pathology laboratory setting, where such tests are oriented to a companion diagnostic utility. Approaches which rely on widely adopted chromogenic immunohistochemistry (IHC) staining are preferred in the clinical laboratory setting, but the number of assayable markers is limited to 1-3 unique markers in a single tissue. In order to create a useful approach that could be implemented in existing clinical laboratory workflow, Flagship Biosciences has developed an approach for deriving the complex endpoints often necessary in immuno-oncology studies which rely on 1-3 chromagenic stains and computer interpretation of the tissue using only hematoxylin counterstain to identify T-lymphocytes. In a proof-of-concept study, we utilized our Tissue Image Analysis (TIA) tools to identify morphometric parameters which could identify T-lymphocytes, independent of staining for the T-lymphocyte marker CD3. A cohort of non-small cell lung cancer (NSCLC) tissues was stained by CD3 IHC, and both CD3 and isotype-stained tissues were analyzed with Flagship9s CellMap™ software to capture the morphometric and staining features of cells in the tissues. The morphometric features which characterized CD3+ cells were used to approximate the T-lymphocyte population frequency in the isotype-stained tissues. This T-lymphocyte classification scheme was defined based on hematoxylin staining alone, and accuracy of T-lymphocyte classification was verified by CD3 staining. Based on this study, the method described herein could be utilized to reasonably estimate the frequency of T-lymphocyte subsets (e.g. CD4+, CD8+, etc.) or different marker-positive leukocyte (e.g. macrophages) subsets relative to the total T-lymphocyte population without an additional T-lymphocyte marker such as CD3. The approach could, therefore, provide an added dimension of analysis for tissues stained by IHC without adding complexity to the wet assay by necessitating a marker for T-lymphocytes. Citation Format: Nathan T. Martin, Joshua Black, Famke Aeffner, Logan Cerkovnik, Jasmeet Bajwa, Staci J. Kearney, Crystal Pulliam, A.J. Milici, Joseph Krueger. Identifying T lymphocytes in IHC-stained tissues independently of CD3+ staining using morphometric features extracted by image analysis. [abstract]. In: Proceedings of the 107th Annual Meeting of the American Association for Cancer Research; 2016 Apr 16-20; New Orleans, LA. Philadelphia (PA): AACR; Cancer Res 2016;76(14 Suppl):Abstract nr 4162.