A Model Study of In Silico Proficiency Testing for Clinical Next-Generation Sequencing

The CAP reference genome (unpublished data, April 26, 2016) was sequenced using the Illumina TruSeq Amplicon Cancer Panel (Illumina, Inc, San Diego, California) and the Ion Torrent AmpliSeq Cancer Hotspot Panel v2 (ThermoFisher Scientific Inc, Waltham, Massachusetts) on an Illumina MiSeq sequencer and Ion Torrent PGM, respectively. Ion Torrent sequencing used the 316 chip; MiSeq sequencing used 2 × 150-bp reads. The average depth of sequencing was 10 072× and 763× on the Ion Torrent and MiSeq, respectively.

A custom locus walker (MutationMaker v0.3) written in the Java programming language using the Genome Analysis Toolkit (GATK v1.6)^8,9  was used to insert SNVs and small indels into the sequence files as outlined in Figure 1. Briefly, reads are first mapped to the hg19 reference¹⁰  in a quality-weighted manner using BWA-mem (Illumina) or T-map (Ion Torrent); for this alignment hard clipping is disabled, and adapter sequences are retained and soft clipped. The resulting Binary Alignment/Map (BAM) files are sorted and indexed using Picard tools,¹¹  and specific point mutations are then introduced into the BAM files via a BED file containing user-supplied mutations (specified by chromosome, position, nonreference DNA sequence, and target VAF as input).

MutationMaker uses a locus walker with a read-backed pileup to iterate over all target genomic locations and mutate the desired proportion of reads at that position, while preserving the general error structure and sequence base qualities present in the original data file. For a particular position, overlapping reads are chosen at random for mutagenesis, and the desired base(s) are added or deleted; quality scores for the inserted bases are simulated based on the quality of adjacent bases and bases that were removed. For Ion Torrent–generated files, flow space data are altered to be consistent with the introduced mutations; for Illumina sequencing data, the VAFs of inserted mutations are based on unique (nonduplicate reads).

The MutationMaker program outputs randomly ordered, mutagenized FASTQ and/or unaligned BAM file(s) that, with the exception of the inserted mutations, are indistinguishable from the original input files. Data generated by MutationMaker are then remapped, and variants called using the Torrent Suite or the Genome Analysis Toolkit to ensure that the added mutations are detected at the indicated VAFs. Further, the error logs and output files from the read mapping and variant calling steps are checked to ensure there is no evidence that the files have been altered.

Two ISPT challenges were designed. In the first (Table 1), 26 variants were introduced, including 24 SNVs (VAFs 10%–50%) and 2 deletions (2 and 15 bp); this challenge was distributed to 3 laboratories using the Ion Torrent sequencer and 2 laboratories using the MiSeq. In the second challenge (Table 2), a total of 26 variants were introduced, including 22 SNVs (VAFs 10%–50%), 1 deletion (of length 18 bp), and 3 dinucleotide substitutions; this challenge was distributed to 4 laboratories using the Ion Torrent and 1 laboratory using the MiSeq. All mutations introduced into the sequence files were modeled from actual somatic mutations reported in the COSMIC database, and the same mutations were introduced into the sequence files from both vendor platforms. The resulting sequence files were distributed to participating laboratories electronically as either paired FASTQ files (Illumina platform) or unaligned BAM files (Ion Torrent platform).

Participating laboratories (all of which are performing clinical NGS of oncology specimens in CAP-accredited, Clinical Laboratory Improvement Amendments of 1988–licensed laboratories) were blinded as to the number, type, location, and VAF of the inserted mutations. The laboratories downloaded the simulated files from the central portal, applied their validated bioinformatics pipeline to align the data and call variants, and reported their results via a standardized form (variants [in g. syntax] and VAFs were reported).

The first challenge was evaluated by 4 laboratories, including 2 running the Ion Torrent sequencer only, 1 running the MiSeq only, and 1 laboratory that used both technologies; a total of 26 variants were introduced into the data files, including 24 SNVs (VAFs 10%–50%) and 2 deletions (2 and 15 bp). On average, 23.2 of 24 SNVs (range, 21–24) and 1.6 of 2 deletions (range, 1–2) were correctly identified (Figure 2, A). The second version of the challenge included 4 Ion Torrent–based laboratories and 1 MiSeq–based laboratory; in this challenge, a total of 26 variants were inserted into the sequence files, including 22 SNVs (VAFs 10%–50%), 1 deletion (size 18 bp), and 3 dinucleotide substitutions. On average, 20.8 of 22 SNVs were correctly identified (laboratory range, 19–22), 0.6 of 1 deletion (laboratory range, 0–1), and 3 of 3 dinucleotide substitutions (Figure 2, B).

Across both challenges, the most commonly missed SNVs were those with low simulated VAFs (10%–15%), accounting for 2 of 4 missed SNVs in the first challenge and 4 of 6 missed SNVs in the second challenge. Many of the remaining missed SNVs with higher VAFs were noted to occur adjacent to single-nucleotide polymorphisms and triggered single-nucleotide polymorphism filters in some laboratories, resulting in false-negative calls. In the first challenge, the 2-bp deletion (simulated VAF = 50%) was missed by 2 of 5 laboratories, whereas the 15-bp deletion was correctly called by all laboratories; it is unclear from laboratory feedback why the smaller deletion was missed. In the second challenge, the 18-bp deletion (simulated VAF = 40%) was detected by 4 of 5 laboratories.

Although not all laboratories include VAFs in their clinical reports, we sought to determine the level of reported VAF concordance in this data set. As VAF determination is highly subject to platform biases, we analyzed only data generated on the Ion Torrent platform, which had the greatest number of cases available for comparison. In the first challenge, the median absolute difference between the reported versus simulated VAF across all substitutions was 0.69% (range, 0%–19%). Across deletions, the median absolute difference between the reported versus simulated VAF was 2.1% (range, 0.6%–6.7%; Figure 3, A). In the second challenge, the median absolute difference between the reported versus simulated VAF across all substitutions was 1% (range, 0%–20%). Across all dinucleotide substitutions, the median absolute difference between simulated and reported VAFs was 1% (range, 0%–2%), and for the simulated deletion, the median absolute difference in VAFs was 0.5% (range, 0%–2%; Figure 3, B).

The Clinical Laboratory Improvement Amendments of 1988 mandate PT for external quality assessment as part of the laboratory accreditation process,^2,3  although the precise rules and regulations that govern PT continue to evolve. Many PT programs are based on an individual analyte, and are appropriately termed analyte-specific or disease-specific PT programs. The utility of analyte-specific approaches for DNA analysis has been well documented,^12–14  and laboratories that do not perform disease-specific surveys have more errors than laboratories that do.¹⁵  However, given the number of genes that are routinely evaluated in clinical practice by NGS-based approaches, and the range of mutations for which testing is performed by NGS, it is virtually impossible for laboratories to follow an analyte-specific PT approach in routine clinical practice. For this reason, MBPT paradigms have been developed that are centered on the method of analysis rather than the specific analyte being tested.^4,16  MBPT has some distinct advantages over analyte-specific approaches. MBPT makes it possible to provide comparisons among laboratories for dozens (if not hundreds or thousands) of genes by very complex methods such as NGS, and makes it possible to evaluate proficiency in detection of a wide range of variants. In addition, laboratories that participate in the MBPT challenges are not penalized for the inability to detect a sequence variant that lies in a region outside the scope of their validated test, or types of sequence variants that are not validated within their NGS approach. The MBPT approach has been endorsed by CAP, the American College of Medical Genetics and Genomics, and the Centers for Medicare and Medicaid Services.^4,16 

In this context, the fact that there are 3 independent aspects of NGS (sequence platform, wet-bench protocols, and bioinformatics/dry-bench analysis of the sequence reads) complicates surveys designed for PT of NGS assays, whether via an analyte-specific or a methods-based paradigm. The emphasis to date has been on the development of comprehensive PT surveys that evaluate all 3 aspects of an NGS test based on well-characterized genomic DNA samples, and have generally used nucleic acids of 2 types. The first type, synthetic DNA fragments, has particular advantages because it can be designed to incorporate specific sequence variants, at known ratios, at known positions, and in known allelic ratios, to simultaneously evaluate many aspects of not only platform performance, but also library preparation and bioinformatics analysis.¹⁷  The second type is genetically characterized cell lines; because cell lines are an inexhaustible reagent, and because formalin-fixed, paraffin-embedded cell blocks can easily be produced from cell lines, they are a particularly useful source of reference material for PT application in molecular oncology. It is worth noting that both the Genetic Testing Reference Materials Coordination Program¹⁸  of the Centers for Disease Control and Prevention and the National Institute of Standards and Technology¹⁹  have developed several well-characterized cell lines for various variants specific to many genetic conditions, and that several commercial vendors and professional organizations (eg, CAP) incorporate cell lines into the reference and/or PT materials they offer for NGS.

However, a recurring theme in clinical NGS testing is that bioinformatics pipelines are not standardized across laboratories. Some clinical laboratories use software supplied by platform manufacturers (which may or may not have been locally modified to improve performance), others use bioinformatics pipelines licensed from software vendors, and others rely on software packages developed in-house. Further complicating matters is the fact that software packages optimized to detect one class of variants in routine clinical use are not necessarily optimized for clinical laboratory use to detect other classes of variants,^20–24  and that there are differences between optimized pipelines for constitutional versus somatic analysis.^9,25  Traditional analyte-specific and MBPT paradigms for NGS do not comprehensively evaluate bioinformatics pipelines because of the expense and difficulty involved in creating a full spectrum of mutations and range of VAFs in the PT challenge materials. On the other hand, although ISPT comprehensively addresses bioinformatics pipelines,^4,26,27  it is limited to this component of NGS testing and thus is an approach to augment traditional analyte-specific and MBPT programs rather than replace them. In this context it is worth mentioning that the ISPT model presented here has a number of clear synergies with the recently launched PrecisionFDA Web portal,²⁸  including the opportunity to use simulated data files with the tools contained in the Web environment to optimize bioinformatics pipelines.

It is important to note that the results we report from our study of a model ISPT have some limitations. Although the results from participating laboratories indicate that the logistics of the approach are straightforward, all the participating laboratories were affiliated with academic medical centers with considerable experience with NGS and in-house expertise in managing the file-sharing protocols among Web sites and bioinformatics tools that are intrinsic to ISPT. This level of experience and expertise may not be widely shared among all clinical NGS laboratories, which may complicate broad implementation of ISPT. Similarly, although the high accuracy of variant identification and VAF estimation by the academic laboratories in this study is reassuring, participation in ISPT by a broader range of clinical NGS laboratories may uncover quality issues that are not apparent in this feasibility study. Another limitation of our study is that it only addressed laboratories performing an amplification-based “hot-spot” assay using a commercial kit (and only 2 commercial kits were modeled). Many clinical NGS laboratories perform amplification-based tests that were developed internally that target different genes and variants, and many clinical NGS laboratories perform hybrid capture–based tests. Clearly, for ISPT to have wide utility, the paradigm must be applicable to a much broader range of NGS assays, and, to address this, we are pursuing a feasibility assessment of a per-laboratory–customized ISPT.

The mutagenesis method presented in this study does not use simulated read data. Instead, actual sequence files from NGS of a well-characterized specimen are manipulated by the MutationMaker algorithm to introduce relevant sequence variants into the sequence files. Our approach retains the heterogeneity of sequence reads that is intrinsic to data from biologic specimens (eg, distribution of quality scores for individual bases within and between individual sequence reads and local sequence contexts; distribution of depth of coverage across a target region). However, the ISPT approach presents its own technical and logistical issues. First, when platform vendors introduce new sequence file types, reference DNA samples must be resequenced to produce the files for in silico mutagenesis. Second, as clinical NGS laboratories increasingly rely on vendor-supplied or licensed bioinformatics pipelines, they may not have the in-house expertise to manage the file-sharing protocols intrinsic to ISPT challenges and may require technical support to participate in ISPT.

In conclusion, the results of our model system indicate that ISPT is a feasible approach to create sequence files containing mixtures of variants that mimic the complexity of clinical samples, and that these simulated sequence files can be used as a type of MBPT to challenge bioinformatics pipelines of amplification-based NGS of oncology specimens. Our results suggest that ISPT is likely to be useful in a broader range of NGS assay designs, including hybrid capture–based tests as well as amplification-based tests. Our results also suggest that ISPT can be used to create simulated files that can be used in MBPT for a broader range of NGS tests, including tests designed to detect germline as well as somatically acquired variants, mitochondrial as well as nuclear variants, and so forth.

First challenge participating laboratories: Julia A. Bridge, MD, University of Nebraska Medical Center, Omaha; Suzanne Kamel-Reid, PhD, University Health Network, Toronto, Canada; Alexander J. Lazar, MD, PhD, and Keyur P. Patel, MD, PhD, University of Texas MD Anderson Cancer Center, Houston; Iris Schrijver, MD, Stanford University, Stanford, California. Second challenge participating laboratories: Anonymous.