BOARD ON LIFE SCIENCES, DIVISION ON EARTH AND LIFE STUDIES

BOARD ON HEALTH SCIENCES POLICY, INSTITUTE OF MEDICINE

MAY 2015

BioWatch PCR Assays:

Building Confidence, Ensuring Reliability
BIOWATCH IS AN AIR MONITORING
SYSTEM deployed in jurisdictions around the
country with the goal of detecting the presence of
certain high risk pathogenic microorganisms. It relies
on a network of federal and nonfederal collaborative relationships to be successful, and is one part
of a larger array of disease surveillance, intelligence-
gathering, and biomonitoring activities in support of
public safety and health.
The assays used in the BioWatch system to detect
the presence of pathogens in collected samples rely
on the technique of polymerase chain reaction (PCR)
to sensitively and specifically amplify target nucleic
acid sequences. The program and its users need an
understanding of each assays performance characteristics in order to have confidence in the results and
have the ability to appropriately interpret them. This
confidence is partly provided through a performance
standard, which gives information on the minimum
requirements that must be met for the assay to be
considered acceptable for its intended purpose and
describes how testing to validate this performance is
to be carried out.
The National Research Council convened a
committee of experts to examine standards for PCR
assays appropriate for the BioWatch program. The
committees report discusses principles of performance standards, reviews information from several
existing guidance documents and standards that
might be applicable to BioWatch, and discusses assay
testing efforts that have occurred or are ongoing. It
provides recommendations on general principles and
approaches for a performance standard and validation framework to meet BioWatchs mission. It also
provides a starting point for an approach to in silico
and laboratory assay characterization for consideration by BioWatch and its stakeholders, with the aim
of providing a reasonable amount of information on
performance parameters such as an assays limit of

detection, sensitivity,
and specificity. As
tasked, the report
focuses on the PCR
reaction portion of a
PCR assay; although
it discusses key
processes before and
after the PCR, these
are not addressed
indetail.
One fundamental
aspect of assay validation for a system such
as BioWatch should be
recognized and clearly
communicated to program stakeholdersBioWatch is
designed to detect rare but high-consequence events
and all assays will have some associated false positive
and false negative rates. It is not practical in time,
effort, or money to conduct laboratory-based assay
performance testing using large enough numbers of
replicates under enough different conditions to validate
the assays long-term false positive or false negative
rate at the high levels of statistical confidence program
users are likely to desire for operational decision
making (e.g., to determine with statistical confidence
that a false result will not occur more than once per x
thousands or hundreds of thousands of samples).
The assay design and laboratory performance
validation that is the focus of the present report thus
provides initial information that the assay is anticipated to perform reasonably well. It is critical for the
systems users to understand the testing that has been
undertaken and the limitations of the data, and to
combine this knowledge with ongoing analysis of data
obtained from verification in jurisdictional laboratories
and from operational deployment of the assay in order
to achieve confidence in the results and to facilitate

responses to positive BioWatch detections. The

report emphasizes the importance of communication
and, particularly, of discussing the details of performance data with jurisdictional laboratory experts and
officials. The report also emphasizes the importance
of obtaining a better understanding of the microorganism background that is present in BioWatch
jurisdictionsinformation that remains very limited.
Finally, the report considers how developments in
technology, particularly in multiplex PCR and nextgeneration sequencing (NGS), can contribute to the
ability of the BioWatch program to meet current and
future challenges. Sequencing currently can be used
to follow up on unexpected assay results from the
jurisdictions. In the nearer term, targeted approaches

coupled with NGS may be useful as a replacement for

the current real-time PCR secondary assays because
of their ability to analyze many more genomic regions
for identification and characterization. The applicability of techniques such as metagenomic NGS to the
program would be a longer-term prospect. The report
recommends that the Department of Homeland
Security and the BioWatch program continue to
monitor and evaluate technologies as they develop.
The program also should plan to work with laboratory users in the BioWatch jurisdictions, along with
technology experts, to ensure that new technology
brought into the program not only incorporates the
best technology but also functions smoothly for the
stakeholder community.

Locate additional information, including related reports, at http://dels.nas.edu/bls

Read, purchase, or download a free PDF of this report at http://www.nap.edu

Committee on PCR Standards for the BioWatch Program: Georges C. Benjamin (Co-Chair), Executive Director, American
Public Health Association, Washington, DC; Kenneth I. Berns (Co-Chair), Distinguished Professor of Molecular Genetics and
Microbiology Emeritus, University of Florida College of Medicine, Gainesville; Bruce Budowle, Executive Director, Institute of
Applied Genetics, and Professor, Department of Molecular and Medical Genetics, University of North Texas Health Science
Center, Fort Worth; Charles Chiu, Associate Professor, Laboratory Medicine and Medicine, Infectious Diseases; Director,
UCSF-Abbott Viral Diagnostics and Discovery Center at China Basin; and Associate Director, UCSF Clinical Microbiology
Laboratory, University of California, San Francisco; John M. Hardham, Associate Research Fellow and Chair, Emerging
Infectious Disease Program, Zoetis, Inc., Kalamazoo, MI; Grace Kubin, Director, Laboratory Services Section, Texas
Department of State Health Services, Austin; M. Allen Northrup, Principal, Northrup Consulting Group, San Francisco, CA;
Tom Slezak, Program Leader, Global Security Program, E.O. Lawrence Livermore National Laboratory, Livermore, CA; Peter M.
Vallone, Leader, Applied Genetics Group, Biomolecular Measurement Division, National Institute of Standards and Technology,
Gaithersburg, MD; Katherine Bowman (Study Director, Board on Life Sciences), Autumn Downey (Program Officer, Board on
Health Sciences Policy), Kathryn Hughes (Senior Program Officer, Board on Chemical Sciences and Technology), Jon Q. Sanders
(Program Coordinator, Board on the Health of Select Populationsuntil November 2014), Nia D. Johnson (Senior Research
Associate, Board on Army Science and Technologyfrom November 2014), Bethelhem Mekasha (Financial Associate, Board on Life
Sciences), Andrew Pope (Director, Board on Health Sciences Policy), Frances Sharples (Director, Board on Life Sciences), National
Research Council
The National Academies appointed the above committee of experts to address the specific task requested by the Department of
Homeland Security. The members volunteered their time for this activity; their report is peer-reviewed and the final product
signed off by both the committee members and the National Academies. This report brief was prepared by the National
Research Council based on the committees report.
For more information, contact the Board on Life Sciences at (202) 334-2187 or visit http://dels.nas.edu/bls. Copies of BioWatch
PCR Assays: Building Confidence, Ensuring Reliability areavailable from the National Academies Press, 500Fifth Street, NW,
Washington, D.C. 20001; (800) 624-6242; www.nap.edu.
Permission granted to reproduce this document in its entirety with no additions or alterations.
Permission for images/figures must be obtained from their original source.
2015 The National Academy of Sciences