FAQs

Overview | FAQs | Executive Summaries for AMR Phase 2 Solvers

FAQs for Step 3 of the AMR Diagnostic Challenge Competition

 

• Is Step 3 of the AMR Diagnostic Challenge competition open to the public and/or individuals/teams who submitted previously to either Step 1 or Step 2 of the competition?

Response:  Step 3 of the AMR Diagnostic Challenge competition is only open to the 5 Semi-Finalists from Step 2 of the competition.  The five Step 2 Semi-Finalists can be found on https://dpcpsi.nih.gov/AMRChallenge/FinalistsAnnouncement.

• What does Step 3 of the AMR Diagnostic Challenge competition involve?

Response:  Step 3 of the AMR Diagnostic Challenge competition involves the testing of the Step 2 Semi-Finalists’ in vitro diagnostic prototypes by two independent CLIA-certified laboratories to assess the performance of the assays compared to the performance claims from the Step 2 Semi-Finalists and FDA-approved assays.

• Where can I find additional information on the submission requirements and criteria for this stage of the competition?

Response:  Submission requirements for Step 3 of the AMR Diagnostic Challenge competition can be accessed either in the Federal Register Notice issued on January 29, 2019, or the Announcement in the NIH Guide for Grants and Contracts issued on January 9, 2019.

• Have the two independent CLIA-certified laboratories been selected and will Step 2 Semi-Finalists be provided the names of these laboratories? Have these laboratories signed non-disclosure agreements?

Response:  The Clinical Laboratory Improvement Amendments (CLIA) regulate laboratory testing and require clinical laboratories to be certified by their state as well as the Center for Medicare and Medicaid Services (CMS) before they can accept human samples for diagnostic testing.  The two CLIA-certified laboratories have been selected.  To ensure independent testing of the “submissions,” the laboratories will not be identified to the Step 2 Semi-Finalists or anyone else outside of the government review process.  These laboratories and their personnel have signed non-disclosure agreements (NDAs) that have been reviewed by both Capital Consulting Corporation (CCC), NIH, and BARDA.  Any real or perceived conflicts of interest between these laboratories and the Step 2 Semi-Finalists also have been reviewed to ensure the laboratories’ ability to independently perform the testing.

• Do the two CLIA-certified laboratories have the technical skills and expertise to conduct the performance evaluation of the Step 2 Semi-Finalists’ in vitro diagnostic prototypes? When will the performance evaluation testing be conducted?

Response:  The two CLIA-certified laboratories have highly trained technical staff to perform the testing of the in vitro diagnostic prototypes submitted by the Step 2 Semi-Finalists.  These laboratories were selected based on their previous experience in performing CLIA-validation/verification testing of in vitro medical diagnostics.  Testing of the Step 2 Semi-Finalists’ prototypes will be conducted from January 2020 through April 2020.

• Will Step 2 Semi-Finalists submit their diagnostic prototypes, consumables, and reagents to CCC or to the two CLIA-certified laboratories?

Response:  As indicated in the Federal Register Notice, each Step 2 Semi-Finalist will need to provide two sets of prototype devices, reagents, consumables, etc. along with test procedures/instructions for use so that each CLIA-certified laboratory can independently assess the performance of the prototypes.  This will require that each Step 2 Semi-Finalist send two parcels/crates with these contents (one set for each testing laboratory) to CCC which will then arrange shipment to the two CLIA-certified laboratories.  There will be no direct contact between the Step 2 Semi-Finalists and the two CLIA-certified laboratories to ensure that there is independent testing of the in vitro diagnostic prototypes.

• Will the Step 2 Semi-Finalists provide on-site setup, training, and/or onsite technical assistance to the two independent CLIA-certified laboratories?

Response:  The Step 2 Semi-Finalists will not have any contact with the two CLIA-certified laboratories to ensure an independent assessment of the “submissions.”  Based on the information provided by the Step 2 Semi-Finalists in their Step 2 and Step 3 written submissions, the two CLIA-certified laboratories will setup the instrumentation and assays.   If these two laboratories have any questions about setting up the instrumentation or performing the assay, they will submit the questions to CCC which in turn will contact the appropriate Step 2 Semi-Finalist.  The Step 2 Semi-Finalists will not provide onsite assistance or have any direct contact with these laboratories.

• What is the testing protocol that the two independent CLIA-certified laboratories will use for the performance evaluation of the Step 2 Semi-Finalists in vitro diagnostic prototypes? Will the Step 2 Semi-Finalists be provided with the same protocol so that they can assess the performance of their in vitro diagnostic prototypes?

Response:  A protocol has been developed by NIH, BARDA, CDC, and the two independent CLIA-certified laboratories to evaluate the performance of the in vitro diagnostic prototypes compared to FDA-approved in vitro assays.  The protocol will be provided to the Step 2 Semi-Finalists for their own assessment of their diagnostic prototypes.

• What types of biospecimens will be used in the performance evaluation conducted by the two CLIA-certified laboratories? Will the Step 2 Semi-Finalists need to provide biospecimens to the testing laboratories?

Response:  The two CLIA-certified laboratories are embedded in major medical centers and have access to patient samples including blood, urine, CSF, and other specimens.  In addition, these laboratories will use well-characterized panels of organisms from the CDC  and FDA Antibiotic Resistance Isolate Bank and/or contrived samples using these isolates to assess the performance of the in vitro diagnostic prototypes submitted by the Step 2 Semi-Finalists.

• Will the prototype instruments be returned by the two CLIA-certified Laboratories to the Step 2 Semi-Finalists?  Will the Step 2 Semi-Finalists pay for the return shipment of the prototype instruments?

Response:  Instrumentation will be returned to the appropriate Step 2 Semi-Finalist following relevant decontamination processes.  Return shipment of the instruments will not be charged to the Step 2 Semi-Finalists.

• Is the Step 2 Semi-Finalist responsible for the cost of the performance evaluation by the two CLIA-certified laboratories?

Response:  Step 2 Semi-Finalists will not be responsible for the cost of the performance evaluation.

• What are the criteria that will be used in evaluating the Step 3 submissions?

Response:  As cited in Federal Register Notice issued on January 29, 2019 and the Announcement in the NIH Guide for Grants and Contracts issued on January 9, 2019, the following four criteria will be used in evaluating the Step 3 submissions:

1. a) “Innovation;
2. b) clinical significance;
3. c) diagnostic performance and feasibility; and
4. d) sample matrix/setting and ease of use/throughput.”

These criteria were further described in the September 8, 2016, Federal Register Notice.  Submitters for Step 3 are encouraged to review the descriptions as indicated below of these criteria prior to submission of their prototypes:

1. “Must be clearly novel and innovative technology representing an advance beyond the current state-of-the-science.
2. Likelihood of improving the use of antibiotics in patients.
3. Diagnostic performance. The performance characteristics (e.g., sensitivity, specificity, positive predictive value, and negative predictive value) of the in vitro diagnostic test using the prototype and likely impact of the performance on utility in combating antibiotic resistance.
4. Sample matrix. The development of an effective in vitro diagnostic test that uses human samples (e.g., blood, urine, sputum, tissue fluid, multiple or other sample specimens).
5. Time to test result. The development of an effective in vitro diagnostic test that rapidly produces results. Specifically, what would be the maximum acceptable time-to-result for an in vitro diagnostic test to be of significant utility (i.e., from the time that a sample is collected from a patient to the time that the result is available to the healthcare provider).
6. Setting and Ease of Use. The settings or venues in which the proposed point-of-need in vitro diagnostic test may be most needed for combating antibiotic resistance. The development of an effective in vitro diagnostic test that is easy to use. Recognizing that diagnostics often require specialized equipment for sample storage, processing and/or analysis, considerations about how such specialized equipment may affect an in vitro diagnostic test’s ease of use or otherwise limit its utility.”

• Are the Step 2 Semi-Finalists allowed to set up the instrumentation in the CLIA-certified laboratories and demonstrate how to perform the assay systems to the CLIA-certified laboratory staff who will conduct the performance assessment?

Response:  Step 2 Semi-Finalists are not allowed to set up the instrumentation or directly instruct the CLIA-certified laboratory staff in how to perform the assay.  In addition to the test procedures/instructions for use, Step 2 Semi-Finalists may provide an instruction manual and/or video along with their prototype devices on how to set up the instrumentation, ensure it is operating properly and perform the assay.  The instruction manual and/or video must be submitted no later than January 3, 2020, at 5:00 pm ET along with the instrumentation and prototype devices.  If the CLIA-laboratories have any questions on how to set-up or perform the assay, they will provide their questions to CCC which will contact the relevant Step 2 Semi-Finalist for a response.

As indicated in the response to FAQ #4, the identity of the two CLIA-certified laboratories will NOT be provided to the Step 2 Semi-Finalists to ensure independent testing of the diagnostic prototypes.  This ensures that there is no direct contact between the submitters and the CLIA-certified laboratories, in order to promote fairness and objectivity of review.

• Do the Step 2 Semi-Finalists need to provide specimens to ensure their prototype diagnostic devices and instrumentation are performing correctly?

Response:  Step 2 Semi-Finalists are requested to provide two sets of 2 positive (antibiotic sensitive microorganisms) and 2 negative (drug-resistant microorganisms) controls along with their prototype diagnostics and instrumentation so that each CLIA-certified laboratory can ensure that the assay is performing correctly.

• When will the CLIA-certified laboratories’ testing protocol, cited in FAQ #8, be provided to the Step 2 Semi-Finalists?

Response:  CCC will provide the testing protocol, developed by NIH, BARDA, CDC, and the CLIA-certified laboratories, to the Step 2 Semi-Finalists by mid-June 2019.

• How will the prototype instrumentation be decontaminated prior to return shipment to the appropriate Step 2 Semi-Finalist?  When will the prototype instrumentation be returned?

Response:  Step 2 Semi-Finalists will need to provide the methodology for decontaminating their prototype instrumentation in the instruction materials that they provide with their prototype instrumentation.  If the laboratories have any questions about this process, they will contact CCC which will directly contact the relevant Step 2 Semi-Finalist for additional information.  The prototype instrumentation will be returned shortly after the performance evaluation testing has been completed and the data submitted to CCC.

• How will the Step 2 Semi-Finalist submissions be evaluated and winner(s) determined?

Response:  The evaluation criteria and process are described in the September 8, 2016 Federal Register Notice, the January 29, 2019 Federal Register Notice, and the January 9, 2019 Announcement in the NIH Guide for Grants and Contracts.  Specifically, “the data provided by each of the Step 2 Semi-Finalists and results from the CLIA-certified laboratories’ performance testing will be evaluated by the Technical Evaluation Panel using the following four criteria:

1. a) Innovation;
2. b) clinical significance;
3. c) diagnostic performance and feasibility; and
4. d) sample matrix/setting and ease of use/throughput.”

The outcome of their evaluation will be submitted to NIH and BARDA scientific staff to review the highly rated solutions for alignment with the National Action Plan for Combating Antibiotic Resistant Bacteria goal for a rapid, point-of-need diagnostic test that has the ability or potential to improve clinical decision making such that antibiotic use and/or outcomes of patients with drug resistant pathogens are fundamentally improved compared to current standard of care and /or reduce transmission of drug pathogens.  The Judging Panel will use the technical and programmatic evaluations to determine the Step 3 winner(s).  Prize(s) will be approved by the Secretary, Department of Health and Human Services.

• When is the Judging period and when will the winner(s) for the AMR Diagnostic Challenge be announced?

Response:  The Judging period is May 2020 to July 2020.  Up to 3 winner(s) will be announced on July 31, 2020.

• May the assay prototypes that were submitted for the Step 2 submission be modified in a Step 3 submission? Will modifications disqualify a Step 3 submission?

Response:  Modifications to the prototypes are not prohibited per se; however, Step 2 Semi-Finalists that are planning to participate in Step 3 of the Challenge are reminded of the following requirements and limitations set forth in the September 8, 2016 Federal Register Notice and the January 29, 2019 Federal Register Notice. For Step 3, each Step 2 Semi-finalist must submit diagnostic tests that are based on the Step 2 submissions.  A description of any changes from the original design (Step 2 solution) must be documented and explained. Submissions must conform to the scope of the Challenge and the requirements for submission. NIH will perform an initial review of all submissions to ensure they are complete and within the scope of the Challenge.  Submissions that are incomplete or outside the scope of the Challenge will be administratively disqualified and will not be evaluated by the CLIA-certified laboratories, the Technical Evaluation Panel, or the Judging Panel.  Therefore, modifications that would render a submission incomplete or outside the scope of the Challenge will disqualify the submission.  For disqualified submissions, the prototype instrumentation and unused cartridges will be returned to the Step 2 Semi-Finalist, but the written narrative, data, and video will not be returned to the Step 2 Semi-Finalist.

Even if a modification does not render a submission incomplete or outside the scope of the Challenge, Step 2 Semi-Finalists are reminded of the criteria upon which winners will be evaluated and selected, summarized below.  A submission for Step 3 that no longer satisfies these criteria will not be selected as a Step 3 winning solution.

Accepted submissions will be tested independently by two CLIA-certified laboratories against standard FDA-cleared and/or CLIA compliant in vitro assays to demonstrate usability, stated time to result, appropriate analytical sensitivity/specificity, as well as confirmation of analytical performance reported in the data submitted by the Step 2 Semi-finalist. The Technical Evaluation Panel will evaluate the Step 3 submissions and test results from the two CLIA-certified laboratories using the following 4 criteria: (a) innovation; (b) clinical significance: (c) diagnostic performance and feasibility; and (d) sample matrix/setting and ease of use/throughput.  See the September 8, 2016 Federal Register Notice for further information about these criteria.  As stated in the January 29, 2019 Federal Register Notice, the NIH and BARDA may determine that based on the number of submissions received for Step 3 that less competitive submissions will not be discussed by the Technical Evaluation Panel during the Panel’s meeting.  NIH and BARDA staff will review highly rated solutions for scientific alignment with the National Action Plan for Combating Antibiotic Resistant Bacteria goal for a rapid, point-of-need diagnostic test that has the ability or potential to improve clinical decision making such that antibiotic use and/or outcomes of patients infected with drug resistant pathogens are fundamentally improved compared to current standard of care and/or reduce transmission of drug resistant pathogens. The Judging Panel will determine which of the diagnostic test solutions are of relevance to BARDA’s and NIH’s missions, and the degree of innovation advancing existing clinical diagnostics, using the technical and programmatic evaluations to determine the Step 3 prize winner(s). The prize(s) will be approved by the Secretary, Department of Health and Human Services.

• Is a Letter of Intent from the Step 2 Semi-Finalist required to submit a prototype assay for Step 3 of this Challenge competition?  If so, what information needs to be included in the Letter of Intent?  When is the Letter of Intent due?

Response:   As cited in the January 29, 2019 Federal Register Notice, a Letter of Intent is required from each Step 2 Semi-Finalist if the company intends to submit for Step 3 of the AMR Diagnostic Challenge. The Letter of Intent is due to Capital Consulting Corporation (CCC) on or before Monday, November 4, 2019.  A Step 2 Semi-Finalist who does not submit a Letter of Intent is not eligible to submit the prototype instrument for Step 3 of the competition.  NIH and CCC will use the Letter of Intent to help prepare the laboratory facilities that will conduct the performance evaluations.  It is important that the Step 2 Semi-Finalist include in the Letter of Intent:

• a list of all instrument/platform components;
• size (height, width, depth) and weight of the prototype instrument(s);
• power, water, and drainage requirements that will be needed for their prototype instrument;
• any special shipping requirements; and
• estimated insurance value of the instrument, reagents, and other commodities that will be submitted for Step 3 of the competition.

The latter is requested to ensure an appropriate amount of liability insurance is arranged in advance by CCC for shipping purposes and by the CLIA laboratories while the prototype instruments are in their facilities.

Please note:  There will be no connection between the prototype instrument and the internet during the evaluation period.  All software needed to perform the assay and/or obtain or transmit test results must be integrated into the instrument or into the computer that is attached by cable to the instrument.

• Who is responsible for the costs associated with shipping the prototype instrumentation, reagents, and related components to CCC, the CLIA laboratories, and return to the Step 2 Semi-Finalists?

 Response:  The Step 2 Semi-Finalist is responsible for any and all costs and insurance for shipping its prototype instrumentation to CCC.  CCC is responsible for any and all costs associated with shipping the prototype instrument to the CLIA laboratories and to the Step 2 Semi-Finalist after the testing is completed.  CCC will insure the prototype instrumentation during shipment to the CLIA laboratories and when it is returned from CCC to the Step 2 Semi-Finalist at the same value that the Step 2 Semi-Finalist used in shipping the instrument to CCC.

• Who is responsible if the prototype instrument is damaged during the shipping and/or during the time that it is being tested and located in the CLIA-certified laboratories?

Response:  The Step 2 Semi-Finalist is responsible for providing to CCC all necessary details pertinent to properly insure the prototype instrumentation. The Step 2 Semi-Finalist is responsible for any damage while the instrument is in transit to CCC and hold insurance that covers that transit.  CCC will be responsible while the instrument is in its possession and during transit to the CLIA laboratories and return to the Step 2 Semi-Finalist.  The CLIA laboratories will be responsible for the instrument while it is in their possession.  CCC and the CLIA laboratories will use the same insurance value that the Step 2 Semi-Finalist used when the instrument was shipped to CCC.

• Since the prototype will be tested by two separate CLIA laboratories, should the Step 2 Semi-Finalist separately send each set of prototype instrumentation, reagents, and related components?

Response: Yes, duplicate sets must be packaged separately.  CCC will not unpackage or repackage any items.

• If the prototype instrument is lost or damaged during the shipping and/or during the time that it is being tested and located in the CLIA certified laboratories, will the Step 2 Semi-Finalist be allowed to replace the damaged or lost instrument so that it may continue to be considered for a prize?

Response:  If a prototype instrument is lost or damaged during the shipping and/or during the time that it is being tested and located in the CLIA certified laboratories, CCC will notify the Semi-Finalist so that a replacement can be shipped within one week’s time to CCC and from CCC to the CLIA certified laboratories.  This would allow the Semi-Finalist’s prototype instrument to continue to be considered for a prize, subject to the deadlines cited in the FRN of September 8, 2016.

• What types of specimens will be tested?

Response: The types of specimens for which each diagnostic prototype can be used are relevant to one of the evaluation criteria for Step 3 of the Challenge—“Sample matrix. The development of an effective in vitro diagnostic test that uses human samples (e.g., blood, urine, sputum, tissue fluid, multiple or other sample specimens).”  In the performance evaluation conducted by the CLIA-certified laboratories, only specimen types consistent with the intended use statement in the test procedure/instructions for use provided by each Semi-finalist will be used.  Each Semi-finalist should list all specimen types acceptable for testing with their prototype system.  Contrived samples will be prepared using clinical samples received in each laboratory that do not contain any pathogens of interest when tested using the CLIA-certified laboratories’ standard methods.  If a diagnostic prototype requires proprietary collection media, Semi-finalists should provide a sufficient number of collection devices and the required collection media (as calculated per test protocol) identified as proprietary with information on any limitations on its use, to CCC by September 1, 2019, so that the CLIA-laboratories can begin to collect and store samples.  All specimens will be collected by health care professionals in accordance with standard laboratory protocols.  Pooled sample matrices will not be used.

• Will samples be tested in duplicate or triplicate?  How will Limit of Detection be determined?

Response: Evaluation of the diagnostic prototypes in the CLIA-certified laboratories is intended to represent performance of the Semi-finalists’ prototype systems in a routine clinical setting.  Therefore, specimens will be tested one time only, unless the test procedure calls for retesting (e.g., invalid or equivocal result).

The evaluation will include Limit of Detection (LoD) testing, consistent with the January 29, 2019 Announcement (“Each solution will be tested…to demonstrate…confirmation of analytical performance (e.g., limit of detection…). LoD testing for purposes of Step 3 evaluation is not intended to calculate LoD for purposes of any FDA approval.  The purpose of LoD testing in this performance evaluation is to demonstrate that clinical test performance is consistent with the Semi-finalist’s claimed LoD, as described in the “detailed description of the proposed in vitro diagnostic and the claims of performance” to be included in the Step 3 submissions.  Specimens will be contrived using QC strains, clinical isolates or CDC and FDA Antibiotic Resistance Isolate Bank samples at approximately 1-4x the LoD stated in the Semi-finalist’s test procedure/instructions for use.  Examples of isolates that may be used are listed in the “General Protocol for Comparison of Phase 3 Semi-Finalists of the AMR Point-of-Care Diagnostic Challenge” that has been provided to the Semi-Finalists and the testing labs.

• Is the nonexclusive license grant to the federal government made legally binding upon receipt and acceptance of an award? (See Federal Register: “To receive an award, Solvers…must grant to the federal government a nonexclusive license to practice their solutions and use the materials that describe them.”)

Yes, and by receiving and accepting the award, you thereby grant the license set forth in the rules.

• Is the scope of the nonexclusive license grant negotiable?  (See Federal Register grant to “any invention made by the Solvers that covers the Submission”)

No.

• Specifically, is it possible to exclude from the license grant any inventions belonging to the Solver that cover the Submission, such as patents that preexisted the competition?

No.