Bringing doctors together

or

A free, private network for verified physicians

Leading voices

Dr. Stefan Wheat

General Emergency Medicine · UW

View

Dr. Ryan Stanton

General Emergency Medicine · ACEP

View

Dr. Jim Dahle

General Emergency Medicine · The White Coat Investor

View

Dr. Mohamed Hagahmed

General Emergency Medicine · University of Pittsburgh

View

Dr. Sorabh Khandelwal

General Emergency Medicine · OSU

View

Dr. Jennifer Wiler

General Emergency Medicine · UCD

View

Dr. Gita Pensa

General Emergency Medicine · Brown University

View

Dr. Heidi Best

General Emergency Medicine · Emergency Physicians of Tidewater

View

Dr. Kimberly Chernoby

General Emergency Medicine · GW

View

Dr. Amal Mattu

General Emergency Medicine · UMSOM

View
  • Discover your community

  • Stay current

  • A profile that represents you

  • Find your people

About RoonPrivacyTerms

    Overview

    Development of a Reference Standard to Assign Bacterial Versus Viral Infection Etiology Using an All-inclusive Methodology for Comparison of Novel Diagnostic Tool Performance.

    Apr 30, 2025DOI: 10.1093/cid/ciae656Journal ArticleCID

    Written by: Coburn AllenJ Kate DeanehanYaniv DotanMatthew A EisenbergAndrew M FineJonathan IsenbergAnn KaneDani KirshnerTodd W LyonsYasmin MaorAmi NeubergerDaniel G OstermayerSharona PazOded ScheuermanShachaf ShiberVictoria A StatlerMichal SteinRenata YakubovShirly YanaiRoy NavonLior KellermanTanya M GottliebEran Eden

    Find these authors on Roon

    DO
    Dr. Daniel Ostermayer
    Clinical Informatics · University of Texas System

    METHODS

    The multi-cohort study population was drawn from 3 prospective studies (NCT04690569, NCT03011515, and NCT03052088; see Supplementary Methods for details regarding studies). This study's inclusion criteria were age over 90 days, suspicion of acute bacterial or viral infection by the attending clinician, fever ≥37.8°C (100°F) or self-reported fever at least once within the last 7 days (ie, not required at presentation), and duration of current symptoms ≤7 days. The major exclusion criterion was immunosuppression. The eligibility criteria are listed in Supplementary Methods.Informed consent was obtained from each subject or their legal guardian. Depending on local requirements, assent from children under 18 years old was obtained. Institutional Review Board or Ethics committee approval was obtained in the participating medical centers.The adjudication process was designed to mitigate potential biases (Table 1). All cases included in the current study were adjudicated by the same pool of 21 international clinicians. Each clinician had at least 7 years of relevant clinical experience and was individually trained by the research team using a set of historical cases that had been adjudicated by other clinicians in previous studies [13]. The goal of the training was to clarify that labeling of the adjudicated infection etiology should be based on interpretation of all the clinical data and represent the adjudicator's conjectured most likely etiologic cause of infection at the time of the patient's blood draw. The adjudicators independently assigned the etiologic classification for each case based on comprehensive clinical, laboratory, radiological, microbiological (including mandatory syndromic testing), and follow-up anonymized patient data exceeding the information typically available in routine care. To clarify the difference between the diagnostic decision-making process based on patient data routinely available to an attending clinician versus the diagnostic decision-making process of the adjudicator, a comparison Table is provided in the Supplementary Methods. There were no pre-defined clinical criteria that adjudicators were required to follow. CRP, PCT, and MMBV data were withheld from the adjudicators.Mitigations to Avoid Potential Risks of Bias in the Adjudication ProcedureAbbreviations: CRP, C-reactive protein; PCT, procalcitonin.In the first round of adjudication (Figure 1), each panel comprised 3 adjudicators who independently reviewed and assigned to each case one of the following 6 etiologic labels: Viral >90% confidence, viral 70%–90% confidence, inconclusive, bacterial (including bacterial-viral coinfection) 70%–90% confidence, bacterial (including bacterial-viral coinfection) >90% confidence, or non-infectious. The confidence level was a subjective measure.Construction of adjudication-based reference standard cohorts. In total, 3048 adjudications were performed in the first round and 222 additional adjudications in the second round; this process enabled assignment of an infection etiology to all cases and creation of an all-inclusive cohort. For details regarding the 2 rounds of adjudication, see Methods.A second round of adjudication was performed to classify the difficult-to-diagnose cases (see definition below). Round 2 was performed with the following additional steps: (1) Etiologic labels assigned by the 3 independent adjudicators in round 1 were maintained, but the confidence levels were removed from the viral and bacterial etiologic labels, for example, both viral 70%–90% and viral >90% confidence became viral. (2) Anonymized data were sent to additional adjudicators (who were blinded to previous adjudication labels) in a stepwise manner, until a majority adjudication was reached. The adjudicators could assign each patient one of the following etiologic labels: viral, bacterial or non-infectious (“inconclusive” was no longer allowed). (3) In the scenario that the patient data was sent to 10 adjudicators without reaching a majority, the most common etiologic label was considered the final adjudication outcome.Based on the adjudication labels assigned, 4 different reference standard cohorts were generated:Microbiologically confirmed cohort, comprising patients for whom 3/3 adjudicators in round 1 assigned the same etiologic label with ≥90% confidence, and there was a concordant microbiological confirmation. These cases were considered easy-to-diagnose.Unanimous cohort, comprising patients for whom 3/3 adjudicators in round one assigned the same etiologic label with ≥90% confidence. These cases were considered easy-to-diagnose.Suspected cohort, comprising all easy-to-diagnose cases, as well as patients for whom 3/3 adjudicators in round one assigned the same etiologic label with ≥70% confidence, or 2/3 adjudicators in round 1 assigned the same etiologic label with ≥90% confidence. These additional cases were considered moderate-to-diagnose.All-inclusive cohort, comprising the easy and moderate-to-diagnose cases, plus the remaining patients who were assigned a reference standard label only in the second adjudication round. Cases that required a second round of adjudication were considered difficult-to-diagnose.Of note, the classification of easy, moderate and difficult to diagnose is with reference to the adjudication process and not with reference to the diagnostic decision-making of an attending clinician in real time.Measurements were described in the original papers [14–16]. For more details, see Supplementary Methods.Descriptive statistics were used to summarize the data. Continuous data were summarized as medians and interquartile ranges (IQR). Categorical data were summarized as numbers and frequency of occurrence.There were 3 statistical frameworks employed to measure the performance of the diagnostic biomarkers, MMBV, and PCT. In each framework the biomarker was compared to the reference standard infection etiology, with bacterial reference standard etiology considered “positive” versus viral/non-infectious reference standard infection etiology as “negative”. Analyses were performed for each cohort and for the sub-cohorts of children and adults.Area under the receiver operating characteristic curve (ROC-AUC) analysis was conducted. MMBV equivocal cases were included in this analysis.Sensitivity was defined as the number of cases that received a bacterial reference standard infection etiology and a bacterial biomarker score, divided by the number of cases that received a bacterial reference standard infection etiology. Specificity was defined as the number of cases that received a viral reference standard infection etiology and a viral biomarker score, divided by the number of cases that received a viral reference standard infection etiology. For MMBV, scores >65 indicated a bacterial infection (or co-infection), and scores <35 indicated a viral infection (or other non-bacterial etiology). Equivocal cases (35 ≤ scores ≤65) were excluded from this analysis and the rate and/or number of cases reported (in Figure 3 legend and Supplementary Figure 1 legend). For PCT, cases with values >0.25 ng/mL were considered bacterial.The likelihood of bacterial infection was established as an increasing function of the biomarker using a “bin analysis.” For this purpose, cases were assigned to bins and within the bins according to their reference standard infection etiology.The 5 MMBV score bins were as reported previously and in alignment with the manufacturer's instructions for use as regards interpretation ≤ score ≤100 (high likelihood of bacterial infection or coinfection); 65 < score <90 (moderate likelihood of bacterial infection or coinfection); 35 ≤ score ≤65 (equivocal); 10 < score <35 (moderate likelihood of viral infection or other non-bacterial etiology); 0 ≤ score ≤10 (high likelihood of viral infection or other non-bacterial etiology). MMBV equivocal cases were included in this analysis.The 4 PCT bins were based on commonly applied thresholds (ng/mL): 0 ≤ PCT ≤ 0.1; 0.1 < PCT ≤ 0.25; 0.25 < PCT ≤ 0.5; PCT > 0.5.AUC of MMBV versus PCT. Patients with equivocal MMBV results were included in this analysis. The all-inclusive cohort included n = 1016 patients (584 adults and 432 children); the suspected cohort included n = 860 patients (485 adults and 375 children); the unanimous cohort included n = 565 patients (315 adults and 250 children); and the microbiologically confirmed cohort included n = 427 patients (226 adults and 201 children). Numbers in parentheses represent 95% confidence intervals. P-values were obtained using the method suggested by Hanley and McNeil [17]. Abbreviations: AUC, area under the curve; MMBV, MeMed BV; PCT, procalcitonin.Sample size calculation is provided in the Supplementary Methods.

    RESULTS

    The multi-cohort study population included 1016 patients recruited in 3 studies (Figure 1; see Methods and Supplementary Methods for details regarding the study participants, eligibility criteria and patient selection). Of them, 50.1% were male (Table 2); the median age was 25 years (interquartile range [IQR] 4–47); 81.1% of the patients were recruited at the emergency department (ED); the main clinical syndromes were upper respiratory tract infection (URTI, 49.7%) and lower respiratory tract infection (LRTI, 28.7%). At least 1 viral pathogen was detected in 65.1% of the patients, and at least 1 bacterial pathogen was detected in 17.7% of the patients. Pathogen detections are listed in Supplementary Table 1. Clinical syndromes definitions are given in Supplementary Table 2. There were fewer hospitalizations, comorbidities and LRTI cases in the suspected, unanimous, and microbiologically confirmed cohorts as compared to the all-inclusive cohort (Table 2). Conversely, there was a higher number of viral and bacterial pathogens detected in the suspected, unanimous and microbiologically confirmed cohorts as compared to the all-inclusive cohort (Table 2).Patient CharacteristicsAbbreviations: COPD, chronic obstructive pulmonary disease; ED, emergency department; IQR, interquartile range; LRTI, lower respiratory tract infection SD, standard deviation; URTI, upper respiratory tract infection.aRace information was available only for US patients (n = 372 of the all-inclusive cohort, n = 327 of the suspected cohort, n = 229 of the unanimous cohort and n = 162 of the microbiologically confirmed cohort). Each patient can only be included in 1 race. “Other” includes patients labeled as having more than 1 race. “Unknown” includes patients recruited outside the United States.bA patient can have more than 1 comorbidity.cThe clinical syndrome relates to the discharge diagnosis. Patients can be included in more than one clinical syndrome. The definitions of clinical syndromes are given in Supplementary Table 2.Out of 1016 adults and children comprising the study population, 565 (55.6%) had 3/3 adjudicators concur with high confidence and comprised the unanimous cohort (Figure 1). Among those, 427 had a concordant microbiological finding and comprised the microbiologically confirmed cohort (42.0% of the study population). An additional 295 cases had 3/3 adjudicators assign the same label with moderate or high confidence or 2/3 adjudicators concur with high confidence, and together with the 565 unanimous cases, comprised the suspected cohort (n = 860; 84.6% of the study population). The remaining 156 cases (15.4%) were inconclusive after the first adjudication round and required additional adjudication rounds to reach a conclusive reference standard infection etiology: 111 cases required one; 36 cases required 2; 5 cases required three; 1 case required 4; 1 case required 6; and 2 cases required 7 additional adjudicators. Details of selected patients that underwent the second round of adjudication are given in Supplementary Table 3.The AUC values in the microbiologically confirmed, unanimous, suspected, and all-inclusive cohorts for MMBV were 0.98 (95% confidence interval 0.98 (.95–1.00), 0.95 (.92–.98), and 0.90 (.87–.93), respectively, and for PCT were 0.69 (.57–.81), 0.77 (.68–.86), 0.74 (.68–.80), and 0.70 (.65–.75), respectively (Figure 2). Similar AUC values were observed for MMBV and PCT when calculated separately for adults and children. A delta in MMBV and PCT AUC values was observed across the different reference standard cohorts.For both MMBV and PCT, higher values correlated with an increased likelihood of bacterial infection across all four cohorts, as well as within the adult and pediatric subgroups analyzed separately (Supplementary Tables 4 and 5). Of note, both of the aforementioned performance comparisons (ie, AUC and bin analyses) were performed with MMBV equivocal cases included.The sensitivity of MMBV was 100.0% (95% CI: 81.0–100.0), 100.0%, (88.8–100.0), 95.9% (89.5–98.7), and 88.9% (82.6–93.1) for the microbiologically confirmed, unanimous, suspected, and all-inclusive cohorts, respectively. In comparison, the sensitivity of PCT for these cohorts was lower at 50.0% (31.4–68.6), 61.0% (45.7–74.4), 60.8% (51.3–69.5), and 55.6% (47.9–63.1), respectively (Figure 3). Similarly, the specificity of MMBV was 92.0% (88.8–94.4), 91.4% (88.5–93.6), 86.9% (84.1–89.2), and 82.5% (79.6–85.0) for the microbiologically confirmed, unanimous, suspected, and all-inclusive cohorts, respectively, while the specificity of PCT was lower, at 78.4% (74.1–82.2), 80.2% (76.5–83.4), 77.3% (74.2–80.1), and 76.1% (73.1–78.8), respectively. Performance superiority was maintained when cases with MMBV equivocal results were also removed from the PCT performance calculations (Supplementary Figure 1; P < .01 for both sensitivity and specificity).Sensitivity and specificity of MMBV versus PCT. For MMBV, manufacturer's cutoffs were applied, ie, cases with score <35 were considered viral and with score >65 were considered bacterial; cases with MMBV equivocal results (35 ≤ score ≤65) were removed from performance calculation. For PCT, cases with values >0.25 ng/mL were considered bacterial. Error bars indicate 95% confidence intervals. The microbiologically confirmed cohort comprised n = 427, the unanimous cohort n = 565, the suspected cohort n = 860, and the all-inclusive cohort n = 1016 cases. For the calculation of MMBV performance, 30 equivocal cases were excluded from the microbiologically confirmed cohort, 41 from the unanimous cohort, 77 from the suspected cohort, and 102 from the all-inclusive cohort. Abbreviations: MMBV, MeMed BV; PCT, procalcitonin.

    DISCUSSION

    This study introduces a new approach to establish a reference standard for bacterial and viral infection etiology. The approach addresses the diagnostic uncertainty faced by clinicians dealing with acute infections and the consequent challenges in creating an adjudication-based all-inclusive reference standard for infection etiology. Our findings show that out of 1016 cases, 565 were easy-to-diagnose, 295 were moderate-to-diagnose, and 156 were difficult-to-diagnose and required additional rounds of adjudication. Four reference standard cohorts were created, with different rates of moderate and difficult-to-diagnose cases, and used to compare the performance of MMBV and PCT. The finding that the performance of MMBV and PCT decreased with increasing diagnostic uncertainty (ie, as the number of moderate and difficult-to-diagnose cases increased), while a difference between 2 tools remained, underscores the importance of using the same reference standard method when comparing diagnostic tools.Although randomized controlled trials can assess clinical utility, they fall short in providing insights into diagnostic accuracy. Diagnostic accuracy studies compare the biomarker to a reference standard. The motivation to develop an all-inclusive reference standard that includes difficult-to-diagnose cases is driven, in part, by regulatory authorities that want to be sure of a tool's diagnostic performance across the entire target population [19]. An important note is that the difficulty to diagnose is categorized with regards to the diagnostic decision-making of the adjudicator and not with regards to the diagnostic decision-making of the attending clinician in real time. A case may be difficult in real time in the ED (eg, as culture results are not available) but is straightforward for the adjudicator. Conversely, a case may appear straightforward in real time in the ED (eg, based on clinical exam but before test results are available) but can be difficult for the adjudicators when the x-ray, PCR, and treatment data from the electronic record are available but do not appear to align. Taking this into account, the value of the tools assessed in this study is not solely for the “difficult-to-diagnose cases, as the difficulty does not account for timing of test results.A strength of this all-inclusive methodology is that adjudicators were allowed to express uncertainty (“I don't know”;) in the first round of adjudication. This emulates real-world diagnosis and highlights cases with diagnostic uncertainty, which are then resolved in the second round of adjudication. Another strength is that adjudicators were blinded to PCT, CRP, and MMBV, avoiding incorporation bias in the reference standard. Additional strengths are that the adjudicators were trained but not required to follow formal guidance and that there was no “super” adjudicator who served as tiebreaker. Finally, the comprehensive collection of patient information that are not routinely available to providers in real time, along with patient follow-up data, provided adjudicators with a more robust basis for reaching a diagnosis.The all-inclusive methodology has limitations. First, it is resource-intensive, requiring 2 adjudication rounds with multiple adjudicators involved in the second round. The collection of patient information beyond routine care and the structured follow-up add to the resource demands. Second, the study population included only immunocompetent patients. Inclusion of immunocompromised patients may increase the proportion of difficult-to-diagnose patients. Third, withholding PCT and CRP from the adjudicators constrained the information available to them. Despite this, a recent study showed no significant change in the performance of MMBV when measured against two reference standards—one where CRP and PCT were withheld from the adjudicators and another where the biomarkers were provided [16].Difficult-to-diagnose cases are highly susceptible to healthcare variation and consequently to sub-optimal patient outcomes [20–24]. The study's findings underscore that even with the benefit of hindsight (ie, follow-up), multiple rounds of adjudication are often required to reach a conclusive diagnosis. A corollary is that a diagnostic test that performs well against a reference standard that includes difficult-to-diagnose cases has potential to improve etiology decision-making in clinical scenarios. We hypothesize, therefore, that adoption of high-performing diagnostic tests in real-time can help reduce healthcare variability and improve patient outcomes. Future studies should focus on validating this hypothesis and exploring practical implementation of such diagnostic tools in routine clinical practice.In summary, this study presents a methodology that provides a comprehensive framework for infection etiology classification with a standardized approach to difficult-to-diagnose cases. This all-inclusive reference standard can be generally applied when evaluating acute infection diagnostics. Finally, this study highlights the importance of employing the identical reference standard when comparing acute infection diagnostics.
    DODr. Daniel Ostermayer

    Dr. Daniel Ostermayer

    Emergency Medicine · Clinical Informatics
    The University of Texas System
    danielo