Bringing doctors together

or

A free, private network for verified physicians

Leading voices

Dr. Connor O'Brien

Cardiac Critical Care · UCSF Medical Center

View

Dr. Rosy Thachil, MD, MBA

Cardiac Critical Care · Mount Sinai

View

Dr. Dorairaj Prabhakaran

General Cardiology

View

Dr. Pugazhendhi Vijayaraman

EP · Geisinger

View

Dr. Robert Schaller

EP · HUP

View

Dr. Trey Woods

General Cardiology · Emory Healthcare

View

Dr. Brahmajee Nallamothu

Interventional Cardiology · UMHS

View

Dr. Felix Yang

EP · NYU Langone

View

Dr. Jennifer Rymer

Interventional & Peripheral Cards · Duke University

View

Dr. Ami Bhatt

Adult Congenital Heart Disease · MGH

View
  • Discover your community

  • Stay current

  • A profile that represents you

  • Find your people

About RoonPrivacyTerms

    Overview

    Engineering of Generative Artificial Intelligence and Natural Language Processing Models to Accurately Identify Arrhythmia Recurrence.

    Dec 16, 2024DOI: 10.1161/circep.124.013023Journal ArticleAHA

    Written by: Ruibin FengKelly A BrennanZahra AziziJatin GoyalBrototo DebHui Ju ChangPrasanth GanesanPaul CloptonMaxime PedronSamuel Ruipérez-CampilloYaanik B DesaiHugo De LarochellièreTina BaykanerMarco V PerezMiguel RodrigoAlbert J RogersSanjiv M Narayan

    Find these authors on Roon

    BD
    Dr. Brototo Deb
    Cardiac Critical Care · BWH

    Introduction

    Large language models (LLMs) are generative artificial intelligence (AI) systems trained on massive amounts of textual data, enabling them to understand and generate human-like natural language responses to open-ended user prompts as a chatbot1. LLMs such as GPT2, Turing-NLG3, and PaLM4, provide remarkable performance in answering specialized questions on extensive pubic data, such as classifying tweets of US legislation5 or solving complex mathematics6. However, “out-of-the-box” LLMs struggle to accurately answer questions related to specialized fields, particularly law and medicine where datasets and electronic health records (EHR) are private7–9. This prevents non-specialists, including patients, from using LLMs to understand their medical records or applying digital tools for automated personal diagnostics.One solution would be to train a LLM with specialized data, but this would be limited by privacy regulations10, require access to proprietary source code, and introduce delays in retraining4. Collaborating with experts is another option, but this neither expands data access nor avoids the need for time-consuming manual evaluation. Prompt engineering alters the way that queries are posed11 and may improve chain-of-thought prompting12 for math problems or instruction prompting13 for medical exams, yet a comprehensive benchmarking of prompt engineering for EHR analysis is lacking. Traditional natural language processing (NLP) such as logic-based algorithms14 and pre-trained models such as ClinicalBert15 and SapBERT16 have performed well in some EHR studies 17–19 and even outperformed GPT-320,21 but require expert input to generate rules, and have not been systematically compared against novel LLM22–24. We hypothesized that carefully designed prompt engineering would enhance the accuracy of LLMs for interpreting EHRs, beyond the accuracy of traditional NLP, and without the need for domain knowledge.We tested whether commercial public LLMs could accurately answer questions at the level of medical experts from full-text EHR notes without expert input, using prompt engineering. We developed and systematically tested various prompt engineering strategies on N=490 unstructured EHRs from a cohort of 125 patients to detect recurrences of potentially life-threatening ventricular tachycardia (VT) or fibrillation (VF)25. We compared the impact of varying the questioner role, requiring the LLM to provide a rationale (supporting statements), or provide answers in structured digital formats, and inputting annotated exemplars on GPT-4-turbo performance. Since the accuracy of LLM has rarely been compared to NLP for nuanced tasks, we compared the impact of optimized LLM to a logic-based NLP algorithm and a BERT-based NLP model, and validated our finding with two other LLMs.

    Methods

    The study was approved by the Stanford University Institutional Review Board (IRB 74318) who reviewed this specific use of LLM for this study. Data were de-identified using Philter (“Protected Health Information filter”)26 prior to analysis (see Supplemental Material for details). The de-identified clinical dataset in this study will not be released publicly to maintain tight privacy controls and given the evolving nature of LLM. Code and prompt design have been made publicly available at https://github.com/*** (the link will be released upon acceptance).We focused on the question: Does a patient with a known history of VT have a recurrent episode, distinct from prior events? Since all patients had prior VT, experts resolve such questions using contextual evidence. From the Stanford VT Registry, the EHR data were systematically retrieved using SQL queries by the STAnford medicine Research data Repository (STARR) to ensure precision and reproducibility. The specific search terms used were: (1) ICD code I47.2, (2) >18 years old, and (3) CPT code 93654. We identified N = 125 patients diagnosed with VT who underwent ablation procedures at Stanford Health Care between 2013 and 2022. From these patients we curated N=490 full-text notes from which we used stratified sampling and created a 20% Exemplar Cohort ℰ and an 80% Test Cohort 𝒯 whose demographics were similar (p ≥ 0.05). Note that our Test Cohort (N=389) is larger than several prior EHR NLP studies, such as Yu et al. who used 100 notes 27, Mitra et al. who used 209 notes 28, and Agnikula Kshatriya et al. who used 315 notes 29. Table 1 describes the demographics and baseline characteristics of the patients. There were 4 cases with missing ethnicity data that were excluded from the demographic analysis. Full details of the dataset are provided in Supplemental Material.The ground truth label for all notes was the determination by expert raters for quality control and to ensure that only “post-ablation” notes were included. To evaluate the reliability of annotations among different raters, we randomly selected 84 clinical notes, representing 21.1% of our Test Cohort 𝒯, to be independently reviewed by all four experts. Agreement among the raters was 98%, with a Fleiss’ Kappa value of 0.88, indicating strong agreement30,31. Thus, the remaining notes were each annotated by one reviewer.Figure 1 provides excerpts of notes. Figure 1a indicates a new episode of VT lasting > 30 seconds that was coded as “Yes” by experts. In figure 1b, a new VT event occurred but was considered not relevant by experts due to its short duration of 5 beats, since the clinical threshold for significant VT is 30 seconds25, and was labeled “No”. Full notes for these excerpts are provided in Supplemental Table S1.We only varied queries (prompts) to each LLM, and did not train, fine-tune nor provide any domain rules to improve performance. We primarily evaluated GPT-4-turbo accessed through the OpenAI API for programmatic interaction. We set the temperature to zero, ensuring that the model’s responses were most deterministic and stable. We did not impose a specific limit on the number of tokens generated per response, allowing the model to develop its answers fully based on the complexity of each prompt.Because the phrasing of a question may influence its interpretation by LLMs32, we generated a family of related questions:𝒬1 “Has the patient had sustained ventricular arrhythmias after ablation?”𝒬2 “Has the patient had sustained ventricular tachycardia (VT) or ventricular fibrillation (VF) after ablation?”𝒬3 “Has the patient had sustained ventricular tachycardia (VT) or ventricular fibrillation (VF) after ablation? Consider mentioning any episodes of ventricular tachycardia, wide complex tachycardia, sustained or nonsustained VT, and any device interrogation results, such as delivered therapy (shocks, ATP) or VT and VF detection.” Note that 𝒬3 was generated automatically by using GPT-4 to refine question 𝒬2, inspired by Zhou et al.33 who showed that LLMs may craft better or comparable prompts to those developed by humans.We engineered five prompt interventions on top of the ‘out-of-the-box’ baseline: providing in-context exemplars to illustrate input-response pairs, varying the role of the questioner, requiring LLM to provide rationales for its answer, and asking LLM to answer in structured data formats. Interventions were developed in the Exemplar Cohort ℰ then tested in the Test Cohort 𝒯. Detailed illustrations of all interventions can be found in Supplemental Table S2.The baseline performance was established by asking the LLM to answer questions (𝒬1-𝒬3) for each full-text note in the Test Cohort 𝒯. The LLM response is an unstructured text string starting with “Yes” or “No”, which we took as the binary response.Our first prompt engineering intervention was to define a Questioner Role within the query, which could improve LLM responses6. We selected the following roles (ℛo1 Member of the general public; ℛo2 Healthcare worker; ℛo3 Medical doctor; ℛo4 Cardiologist; ℛo5 Cardiac electrophysiologist).The second prompt engineering intervention was to ask for a Rationale. Here, the LLM was asked to provide excerpts from the input note that support its answer/prediction.We added to the prompt “Write a response in JSON format that answers the question.” JSON (Javascript Object Notation) is a well-recognized structured format, selected because structured formats such as bulleted lists improve post-processing34 and have been recommended by OpenAI35, yet their impact on LLM performance in medical tasks is unclear.In-context exemplars are designed to illustrate input-response pairs to the LLM. We used non-repetitive full-text notes paired with their label (“Yes” or “No”), for 1 to 5 exemplars, repeated 10 times using the random.choice() function from the NumPy library with different random seeds, and then reported as mean and standard deviation across trials. All exemplar notes were randomly selected from the Exemplar Cohort ℰ, separate from the Test Cohort 𝒯.We studied the effect of integrating prompt interventions. When combining Exemplar with Rationale interventions, the contextually annotated excerpts (Figure 1) were provided to LLMs to illustrate how humans identify key excerpts (see Supplemental Table S2, Row 8 and 10 for detailed prompt designs). Importantly, we did not provide any type of annotation of test notes to the LLM.To test the generalizability across LLMs beyond GPT-4-turbo, we extended comparisons to Jurassic-2 (Amazon Bedrock) and GPT-3.5-turbo (OpenAI) on the test cohort without changing prompt interventions. Jurassic-2 was selected since it is fundamentally distinct from GPT-4-turbo and ensured that our data would not be utilized for training or tuning purposes. GPT-3.5-turbo, a predecessor model of GPT-4-turbo, was selected due to its promising performance for clinical note analyses in prior studies 36,37.Second, we compared our results from LLM with optimized prompts against BERT-based models and a logic-based NLP algorithm. To ensure consistency across evaluations, all models were optimized using the same Exemplar cohort ℰ, with model-specific optimization strategies, then evaluated on the same Test Cohort 𝒯. For BERT-based models, we evaluated SapBERT16 and Bio_ClinicalBERT38. The better one, SapBERT, was used for comparison, which has been reported to robustly encode biomedical knowledge from the Unified Medical Language System (UMLS)29, but has not been applied to detect recurrent VT or VF. We optimized BERT model hyperparameters, as well as trained the last fully connected layer on the Exemplar cohort ℰ, while the pre-trained embedding layers of BERT remained in their original state. This transfer learning approach is analogous to NLP research by Peters et al.39 and in biomedical tasks by Shrivastava et al.40 and Lai et al.41 We also used a class-balanced loss function following Cui et al.42 to address the unbalanced class distribution problem. Additionally, we developed a logic-based NLP algorithm that encodes expert domain knowledge of VT by tuning keywords, negation detection, and context logic on the Exemplar cohort ℰ, which is in agreement with studies by Zheng et al.43. The detailed implementation of both approaches is provided in Supplemental Material. Their performance was benchmarked on the Test Cohort 𝒯 against the best outcomes achieved by GPT-4-turbo.Continuous data are represented as mean ± standard deviation or as medians with interquartile range (IQR), and categorical data are represented by percentages. Student’s t-test, Mann-Whitney U test, chi-square tests, or Fisher Exact tests were applied to distinguish baseline characteristics between Exemplar Cohort ℰ and Test cohort 𝒯 as appropriate. The post-ablation VT recurrence detection performance was quantified per note using balancedaccuracy=12(sensitivity+specificity) 44. This classification metric is more sensitive to the minority but critical class (i.e., post-ablation VT recurrence in our study), which is frequently adopted for datasets characterized by unbalanced class distribution45–48. The increased certainty provided by each intervention was accessed by the Net Reclassification Improvement (NRI)49. For comparisons of various NLP techniques, balanced accuracies were compared with the NRI. The sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and F1 are also reported. Inter-observer agreement was quantified by percentage and evaluated by Fleiss’ kappa. For the text similarity for rationale comparisons, we first used the “text-embedding-3-small” model from OpenAI to generate text embeddings for each rationale, then the embeddings were then used to calculate the cosine similarity between rationales for comparison. This is analogous to the methods used by Unlu et al.50. A p<0.05 were considered statistically significant without adjustment for multiple comparisons.

    Results

    Figure 2 summarizes the performance of GPT-4-turbo across prompt interventions. At baseline, LLM achieved a balanced accuracy of 64.3% ± 4.7% (Figure 2a) in the Test Cohort, with poor sensitivity of 29.3% ± 8.3%. Figure 2b showed that adding roles did not improve balanced accuracy. Asking for rationale and requiring structured answers significantly raised balanced accuracy to 80.3% ± 5.0% (NRI = 0.392, p = 0.002, vs baseline; figure 2c) and 83.9% ± 4.7% (NRI = 0.493, p < 0.001, vs. baseline; figure 2d), respectively.Figure 2e shows that adding one expert-annotated exemplar improved balanced accuracy to 88.0% ± 3.8% (NRI=0.559, p < 0.001, vs baseline) and five exemplars raised balanced accuracy to 91.4% ± 3.5% (NRI=0.661, p < 0.001, vs baseline). The difference between one exemplar and five exemplars was not statistically different (NRI = 0.102, p = 0.208, 1 vs 5 exemplars).Supplemental figure S1 shows the impact of combining interventions. Combining JSON with rationale achieved a balanced accuracy of 83.0% ± 0.3% (Figure S1a), which is similar to JSON alone (NRI = −0.070, p = 0.328). Integrating in-context exemplars with JSON (Figure S1b), Rationale (Figure S1c), and JSON-Rationale (Figure S1d) did not enhance performance.To determine whether performance improvements from prompt engineering interventions are reproducible beyond their initial optimization, we conducted a follow-up experiment separated in time. We used the identical JSON+Rationale prompt intervention, more than three months later than the primary experiments. The new experiment achieved a balanced accuracy of 85.2% ± 2.7%, similar to the initial results of 83.0% ± 0.3% (NRI=0.043, p=0.297). This consistency and reproducibility across time suggests that the benefits of the prompt engineering techniques are durable and extend beyond a single session with the LLM.Supplemental figure S2 shows that these findings generalized to other structured formats, specifically YAML (YAML Ain’t Markup Language) or INI (Initialization file) outputs. Supplemental Table S3 summarizes prompt designs for these experiments. Results were analogous for GPT-3.5-turbo. From a lower baseline of 52.8% ± 2.0%, balanced accuracy increased to 80.8% ± 2.3% with the JSON-Rationale prompt strategy (NRI=0.613, p < 0.001, vs baseline) and 5 exemplars yielded balanced accuracy of 74.9% ± 5.8% (NRI=−0.112, p = 0.221, vs JSON-Rationale) (Supplemental figures S3 and S4). Supplemental figures S5 and S6 summarize that results were also analogous using Jurassic-2 (Amazon Bedrock).As shown in figure 3, the optimized LLM without any model retraining significantly outperformed SapBERT in balanced accuracy (p < 0.001). We evaluated BERT-based models with more training data, but found no significant improvement (Supplemental Results). LLM also surpassed the performance of the logic-based algorithm (p = 0.038). As detailed in Table 2, the LLM with optimized prompts exceeded SapBERT across all 6 evaluated metrics and demonstrated superior or comparable results relative to the logic-based algorithm. The corresponding confusion matrices are provided in Supplemental Figure S7.We further evaluated LLM responses beyond classification accuracy. Specifically, we measured the text similarity using cosine similarities of LLM-generated rationales. First, we analyzed the rationale agreement between different clinical questions under two interventions. For the Rationale intervention, the cosine similarity scores are 0.92±0.09, 0.89±0.10, and 0.89±0.10 for comparisons between 𝒬1 and 𝒬2,𝒬2 and 𝒬3, and 𝒬1 and 𝒬3, respectively. For the Rationale+JSON intervention, the scores are 0.91±0.11, 0.89±0.11, and 0.89±0.12 for the same sets of comparisons. We also compared the Rationale and Rationale+JSON interventions across the three different clinical questions, achieving a similarity score of 0.89±0.10.We further analyzed text similarity for different roles by conducting a new experiment that combined the Questioner Role, Rationale, and JSON interventions. The balanced accuracy across different roles varied from 82.8%±2.1% (ℛo1) to 85.4%±1.2% (ℛo5), comparable to that of the Rationale+JSON intervention (85.2%±2.7%). This supports previous observations that adding roles did not significantly improve balanced accuracy. Cosine similarity scores among the five roles ranged from 0.944±0.100 (ℛo2 vs ℛo5) to 0.960±0.081 (ℛo1 vs ℛo3).In summary, LLM-generated rationale showed high level of similarities across different clinical questions, prompt interventions, and questioner roles.

    Discussion

    This study demonstrates that prompt engineering can substantially improve the balanced accuracy of out-of-the-box LLM to answer nuanced medical questions from the EHR, without the need for expert domain knowledge. In 220,563 query experiments, while “out-of-the-box” GPT-4-turbo showed limited capability in detecting VT recurrence, its performance improved substantially by queries engineered to request rationale, that requiring a structured output, and that providing exemplars. Results outperformed logic-based NLP algorithms and pre-trained BERT models, were consistent across different LLMs, across numerous trials and over time. These enhancements could increase the utility of LLMs, enabling patients to understand their notes and improving existing tools for personalized healthcare advice and education51, for automated diagnostic tools, and for outcomes research.Studies are beginning to apply LLMs to medical text. Kung et al.52 demonstrated that ChatGPT can accurately answer medical questions and pass the United States Medical Licensing Exam (USMLE), and Wu et al.22 demonstrated GPT-4’s human-like test-taking ability on multiple-choice questions in the field of nephrology. However, these analyses focus on structured texts, rather than unstructured full-text medical notes that comprise diverse terminologies, acronyms, abbreviations, formats, styles, and even grammatical and typographical errors. Sushil et al.53 assessed the ability of three recent LLMs, including GPT-4 and GPT-3.5-turbo, to extract detailed breast and pancreatic cancer ontology from progress notes. However, this study used only 40 notes and focused on benchmarking rather than enhancing LLM performance.Our study used fewer exemplars than in prior research2, in which adding 10–100 exemplars to GPT-3 enhances performance in 50 non-medical datasets6,54,55. In contrast to reports for other domains2,25, we found no performance improvement from applying a questioner role and only modest improvement beyond one exemplar. The text agreement analysis showed a high level of similarity across different rationales. This finding suggested that the LLMs apply a coherent methodology to address the clinical questions and identify relevant rationale sentences, further supporting their potential clinical utility. Moreover, while initial prompts are often informed by domain knowledge, users can then engineer prompts that substantially improve LLM performance without detailed knowledge. This is demonstrated by the consistent improvements we saw from prompt engineering across several LLMs and over time. We modularized our prompt design and have made the code publicly available. It allows the lay public to easily use these prompts to understand medical records and other applications beyond the medical field.In addition to prompt engineering, retrieval-augmented generation (RAG)56 has recently gained interest in improving the performance of LLMs in medical applications. RAG employs retrieval techniques to obtain external knowledge to enhance the generative power of LLMs. For instance, Ferber et al.57, Kresevic et al.58, and Luo et al.59 demonstrated the effectiveness of RAG in retrieving clinical guidelines, which improved the performance of GPT-4 in clinical domain. The impact of RAG on LLM for interpreting unstructured EHR data is a fertile area for future study.In testing 3 distinct LLMs, we observed that these prompt engineering strategies resulted similar improvement in results, despite different baseline performances (GPT-4-turbo > GPT-3.5-turbo > Jurassic-2) and different presumed proprietary architectures. This suggests a common underlying representation in LLMs, implying that prompt engineering strategies developed for one model could be generalized to others. However, we noted model-specific differences. For instance, while the inclusion of exemplars generally boosted performance for all models, it was particularly effective for GPT-4-turbo, significantly outperforming other strategies (p < 0.05). This enhanced effectiveness is likely due to GPT-4-turbo’s ability to process considerably longer contexts (128,000 tokens) from the exemplars compared to GPT-3.5-turbo (4,096 tokens) and Jurassic-2 (8,129 tokens). Furthermore, while structured output formats such as JSON, YAML, and INI were beneficial for the GPT models, Jurassic-2 did not show similar improvements. This disparity suggests that the structured formats may be better suited to exploiting the trained internal representations of the GPT models.This approach could be applied to other cardiovascular diseases such as atrial fibrillation or coronary disease, mental health disorders60, pulmonary61 or other conditions. LLMs, with their multilingual processing capabilities62, can also bridge the language gap between healthcare professionals and patients, operating effectively within a single model. Zhu et al.63 showed LLM proficiency in multilingual machine translation across 102 languages and 606 directions. Unlu et al. demonstrated the utility of GPT-4 in screening participants for clinical trials, where it outperformed trained but non-licensed study staff in identifying symptomatic heart failure 50. LLMs can also be applied in other aspects of clinical trial management, such as streamlining trial planning, analyzing free-text narratives for coding and classification, assisting in technical writing tasks, and enhancing the process of obtaining informed consent, as summarized by Ghim et al. 64. Kraljevic et al. utilized LLMs to predict and prognosticate a wide range of future disorders, substances, procedures or findings 65. Additionally, LLM applications extend from immediate needs, such as measuring treatment fidelity, to more long-term uses, like selecting appropriate interventions or developing new therapeutic skills and evidence-based practices, as discussed in several review articles 62,66.A major challenge, however, is that LLMs do not truly understand language; they learn from statistical associations between words as used in training content, which may limit their ultimate utility and introduce potential risks 67. Additionally, LLMs are dense models that require substantial computational resources for each inference. Future studies should explore the integration of cognitive AI systems68 to enhance human-like reasoning abilities and sparse expert models69, which could potentially mitigate the current constraints on computational resource constraints but is largely underexplored in medicine.Recent studies highlight ethical risks of reconstructing training data from LLMs, even when these models are accessed through APIs70–72. To mitigate such risks, it is essential to prevent the use of real clinical data for training LLMs, which we were careful to integrate into the design of our study and selection of LLMs. Additionally, we implemented stringent data anonymization protocols, detailed in the Supplemental Material. Our rationale intervention anchors the LLM responses to the original data, providing insight into the “black-box” and potentially mitigating the “hallucinations” 73 problem where LLM ‘fabricate’ responses which may reduce ethical concerns.This proof-of-concept study should be expanded to include a more diverse patient population, a broader array of notes, and the recruitment of VT patients from external centers, which are planned extensions. It is worth noting that the approach outlined is broadly scaleable – by using regular expression (regex) matching and by adapting the keyword dictionary provided in Supplemental Table S4. Although more diverse questions and prompts could be applied, we carefully designed the primary question and prompt interventions based on domain expertise. Clinicians determined the roles and the clinical questions, while team members with expertise in machine learning, designed and varied prompt interventions, and versions with the highest balanced accuracy in the Exemplar Cohort ℰ were tested on the Test Cohort 𝒯. While prompt interventions enhanced sensitivity, they also increased false positives. This trade-off may not be problematic for some applications and this should be tested. Future efforts should attempt to refine semantic analysis to avoid this issue. We evaluated two BERT-based models which have previously performed well on clinical notes, 18,19 but they were limited by the skewed class distribution even with more training data (detailed in Supplemental Results). This primary challenge has also been demonstrated by Manela et al.74 and Mozafari et al.75 which is relevant to our study and to many clinical applications48,76. We examined only three LLMs, which we plan to extend in future work. Because the source code for LLMs remains proprietary, it is unclear if it would be possible to further optimize results by modifying its code or by adding de-identified domain-specific repositories such as ours to its original training data for modified LLMs.
    BDDr. Brototo Deb

    Dr. Brototo Deb

    Cardiology · Cardiac Critical Care
    Brigham and Women's Hospital
    brototo