The initial iteration of the national VA HF Dashboard (Dashboard 1.0) was released in 2020 and was designed to provide end users with a quick summary of their patients with HF. In order to create such a dashboard, ADS needed to use VA’s central data repository (the Corporate Data Warehouse [CDW]), which includes EHR data from 125 distinct relational Veterans Health Information Systems and Technology Architecture (VistA) instances, each with a unique set of EHR systems.11,12 Several CDW sources may capture left ventricular ejection fraction (LVEF) data, used for HF phenotypes.17–19 The first CDW source is clinical documentation stored in the VA’s nationwide electronic medical record, Computerized Patient Record System (CPRS).20 The second is radiology reports that include multiple imaging modalities except echocardiograms. The third is echocardiography reports, which are stored differently across VistA instances. The fourth is the VA Clinical Assessment and Reporting Tracking (CART) program, a clinical analytic tool used to collect patients’ information pertinent to cardiac procedures that includes a clinician’s input of a singular LVEF value and its source. The source could include echocardiography, ventriculography, magnetic resonance imaging, cardiac computed tomography, and nuclear imaging, such as single-photon emission computerized tomography and positron emission tomography.21Because of the various CDW sources used by the VA, LVEF is largely an unstructured field, not extractable by traditional informatics mechanisms. Each VA health care system uses its own local imaging and data-storage applications, so natural language processing (NLP) served as a solution to capture LVEF estimates accurately from the multiple data sources. The NLP dataset is composed of LVEF values derived from patients’ CPRS, radiology reports and echocardiography reports and has been described previously.22 To automate the identification and characterization of persons with HF, ADS developed an algorithm to extract directly LVEF estimates from the NLP dataset and to classify accurately patients on the basis of extracted values. The CART program has a requirement for a singular LVEF recording and is the only CDW source with a structured LVEF field; as a result, LVEF could be extracted directly, without the use of NLP. All LVEF information is then stored in a separate database and validated for accuracy.Dashboard 1.0 used International Classification of Diseases (ICD) codes or an active HF diagnosis in the patient’s problem list to identify veterans with HF. Regardless of source, the lowest LVEF in the past 3 years was used to determine phenotype classification. In cases where no LVEF was available, ICD codes alone were used for classification. Initial user feedback for Dashboard 1.0 noted significant inaccuracies, such as patients being assigned an incorrect phenotype classification. Studies evaluating ICD-10 codes alone to categorize HF show a lack of sensitivity and specificity23–25; similar concerns were seen with Dashboard 1.0, which noted that sensitivity was adequate (97.9%) for HF with reduced ejection fraction (HFrEF), but specificity was poor (35.1%). In contrast, for HF with preserved ejection fraction (HFpEF), sensitivity was poor (34.9%) despite adequate specificity (93.7%). Taken together, this meant that Dashboard 1.0 had an accuracy (defined as the ratio of true results [true positives + true negatives] to the total population) of 54.1% for HFrEF and 53.9% for HFpEF. Given this issue, ADS decided to improve the dashboard identification and classification schema for persons with HF.HF clinical phenotypes and stages of HF direct the evidence-based treatments for each subtype of HF.17–19 Dashboard 1.0 used HF case definitions based on the most recent American Heart Association (AHA)/American College of Cardiology (ACC) HF guidelines available.19 To begin the improvement process, ADS worked with stakeholders to establish updated HF phenotypes for the development of eCQMs: HFrEF, with LVEF ≤ 40%; HF with mildly reduced ejection fraction (HFmrEF), with LVEF 41%–49%; and HFpEF, with LVEF ≥ 50%.17 These eCQMs were specific to patients with symptomatic HF and advanced HF (ACC/AHA stages C and D); management of patients at risk for HF (stage A) and patients with pre-HF (stage B) has different evidence-based treatments and thus is outside the scope of this project.The Dashboard 1.0 classification model used the lowest LVEF from any source in the past 3 years for phenotype classification. To improve, ADS used a hierarchical algorithmic approach that considered the quality of the data source used to define LVEF. Only patients with LVEF measurements in the past 3 years were classified in the dashboard. LVEF quality was classified as high, medium or low (Table 1). High-quality LVEF included those derived from NLP of echocardiography reports as well as CART-reported LVEF, depending on the original source. Only echocardiography reports with 1 LVEF value were included in the high-quality category. Although invasive ventriculography use has decreased at the VA over time, it remains a widely performed procedure;26 invasive ventriculography can overestimate LVEF,27 so medium-quality LVEF included a CART-reported LVEF if the original source was ventriculography, as well as NLP extraction from CPRS. If NLP identified an echocardiography report with more than 1 LVEF value, it was included in the medium-quality category, with a logic developed to further clarify LVEF. Low-quality LVEF included those derived from NLP of radiology reports as well as a CART-reported EF if the original source was a nuclear imaging study, which were ultimately excluded due to their inaccuracy.If a high-quality image was available from within the past year, this image was prioritized to categorize the patient. If a high-quality image was not available from within the past year, the most recent estimate within the past 3 years was used. A phenotype classification was assigned only if recent LVEF values did not conflict with one another (eg, a patient could not be classified as having both HFrEF and HFpEF). As noted, the team identified issues with NLP recording multiple LVEF values from a single study. In this scenario, a logic was developed to exclude values that did not make sense clinically. If an LVEF value was > 75%, the patient was classified as unspecified. If the difference between the high and low LVEF values was ≤ 10%, an average EF was used. If the difference was > 10%, the high value alone was used.Key performance indicators (KPIs) are metrics used to monitor the progress of a program, and they were aligned with the most recent AHA/ACC HF guidelines.17 The KPIs selected for HF management are reported at the individual provider, facility (and selected subdivisions) and network levels, with the national average displayed for comparison. Notable KPIs included in the VA HF dashboard are: percentage of patients who are at high risk, defined as ≥ 2 HF admissions in the past year and receiving cardiology-specialty services; percentage of appropriate patients on each component of GDMT, depending on LVEF; percentage of appropriate patients on target dosages of GDMT; and percentage of patients taking high-risk medications (Table 2).17,28Based on the user feedback of Dashboard 1.0, the parameters and formatting were updated for a more intuitive interface (Fig. 1). The most recent HF guidelines use a class of recommendation systems to highlight the strength of a recommendation based on the estimated risk/benefit ratio.17 To highlight these classes of recommendation, Dashboard 2.0 incorporated a green/yellow/red coloring system to indicate the weight of each recommendation. This was done in concordance with the HF guidelines as well as prior dashboards, where green/yellow/red color coding was universally understood and considered a simple mechanism for communicating risk.13,17Hyperlinks provide immediate visualization of actionable patients. There is also a hyperlink to the updated AHA/ACC HF guidelines for further review of the recommendations pertinent to HF. Every 24 hours, the CDW extracts data pertinent to the KPIs from every VistA instance, such as medication dosaging and frequency, discharge diagnoses, laboratory results, and medical appointments, and stores them in a dataset used to update the eCQM HF dashboard; LVEF data are updated on a quarterly basis.
Results
The primary hurdle to creating a useful and accurate HF dashboard was ensuring accurate patient identification while considering relevant exclusion criteria. We present the relevant results of the optimization and utilization stages of the Dashboard 2.0 release.Chart review was performed to evaluate patient-inclusion parameters into Dashboard 2.0. Specifically, the team evaluated whether patients who lacked a diagnosis of HF according to ICD codes (or an active HF diagnosis in the problem list) but had reports of an LVEF ≤ 40% met criteria for stage C/D HF.17 In the past 3 years, we evaluated 232 patients with reduced LVEFs but without diagnoses of HF by ICD code. The group included 138 patients who had LVEF of 40% and 94 patients who had LVEFs < 40%. Overall, only 18% of these patients were found to have clinical diagnoses of HFrEF in addition to the reduced LVEF. Given this finding, patients without documented ICD diagnoses of HF or active HF diagnoses in the problem list were excluded from the dashboard.The NLP of the radiology reports were evaluated for LVEF-derivation accuracy. A total of 181 patients were selected for evaluation. Of these, 154 patients had LVEF results derived from an inaccurate or invalid study (85%), with the majority being nuclear stress tests (n = 130) or noncardiac tests such as gall-bladder ejection fraction imaging (n = 24). Given the marked imprecision of LVEFs derived from NLPs of radiology reports, the decision was made to exclude the entire radiology database from the dashboard. Given the inaccuracy of LVEFs derived from nuclear stress tests, any CART-derived LVEFs that reported “NUCLEAR” as the source were similarly excluded.After optimization was performed, a selection of random and nationally representative patients was analyzed to review the overall accuracy of Dashboard 2.0 and to compare it to Dashboard 1.0 (Table 3). The sensitivity and specificity of both the new and old algorithmic decision rules were evaluated and compared to clinician chart reviews. In this cohort of 566 patients, 59 were reported as “unspecified,” meaning their LVEF recordings were discordant or unavailable; the remaining 482 patients underwent classification based on LVEF levels.Preliminary outcomes included sensitivity, specificity and accuracy for classifying HF. Compared to Dashboard 1.0, overall accuracy for the identification of persons with HFrEF improved from 54.1% to 89.2%. Specificity improved from 35.9% to 91.9%, with a slight decrease in sensitivity from 97.9% to 83.0%. For HFpEF, accuracy improved from 53.9% to 88.0%. Specificity remained stable at 93.7%, and sensitivity improved from 34.9% to 81.6%. In sum, Dashboard 1.0 overidentified persons with HFrEF while underreporting persons with HFpEF, whereas Dashboard 2.0 improved accuracy for the HFrEF and HFpEF phenotypes. Furthermore, the updated case definitions allowed for the novel inclusion of persons with HFmrEF. For these patients, Dashboard 2.0 had an 88.0% accuracy, with 84.1% sensitivity and 89.1% specificity. Given this improvement, Dashboard 2.0 was implemented for use nationally.Current performance in quality indicators is readily available, and an example dashboard is shown (Fig. 1). KPIs are shown in a tabular format, followed by a score, usually the percentage of persons with HF who meet the quality indicator. Nationally, Dashboard 2.0 has been well received. As of July 2022, use is increasing, with 4165 hits per month (Fig. 2A) and 405 unique users per month (Fig. 2B).
Discussion
Population-management tools such as clinical dashboards allow for timely monitoring of eCQMs on individual and system levels. Creating a panel dashboard that uses NLP for monitoring and promoting best practices within a health care system is a multistep process and warrants ongoing improvement after initial deployment. We report on the development and optimization of the national VA HF dashboard. The VA system is unique, given the numerous EHR sources used regularly for clinical care. When creating a dashboard that uses different EHR databases on a national scale, NLP serves as a viable solution to gather data pertinent to HF care. However, the use of NLP alone did not provide sufficient accuracy for the initial VA HF dashboard. The stepwise process of using ICD codes and chart diagnoses for identifying persons with HF, followed by an indiscriminate approach to obtaining LVEF and HF phenotypes from NLP, led to an inaccurate identification schema and dashboard output; revision of the dashboard needed to account for imaging quality through the creation of an imaging-hierarchy model.To the best of our knowledge, this is the first report describing the creation and optimization of an HF dashboard for use on a national scale. Prior studies have been conducted at single-center sites, and they highlighted the need for continuous refinement and optimization of HF dashboards to improve performance and use.15 We build on the prior literature by describing dashboard creation on a significantly larger scale, including informatics techniques that can improve accuracy and, subsequently, dashboard use. Ultimately, key steps in improving the HF dashboard included ensuring that case definitions were up-to-date concerning revised HF guidelines and creating an imaging hierarchy with concurrent logic to determine LVEF. The use of both an imaging-quality hierarchy model and NLP dramatically improved the accuracy of the HF dashboard at the VA, reaching 88% or higher for both HFrEF and HFpEF. The positive predictive value (PPV) improved dramatically for HFrEF, from 40% to 82%, whereas the PPV for HFpEF remained above 90%. Furthermore, the new inclusion of HFmrEF, in concordance with updated AHA/ACC guidelines, is an example of how population-management tools such as clinical dashboards can and should be continually monitored and updated to ensure that the data presented stay current.Although current metrics suggest that Dashboard 2.0 has been well received, whether clinician use will translate to meaningful patient impact is yet to be seen. Clinical decision-support tools such as dashboards have generally been associated with positive outcomes13; subsequent studies will have to evaluate whether the optimization of the VA HF dashboard has improved GDMT receipt for veterans with HF.
Conclusion
This study of the VA HF dashboard provides a launchpad and roadmap for other health care networks so they can develop clinical dashboards for HF and other chronic diseases. In conclusion, we propose a 3-step process toward developing and improving a clinical dashboard. The entire process is essential to creating a functional and useful panel dashboard. To begin, we recommend the following: Determine the goal of the dashboard. This will guide case definitions for optimal patient selection and KPIs to gauge performance in terms of optimal clinical practice. Involvement of multidisciplinary stakeholders, such as clinicians, pharmacists and informaticists, can ensure that as practice evolves, case definitions and KPIs will be updated to remain consistent with current best practices. For the HF dashboard, this involved ensuring that case definitions were in concordance with national HF guidelines.Create the necessary search queries and software to extract appropriate patient information from the EHR and present the data in a functional and intuitive manner. When initial methods lead to inaccuracies, revision may involve novel approaches, such as inclusion/exclusion criteria and hierarchical-quality algorithms to ultimately ensure dashboard accuracy.Perform quality control to assess the performance of the dashboard, gauging both the accuracy of patient identification and the usability of the dashboard for end-users. Although it involves substantial manpower, manual chart review ensures that the dashboard output is concordant with the EHR. This step will create a feedback loop such that the dashboard continues to be updated and improved as usage increases.Determine the goal of the dashboard. This will guide case definitions for optimal patient selection and KPIs to gauge performance in terms of optimal clinical practice. Involvement of multidisciplinary stakeholders, such as clinicians, pharmacists and informaticists, can ensure that as practice evolves, case definitions and KPIs will be updated to remain consistent with current best practices. For the HF dashboard, this involved ensuring that case definitions were in concordance with national HF guidelines.Create the necessary search queries and software to extract appropriate patient information from the EHR and present the data in a functional and intuitive manner. When initial methods lead to inaccuracies, revision may involve novel approaches, such as inclusion/exclusion criteria and hierarchical-quality algorithms to ultimately ensure dashboard accuracy.Perform quality control to assess the performance of the dashboard, gauging both the accuracy of patient identification and the usability of the dashboard for end-users. Although it involves substantial manpower, manual chart review ensures that the dashboard output is concordant with the EHR. This step will create a feedback loop such that the dashboard continues to be updated and improved as usage increases.
BZ
Dr. Boback Ziaeian
Cardiology · General Non-Invasive CardiologyUniversity of California, Los Angelesboback