Evaluating the Appropriate Use Criteria for Coronary Revascularization in Stable Ischemic Heart Disease Using Randomized Data From the ISCHEMIA Trial.
Feb 26, 2025DOI: 10.1161/circoutcomes.124.010849Journal ArticleAHA
Written by: James SlaterDavid J MaronPhilip G JonesSripal BangaloreHarmony R ReynoldsZhuxuan FuGregg W StoneRuth KirbyJudith S HochmanJohn A SpertusISCHEMIA Research Group
In contrast to clinical guidelines that synthesize evidence to suggest what could be done to optimize care for patients, and performance measures that identify the strongest evidence to clarify what must be done in certain clinical settings, the Appropriate Use Criteria (AUC) seek to define the appropriateness of a specific test or therapy in certain clinical circumstances 1,2. The AUC blend peer reviewed evidence and clinical judgment to balance potential treatment benefits with procedural risks. The AUC for coronary revascularization, endorsed by numerous professional cardiovascular organizations in 2008, divided the indications into acute coronary syndromes and stable ischemic heart disease (SIHD), focusing primarily on the utility of coronary revascularization. A table separated appropriateness ratings for percutaneous coronary intervention (PCI) and coronary bypass surgery (CABG) for cases of complex coronary anatomy3. These AUC recommendations have become very influential, being used regularly by institutions participating in the American College of Cardiology’s National Cardiovascular Data Registry (NCDR) with evidence of changes in patient selection over time4. Moreover, governmental agencies and payers are increasingly using them to assess the value of care, even though they have not been tested using empirical evidence to validate the rankings of appropriateness5. While observational data have provided support of early versions of the AUCs, the absence of randomization poses an important risk of residual confounding6. Since 2008, the AUC recommendations have been revised once by multi-stakeholder committees of experts to keep abreast of evolving clinical knowledge7. However, the inherent reliance upon the opinions of a small group of experts undermines confidence in AUC ratings8.The International Study of Comparative Health Effectiveness with Medical and Invasive Approaches (ISCHEMIA) provides an opportunity to apply empirical, randomized data to examine the AUC recommendations for SIHD against prospectively assessed outcomes9. ISCHEMIA randomized patients with at least moderate ischemia and stable (or no) symptoms to an invasive treatment strategy with CABG or PCI (if possible) plus guideline-directed medical therapy (GDMT, INV) (n=2295) or a conservative strategy of GDMT alone (CON) (n=2322) with cardiac catheterization and revascularization reserved for failure of medical therapy. ISCHEMIA found no significant differences in the primary outcome, a composite clinical endpoint of death from cardiovascular causes, myocardial infarction, or hospitalization for unstable angina, heart failure, or resuscitated cardiac arrest, suggesting these clinical events could not be used to evaluate the appropriateness of revascularization for the AUC clinical scenarios corresponding to patients included in ISCHEMIA. Interim results from ISCHEMIA-EXTEND found a benefit for CV death, which was offset by an increase in non-CV death with invasive management and no difference in all-cause death when median follow-up was extended from 3.2 to 5.7 years10.
ISCHEMIA did, however, show an important benefit of an invasive approach in improving patients’ HS (symptoms, function, and quality of life), which is a major goal of SIHD treatment11,12. Importantly, ISCHEMIA collected sufficient patient data to enable mapping to different clinical scenarios in the existing AUC. Although ISCHEMIA cannot separate the benefits of PCI or CABG, it can estimate the benefits of an INV treatment strategy (with PCI or CABG if feasible), which is more practical given that the mode of revascularization is typically not known at the initial patient encounter, but rather is driven by coronary anatomy, clinical features portending risk after each procedure and local custom13,14. Moreover, the patient-level factors found most important in defining benefit can inform future iterations of the AUC. We thus leveraged ISCHEMIA to re-evaluate the latest version of the SIHD AUC and provide insights into how future AUCs might be designed to improve their validity and impact.
Methods
AUC clinical scenarios for SIHD from the year 2017 criteria are constructed to encompass a broad range of clinical situations, primarily based upon the extent of myocardial ischemia, coronary anatomy, and the presence of angina and diabetes. There are 96 clinical scenarios for the SIHD AUC, after excluding indications for left main disease, organ transplantation or percutaneous valve procedures, none of which is relevant to the ISCHEMIA population. Since ISCHEMIA excluded patients with local interpretations of stress tests showing no or minimal ischemia, an additional 56 AUC clinical indications could not be examined. This left 36 AUC clinical scenarios that could be mapped with participants enrolled in the ISCHEMIA trial (Table 1). We focused on the potential benefit of an invasive strategy as a whole and did not distinguish between CABG and PCI given that ISCHEMIA, like routine clinical practice, did not randomize patients to the mode of revascularization.
To include CON patients who did not undergo invasive coronary angiography as a comparator group, we used coronary computed tomography angiography (CCTA) derived data to assign patients randomized to both INV and CON strategies to the AUC clinical scenarios. Due to known differences in measurements between CCTA and invasive coronary angiography, we considered two thresholds of diameter stenosis severity based on CCTA, ≥70% and ≥50%, with the former being the primary analysis. These had accuracies of 46% and 43%, respectively, for predicting a - 70% stenosis (the definition of significant angiographic stenosis used by the AUC) among those randomized to invasive angiography. This is similar to the concordance of CCTA and invasive coronary angiography in the overall ISCHEMIA trial of 54.2% between CCTA and invasive angiography to determine the presence of single, double, or triple vessel disease15.We defined the presence of ischemic symptoms as having a baseline SAQ-7 Angina Frequency score <100, which denotes having any angina symptoms in the past 4 weeks. Ischemia severity was based on clinical site interpretations of stress tests rather than core lab assessments, to mirror usual clinical practice. In addition, the number of anti-anginal medications and the presence of diabetes were obtained using the baseline clinical data collected in the ISCHEMIA trial at the time of randomization. Patients with 3-vessel disease were classified as higher or lower SYNTAX score by identifying, among those randomized to INV, the Youden’s point for optimally discriminating the angiographic threshold (>22) based on the CCTA-estimated SYNTAX score. The optimal cut-point was determined to be a CCTA SYNTAX score ≥15 (sensitivity = 66.3%, specificity = 70.8%, PPV = 49.4%, NPV = 83.0%, c-index=0.69). The Syntax score, like all measures of coronary angiography, is also subject to interobserver reproducibility and variability even when calculated from invasive coronary angiography 16.
ISCHEMIA included 4,617 patients with quality of life data16. For the present analyses we excluded from this cohort patients with prior CABG (N=196), no CCTA performed (N=1,075), no obstructive disease (>50%) by CCTA (N=296), those for whom the number of diseased vessels by CCTA was unable to be determined (N=1,214), and those with 3-vessel disease who were missing CCTA SYNTAX scores (N=3). This left a final cohort of 1,833 patients with 928 randomized to CON and 905 randomized to INV (Figure S1). Patients randomized to INV who underwent invasive coronary angiography, but did not have revascularization either because no obstructive disease was present or it was not amenable to treatment by PCI or CABG were included in our analysis (N=172). This mimics clinical practice where a key management decision is whether to perform catheterization, which may or may not result in revascularization.
To estimate the health status benefits of an invasive strategy across the different AUC scenarios, we used the 7-item Seattle Angina Questionnaire (SAQ-7), which is a valid, reliable, and sensitive measure of patients’ disease-specific health status17,18. The SAQ-7 measures frequency of angina (Angina Frequency score), physical limitations due to angina (Physical Limitation score), and disease-specific quality of life (Quality of Life score), which are averaged to obtain a Summary Score reflecting an overall measure of patients’ angina-specific health status over the previous 4 weeks. SAQ-7 scores range from 0 to 100, with higher scores indicating less angina, better function, and higher quality of life. Angina Frequency scores of 0–30, 31–60, 61–99, and 100 denote daily, weekly, monthly and no angina over the past 4 weeks19. SAQ-7 scores closely mirror those of the full 23-item SAQ (concordance coefficients 0.88–1.00), which have been independently correlated with death, myocardial infarction and health care costs20–22. For the present study we focused on the SAQ-7 Angina Frequency and Summary scores one year after randomization. In supplemental analyses we also examined 3-year scores and time-averaged scores over three years (Figures S2–5).
For clinical events, the adjudicated 5-component primary outcome, including cardiovascular death, myocardial infarction, or hospitalization for unstable angina, heart failure or resuscitated cardiac arrest, was used over the course of follow-up for the main trial (median 3.5 years in this subset).
We used descriptive statistics to summarize baseline demographic and clinical characteristics, health status and the frequency of the 36 AUC indications, within each treatment group. We also summarized baseline and 1-year SAQ scores, and 1-year change scores, by descriptive statistics for each of the AUC indications, as well as for the 6 individual component variables comprising the indications, stratified by treatment group.
To estimate the effects of invasive management on HS outcomes within each of the AUC indications, we used Bayesian hierarchical proportional odds models. The proportional odds model is a distribution-free regression model for ordinal or continuous outcomes23. Treatment effects are represented as odds ratios (OR) for a good outcome (i.e. SAQ-7 score ≥ y; the proportional odds model assumes a common OR for values of y) associated with invasive versus conservative treatment; an OR >1.0 denotes a greater likelihood of better HS if treated invasively. We used hierarchical proportional odds models, estimating the effect of invasive treatment separately within each of the 36 AUC indication subgroups. These 36 treatment effects were treated in the model as random “data,” which we regressed on the six factors comprising the AUC criteria definitions: number of diseased vessels, proximal LAD disease, ischemic symptoms, number of anti-anginal medications, diabetes and SYNTAX score. This approach allows the treatment effect estimates within each indication to be stabilized or “shrunk” toward the predicted mean based on the six AUC variables, with the degree of shrinkage inversely related to the sample size for that indication. For example, the estimated treatment effect for patients with three-vessel disease, low SYNTAX score, diabetes, ischemic symptoms and not currently on any anti-anginal medication (Indication 19b, N=2) is substantially pulled toward the mean predicted treatment effect based on these six factors, thus borrowing information from similar patients given the sparsity of data in this particular subgroup. We used non-informative priors for all parameters. In addition to providing estimates of treatment effect for each AUC indication, the model estimates the effect of treatment within the levels of each of the six AUC factors. Results are presented as ORs and 95% highest posterior density credible intervals (CrIs) for the effect of invasive vs. conservative treatment on the likelihood of good health status outcome. Given the Bayesian framework, no p-values are provided; CrIs are interpreted as the 95% most likely values of the true ORs. In supplemental analyses, we provide estimates of each of the six AUC factors but categorizing symptoms as daily/weekly, monthly or no angina rather than dichotomizing as symptomatic vs. asymptomatic.
The primary analyses focused on 1-year SAQ outcomes. In secondary analyses we examined treatment effects within AUC indications on 3-year SAQ scores and time-averaged SAQ scores over 3 years (Figures S2–5). Because follow-up assessment completion was considerably lower at 3 years (52% vs. 91% at 1 year, primarily due to early study termination), we used multiple imputation methods, predicting missing SAQ scores on the basis of treatment, the 6 AUC variables, and other available SAQ scores. A total of 30 randomly imputed data sets were generated (sufficient to yield >97% relative efficiency for model estimates); models were fit on each of the data sets, and the resulting posterior estimates from each fit were pooled to obtain final posterior distributions incorporating uncertainty in the estimates due to missingness.
We compared baseline characteristics of the 1,833 patients in our analysis cohort to the 2,784 excluded from our analyses. (Table S1) We used standardized mean differences (SMDs) to summarize the comparisons. The SMD is the difference in means between groups divided by a pooled estimate of the standard deviation (here expressed as percent of SD; generalizations exist for binary and categorical variables as well). The SMD provides a unitless summary of the magnitude of difference between groups and is not dependent on sample size as p-values are. A threshold of 10% has been used as suggestive a non-trivial difference between groups24,25.
As a secondary analysis, the effect of randomized treatment strategy on 4-year incidence of the primary clinical outcome was estimated using Bayesian hierarchical parametric survival models. The models assumed a Weibull distribution for time to event, regressed on treatment strategy and the six AUC characteristics as fixed effects, and the 36 AUC indications and treatment by indication as random effects. The shape parameter of the Weibull distribution was also regressed on treatment strategy in order to accommodate non-proportional hazards. Absolute differences attributable to randomized treatment strategy (INV vs. CON) and 95% highest posterior density credible intervals for each AUC indication are presented in Figures 5 and 6.
Results
Of the 4,617 participants in the ISCHEMIA trial quality of life cohort, 1,833 were categorized by one of 36 AUC clinical scenarios, of whom 905 had been randomized to INV and 928 to CON. ISCHEMIA trial participants were not mapped to an AUC indication primarily due to lack of a CCTA. AUC-assigned patients differed by being younger, more likely male, fewer cardiovascular risk factors, better renal function and greater burdens of ischemia than those excluded (Table S1). The clinical characteristics of included participants, stratified by randomization to INV or CON are shown in Table 2 and were well-balanced. The mean age was 63±9.2 years, 81% were male, 71% were Caucasian, 24% Asian, 17% Hispanic and 4% black. Hypertension was present in 73% and 38% had diabetes. Two- or 3-vessel disease was present in almost half of patients. Among those with 3-vessel disease, the CCTA SYNTAX score was >15 (corresponding to an invasive angiography-derived SYNTAX score of >22) in 45% of both groups. The average SAQ-7 Summary score was 73.7 ± 18.6 in INV and 74.8 ± 18.4 in CON, 22% having had daily or weekly angina, 44% monthly angina, and 34% being asymptomatic. The categorization to the 36 specific AUC clinical scenarios is shown in Table 1 along with the AUC recommendations for CABG or PCI. ISCHEMIA patients, by trial design, had moderate or severe ischemia on stress testing as interpreted by sites. Therefore the vast majority of recommendations are “appropriate” or “may be appropriate,” except for a few scenarios where CABG was considered “rarely appropriate.”
Figure 1 shows the ORs for good 12-month health status, as measured by the SAQ-7 Summary Score, comparing INV versus CON management within each of the available 36 AUC clinical scenarios, based upon a ≥70% stenosis on CCTA to define the number of diseased vessels. Similar findings were seen using a ≥50% stenosis severity on CCTA (Figures S6– S10). Patients who were asymptomatic at baseline or symptomatic with no anti-anginal therapy derived less health status benefits from an invasive strategy. The vast majority of the 36 AUC clinical scenarios favored an invasive approach for improving health status at 12 months, as compared with a conservative strategy. A similar pattern of better health status outcomes in INV patients was also observed for the SAQ-7 Angina Frequency score (Figures 2, S11).
In an analysis of the effect of the individual components of the AUC clinical scenarios on the ORs for better health status with an invasive approach, diabetes, number of diseased vessels, proximal LAD location and SYNTAX score had little association with the health status benefits from invasive treatment. However, patients who were asymptomatic or on no anti-anginal therapy derived much less benefit from an invasive strategy than those with more symptoms or on more intensive anti-anginal therapy (Figures 3 and 4, S10–11). For example, in Figure 3, the OR for a better HS, as quantified by the SAQ Summary Score, in symptomatic INV patients was 2.04 (95% CrI 1.61 – 2.47) vs 1.32 (95% CrI 0.92 – 1.71) in asymptomatic patients. Similarly, there was minimal benefit of an invasive strategy for participants on no anti-anginal treatment (OR 1.02, 95% CrI 0.54 – 1.61), as compared with participants taking two or more anti-anginal medications (OR 1.99, 95% CrI 1.52 – 2.50). These effects differ from the similar magnitude of benefit from an invasive strategy regardless of whether participants had diabetes, the number of diseased vessels, or the location and complexity of their coronary disease.
Figures 5 and 6 show the absolute differences in event rates of the 5-component primary outcome in ISCHEMIA between INV and CON treatment strategies. All estimated differences were small (<0.25 %, suggesting a number needed to treat of >400 for any AUC subset). A trend for conservative management to be associated with fewer clinical events in patients with 3-vessel disease of low anatomic complexity and no diabetes was seen. Conversely, among patients with 3-vessel disease, complex anatomy and diabetes, the clinical event rates were lower with an invasive treatment strategy.
DISCUSSION
The AUC were a major step forward in quantifying the quality of medical decision making for performing coronary revascularization, based on a consensus process to define when the benefits of treatment were likely to outweigh its risks and costs across a range of common clinical scenarios. The evidence supporting these consensus-driven opinions, however, has been limited. The ISCHEMIA trial provides a unique opportunity to not only evaluate the empiric benefits of an invasive management strategy, as compared with initially conservative one, but also to identify which specific patient characteristics are most strongly associated with improved HS following revascularization. While very small differences in clinical outcomes between treatment strategies were observed in ISCHEMIA, HS was specified as the key secondary outcome in the trial, and an invasive approach was associated with better HS. In this study, a modest association between HS benefits of invasive management across the AUC ratings was observed. However, many of the components of the current clinical scenarios, such as diabetes, number of diseased vessels, magnitude of ischemia, proximal LAD location and SYNTAX scores, were not associated with the health status benefits of an invasive treatment approach in patients with SIHD. The most important factors determining benefit were the presence of angina and the use of prior anti-anginal therapies. As such, it may be possible to simplify future iterations of the AUCs to simplify data collection and to more closely align with emerging evidence while leaving room for clinical judgment in the application of the AUC.
These findings support and substantially extend the prior literature on this topic. For example, the lack of influence of the magnitude of ischemia on outcomes has been reported in secondary analyses of the ISCHEMIA and COURAGE trials 26–28. The importance of angina and pre-randomization medical treatment of angina on health status benefits was also demonstrated in a parallel set of analyses examining all patient factors simultaneously, as opposed to being constrained by the clinical scenarios created for the AUCs29. A prior effort to validate AUC by Ko and colleagues leveraged observational data from 2006–7 in patients with SIHD undergoing invasive angiography in Ontario to compare 3-year mortality/acute coronary syndrome rate between those who did and did not undergo revascularization, stratified by the original AUC classifications7. In contrast to our findings of relatively small differences in clinical events by treatment strategy, they found fewer cardiovascular events in patients rated as appropriate treated with revascularization and no differences between revascularization and medical therapy alone in those with uncertain and inappropriate indications. The potential biases from their observational data, or the unwillingness of physicians to randomize patients in ISCHEMIA, may explain the discrepancy in the findings of the two studies. Nevertheless, the randomization in ISCHEMIA provides an important opportunity to validate the AUC ratings for those in whom there was clinical equipoise about its benefits. Our findings also confirm and extend a single-center study demonstrating that angina frequency and the intensity of anti-anginal therapy better differentiate the benefits of invasive treatment than the other AUC clinical scenario components30.
A key foundation of professionalism in medicine is to develop the infrastructure to measure and continuously improve the value of healthcare31,32. However, the collection of data to measure healthcare quality can be expensive and opportunities to reduce this burden are important. Currently, >100 data elements are required to generate the AUC. Moreover, when key elements, such as pre-procedural stress testing, are unavailable, those patients cannot be included in an institution’s appropriateness reports. Of the 40% of patients in the NCDR that were not mapped to AUC clinical scenarios, over 80% were not assigned because of missing stress tests33. Based on the assumption that the degree of ischemia was less important than the burden of angina, Malik et al suggested substantially more patients could be included in the AUC analyses than in the NCDR analysis above, resulting in an estimated 7-fold greater prevalence of PCIs categorized as rarely appropriate33. Reducing the number of data elements required to place patients into AUC clinical scenarios could therefore increase the applicability and lower the costs of generating the AUC. However, it may also be important to include assessment of angina directly from patients, such as using the SAQ, rather than applying potentially less accurate estimates of angina that may not reflect patients’ self-reported symptoms34.
Another key consideration for future versions of the AUC in SIHD is whether to focus on the mode of revascularization or only emphasize the decision to proceed to revascularization using either PCI or CABG. In 2017, the AUC were expanded to separately rate the appropriateness of each SIHD clinical scenario. Based on empiric evidence from the ISCHEMIA trial that HS benefits of revascularization were primarily in patients with symptoms treated with anti-anginal medications, the choice of revascularization strategy might be best deferred to local expertise. A table comparing PCI v CABG outcomes in major clinical trials where these treatment strategies were randomized could still be included, but may be needed for only a subset of clinical scenarios. Such an AUC redesign could both reduce the burden of data collection and support their prospective implementation to support clinical decision-making.
These findings should be interpreted in the context of the following limitations. Our work seeks to leverage a randomized trial to provide empirical evidence to support and improve the AUC for SIHD. We included all patients randomized to the invasive strategy because the intent was to offer revascularization, if possible, although some did not undergo revascularization because their coronary disease was not significant or too advanced to be treated invasively. Thus, these benefits of revascularization are based upon the intent to revascularize, rather than the procedure itself. Our results should not be applied to AUC recommendations for patients excluded from ISCHEMIA, including patients with left main disease, severely reduced left ventricular function, and unacceptable angina. An additional concern about the generalizability of these findings is that all patients were participating in a clinical trial with protocols and support to optimize GDMT and the revascularization skills of participating centers may differ from other centers. Given the modest accuracy of CCTA to identify vessels with ≥70% diameter stenosis in ISCHEMIA compared with invasive quantitative coronary angiography, the AUC scenario assignments of some patients may have been incorrect with respect to angiographic disease severity. It should be noted, however, that invasive coronary angiography is also subject to substantial variability across physicians, which can also lead to misclassification35,36.
Finally, the primary inferential limitation of our study is that the sample sizes for all 36 AUC indications were small, limiting the ability to precisely estimate the effect of invasive treatment within individual indications. The difference in mean 12-month SAQ-7 Summary Scores between treatment groups in the overall ISCHEMIA trial was 3.0 points, with a standard deviation of 15.6 points. Assuming this was the true average benefit of invasive treatment, the minimum sample size required to detect an effect of this magnitude with 80% power in any subgroup analysis would require ≥850 patients, well beyond the sample size of any of the AUC indication subgroups. To address this challenge, we used Bayesian hierarchical models, regressing the 36 individual AUC treatment effects on the six clinical factors that constitute the AUC definitions. This approach borrows information from similar patients to help inform the estimates for the individual AUC indications. It is essential to understand this borrowing effect when interpreting the results as it implies that the individual AUC treatment effect estimates represent our “best guess” at the effect of treatment for that indication, given both the amount of unique information in that subgroup (i.e., sample size) and the average treatment effect among patients with the same covariates across the entire cohort. As the subgroup sample size decreases, the estimate moves toward the average.
Importantly, the results of our analysis should not be used to infer that there is no benefit of invasive treatment within any individual AUC indication solely on the basis of the credible interval crossing 1.0. Comparisons of estimates between different subgroups should be considered with great caution. Any conclusions should ideally be supported by sound clinical rationale, such as that HS benefit of treatment is minimal among asymptomatic patients. The most appropriate high-level interpretation of our findings is that we did not find evidence of substantial variation in treatment effects across AUC indications, except for patients with no or minimal symptoms and who were not receiving anti-anginal medical treatment at study entry. Though variation may exist, our study was underpowered to detect it. It is puzzling that symptomatic patients on no anti-anginal therapy had minimal symptomatic improvement since angina commonly improves after revascularization. One possibility is that patients in the CON group on no baseline angina therapy had an opportunity to add anti-anginal therapy over time whereas patients already on medication had fewer options to augment therapy.
Conclusions
To empirically support the rankings of AUC scenarios for revascularization in SIHD, we used data from the randomized ISCHEMIA trial and found that the primary drivers of HS benefits from an invasive treatment strategy are the presence of pre-procedural angina and the use of anti-anginal medications. These HS outcomes are important because the absolute differences in clinical events across all AUC categories were small. By focusing future AUC ratings on those factors influencing health status outcomes (except, perhaps for left main disease), it may be possible to reduce the burden of data collection, and enable the AUC to be more inclusive of all treated patient, and to be implemented in clinical workflow to better support clinical decision-making in SIHD.
GS
Dr. Gregg Stone
Cardiology · Interventional CardiologyIcahn School of Medicine at Mount Sinaigregg
