Advertisement
Heart, Lung and Circulation

Evidence-Based Determination of Cut-Off Points for Increased Cardiac-Surgery Mortality Risk With EuroSCORE II and STS: The Best-Performing Risk Scoring Models in a Single-Centre Australian Population

Published:October 27, 2021DOI:https://doi.org/10.1016/j.hlc.2021.08.026

      Objectives

      Risk scoring models (RSMs) are commonly used for estimation of postoperative-mortality risk in patients undergoing cardiac surgery, but their prediction accuracy may vary in different populations and clinical situations. The prognostic accuracies of some RSMs have not yet been fully evaluated in the Australian population. In this retrospective observational study, our aims were to assess the performance of four contemporary RSMs, to identify the best RSMs for prediction of postoperative-mortality in the single-centre cohort, and to determine a statistical threshold for classification of patients with increased or “higher” mortality risk.

      Methods

      The study population included patients who underwent cardiac surgery at Liverpool Hospital between January 2013 and December 2014. Demographic information was collected, and mortality risks were estimated with the ES2 (EuroSCORE II), STS (Society of Thoracic Surgeons Score), AS (AusSCORE total) and ASMR (AusSCORE multi-risk) RSMs. (Additive EuroSCORE) (AES) and LES (logistic EuroSCORE) were included for historical interest. Discrimination, the ability to stratify patients between mortality and no mortality outcomes, and calibration, the comparison of risk score estimated and observed outcome in the population, were evaluated for each RSM, to determine their predictive accuracy in the study population. Discrimination was assessed by the AUC (area under the receiver operating characteristic curve), and acceptable calibration by the p-value greater than 0.05 for the Hosmer–Lemeshow (H-L) test. The best AUCs in contempory models were compared using the DeLong test. For ES2 and STS risk scores, cut-off points, or thresholds, for patients at increased risk of mortality were derived using Youden’s J-statistics, calculated from sensitivity and specificity of models in predicting mortality.

      Results

      From a total study population of 898 patients, 738 had scores for all six RSMs. The three EuroSCORE risk models and Youden’s J-statistics analysis included the total population. Of the models in contemporary use, ES2 had higher discrimination (AUC=0.850) in this population than ASMR (AUC=0.767, p=0.024) and AS (AUC=0.739) and non-significantly higher discrimination than STS (AUC=0.806, p=0.19). All contemporary models had acceptable calibration but the older LES (H-L p=0.024) did not. Estimated mortality was closest to observed mortality with the ES2 model. Both AES and LES over predicted mortality. The RSM with the highest discrimination in isolated coronary artery bypass graft surgery (CAGs) (AUC=0.847), isolated valves (AUC=0.830), and females (AUC=0.784) was the ES2 model. STS discrimination was highest in CAGs plus valve procedures (AUC 0.891), and males (STS AUC=0.891). Cut-off points for risk scores to define increased risk populations were 3.0% for ES2 and 1.7% for STS. Similar proportions of patients in each RSM (ES2-26% to STS-32%) were defined as higher risk by the model threshold score depending on type of procedure.

      Conclusion

      Among RSMs in contemporary use, ES2 and STS showed the best discrimination and acceptable calibration. Caution is recommended in specific subgroups. Increased mortality risk score cut-off points could be identified for these two RSMs in this single-centre cohort.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Heart, Lung and Circulation
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Dewey T.M.
        • Brown D.
        • Ryan W.H.
        • Herbert M.A.
        • Prince S.L.
        • Mack M.J.
        Reliability of risk algorithms in predicting early and late operative outcomes in high-risk patients undergoing aortic valve replacement.
        The J Thorac Cardiovasc Surg. 2008; 135: 180-187
        • Kirmani B.H.
        • Mazhar K.
        • Fabri B.M.
        • Pullan D.M.
        Comparison of the EuroSCORE II and Society of Thoracic Surgeons 2008 risk tools.
        Eur J Cardiothorac Surg. 2013; 44 (discussion 1005): 999-1005
        • Nashef S.A.
        • Roques F.
        • Sharples L.D.
        • Nilsson J.
        • Smith C.
        • Goldstone A.R.
        • et al.
        EuroSCORE II.
        Eur J Cardiothorac Surg. 2012; 41 (discussion 44-5): 734-744
        • Shahian D.M.
        • O'Brien S.M.
        • Filardo G.
        • Ferraris V.A.
        • Haan C.K.
        • Rich J.B.
        • et al.
        The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 1--coronary artery bypass grafting surgery.
        Ann Thorac Surg. 2009; 88: S2-S22
        • O'Brien S.M.
        • Shahian D.M.
        • Filardo G.
        • Ferraris V.A.
        • Haan C.K.
        • Rich J.B.
        • et al.
        The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 2--isolated valve surgery.
        Ann Thorac Surg. 2009; 88: S23-S42
        • Shahian D.M.
        • O'Brien S.M.
        • Filardo G.
        • Ferraris V.A.
        • Haan C.K.
        • Rich J.B.
        • et al.
        The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 3--valve plus coronary artery bypass grafting surgery.
        Ann Thorac Surg. 2009; 88: S43-S62
        • Nilsson J.
        • Algotsson L.
        • Hoglund P.
        • Luhrs C.
        • Brandt J.
        Comparison of 19 pre-operative risk stratification models in open-heart surgery.
        Eur Heart J. 2006; 27: 867-874
        • Reid C.
        • Billah B.
        • Dinh D.
        • Smith J.
        • Skillington P.
        • Yii M.
        • et al.
        An Australian risk prediction model for 30-day mortality after isolated coronary artery bypass: the AusSCORE.
        J Thorac Cardiovasc Surg. 2009; 138: 904-910
        • Billah B.
        • Reid C.
        • Shardey G.C.
        • Smith J.A.
        A preoprative risk prediction model for 30-day mortality following cardiac surgery in an Australian cohort.
        Eur J Cardiothorac Surg. 2010; 37: 1086-1092
        • Billah B.
        • Huq M.M.
        • Smith J.A.
        • Sufi F.
        • Tran L.
        • Shardey G.C.
        • et al.
        AusSCORE II in predicting 30-day mortality after isolated coronary artery bypass grafting in Australia and New Zealand.
        J Thorac Cardiovasc Surg. 2014; 148: 1850-1855 e2
        • Neumann F.J.
        • Sousa-Uva M.
        • Ahlsson A.
        • Alfonso F.
        • Banning A.P.
        • Benedetto U.
        • et al.
        2018 ESC/EACTS Guidelines on myocardial revascularization.
        Eur Heart J. 2019; 40: 87-165
        • Chalmers J.
        • Pullan M.
        • Fabri B.
        • McShane J.
        • Shaw M.
        • Mediratta N.
        • et al.
        Validation of EuroSCORE II in a modern cohort of patients undergoing cardiac surgery.
        Eur J Cardiothorac Surg. 2013; 43: 688-694
        • Paparella D.
        • Guida P.
        • Di Eusanio G.
        • Caparrotti S.
        • Gregorini R.
        • Cassese M.
        • et al.
        Risk stratification for in-hospital mortality after cardiac surgery: external validation of EuroSCORE II in a prospective regional registry.
        Eur J Cardiothorac Surg. 2014; 46: 840-848
        • Wendt D.
        • Osswald B.R.
        • Kayser K.
        • Thielmann M.
        • Tossios P.
        • Massoudy P.
        • et al.
        Society of Thoracic Surgeons score is superior to the EuroSCORE determining mortality in high risk patients undergoing isolated aortic valve replacement.
        Ann Thorac Surg. 2009; 88 (discussion 74-5): 468-474
        • Spiliopoulos K.
        • Bagiatis V.
        • Deutsch O.
        • Kemkes B.M.
        • Antonopoulos N.
        • Karangelis D.
        • et al.
        Performance of EuroSCORE II compared to EuroSCORE I in predicting operative and mid-term mortality of patients from a single center after combined coronary artery bypass grafting and aortic valve replacement.
        Gen Thorac Cardiovasc Surg. 2014; 62: 103-111
        • Wang T.K.
        • Li A.Y.
        • Ramanathan T.
        • Stewart R.A.
        • Gamble G.
        • White H.D.
        Comparison of four risk scores for contemporary isolated coronary artery bypass grafting.
        Heart Lung Circ. 2014; 23: 469-474
        • Hussain A.
        • Arifi A.A.
        • Najm H.K.
        Predicted and observed mortality in adult cardiac surgery in the King Abdulaziz Cardiac Center: application of EuroSCORE I and EuroSCORE II.
        Eur Heart J Suppl. 2014; 16: B55-B58
        • Marney
        • Watson A.
        • Cole C.
        • Mundy J.
        Validation of EuroSCORE II in a single centre Australian coholt.
        Heart Lung Circ. 2014; 23: e41
        • Stähli B.E.
        • Tasnady H.
        • Lüscher T.F.
        • Gebhard C.
        • Mikulicic F.
        • Erhart L.
        • et al.
        Early and late mortality in patients undergoing transcatheter aortic valve implantation: comparison of the novel EuroScore II with established risk scores.
        Cardiology. 2013; 126: 15-23
        • Dhurandhar V.
        • Saxena A.
        • Parikh R.
        • Vallely M.P.
        • Wilson M.K.
        • Butcher J.K.
        • et al.
        Outcomes of on-pump versus off-pump coronary artery bypass graft surgery in the high risk (AusSCORE > 5).
        Heart Lung Circ. 2015; 24: 1216-1224
        • Barili F.
        • Pacini D.
        • Capo A.
        • Ardemagni E.
        • Pellicciari G.
        • Zanobini M.
        • et al.
        Reliability of new scores in predicting perioperative mortality after isolated aortic valve surgery: a comparison with the society of thoracic surgeons score and logistic EuroSCORE.
        Ann Thorac Surg. 2013; 95: 1539-1544
        • Ramponi R.
        • Black D.
        • Bannon P.
        • Reid C.
        • Smith J.
        • Cartwright B.
        Accuracy, calibration, and clinical performance of EuroSCORE II and age creatinine, and ejection fraction score in 53,472 isolated coronary artery bypass surgeries in Australia and New Zealand.
        Heart Lung Circ. 2018; 27: S532
        • Arangalage D.
        • Cimadevilla C.
        • Alkhoder S.
        • Chiampan A.
        • Himbert D.
        • Brochet E.
        • et al.
        Agreement between the new EuroSCORE II, the logistic EuroSCORE and the Society of Thoracic Surgeons score: implications for transcatheter aortic vlave implantation.
        Arch Cardiovasc Dis. 2014; 107: 353-360
        • Nashef S.A.
        • Roques F.
        • Michel P.
        • Gauducheau E.
        • Lemeshow S.
        • Salamon R.
        European system for cardiac operative risk evaluation (EuroSCORE).
        Eur J Cardiothorac Surg. 1999; 16: 9-13
        • Coylewright M.
        • Mack M.F.
        • Holmes D.R.
        • O'Gara P.T.
        A call for an evidence-based approach to the Heart Team for patients with severe aortic stenosis.
        J Am Coll Card. 2015; 65: 1472-1480
        • Rajakaruna C.
        • Rogers C.A.
        • Angelini D.
        • Ascione R.
        Risk factors for and economic implications of prolonged ventilation after cardiac surgery.
        J Thorac and Cardiovasc Surg. 2005; 130: 1270-1277
        • Rodwin B.A.
        • Spruill T.M.
        • Ladapo J.A.
        Economics of psychosocial factors in patients with cardiovascular disease.
        Prog Cardiovasc Dis. 2013; 55: 563-573
        • DeLong Er
        • DeLong Dm
        • Clarke-Pearson D.L.
        Comparing areas under two or more correlated receiver operating characteristics curves: a non parametric approach.
        Biometrics. 1988; 44: 837-845
        • Carter J.V.
        • Pan J.
        • Rai S.N.
        • Galandiuk S.
        ROC-ing along: evaluation and interpretation of receiver operating characteristic curves.
        Surgery. 2016; 159: 1638-1645
        • Youden J.
        An index for rating diagnostic tests.
        Cancer. 1950; 3: 32-35
        • Schisterman E.F.
        • Perkins N.J.
        • Liu A.
        • Bondell H.
        Optimal cut-point and its corresponding Youden index to discriminate individuals using pooled blood samples.
        Epidemiology. 2005; 16: 73-81
        • Ruopp M.D.
        • Perkins N.J.
        • Whitcomb B.W.
        • Schisterman E.F.
        Youden index and optimal cut-point estimated from observations affected by a lower limit of detection.
        Biom J. 2008; 50: 419-430
        • Piazza N.
        • Wenaweser P.
        • van Gameren M.
        • Pilgrim T.
        • Tzikas A.
        • Otten A.
        • et al.
        Relationship between the logistic EuroSCORE and the Society of Thoracic Surgeons Predicted Risk of Mortality score in patients implanted with the CoreValve ReValving system--a Bern-Rotterdam Study.
        Am Heart J. 2010; 159: 323-329
        • Yap C.H.
        • Reid C.
        • Yii M.
        • Rowland M.A.
        • Mohajeri M.
        • Skillington P.D.
        • et al.
        Validation of the EuroSCORE model in Australia.
        Eur J Cardiothorac Surg. 2006; 29 (discussion 6): 441-446
        • Karim M.N.
        • Reid C.M.
        • Cochrane A.
        • Tran L.
        • Billah B.
        When is urgent really urgent and does it matter? Misclassification of procedural status and implication for risk assessment in cardiac surgery.
        Heart Lung Circ. 2016; 25: 196-203
        • Karim M.N.
        • Reid C.M.
        • Cochrane A.
        • Tran L.
        • Billah B.
        Reply to Letter to the Editor - The Definition of 'urgent' in Australia and New Zealand Society of Cardiac and Thoracic Surgeons database.
        Heart Lung Circ. 2016; 25: 1141-1142
        • Zhao M.
        • Stampt S.
        • Valina C.
        • Kienzle R.P.
        • Ferenc M.
        • Gick M.
        • et al.
        Role of EuroSCORE II in predicting long-term outcome after percutaneous catheter intervention for coronary triple vessel disease or left main stenosis.
        Int J Cardiol. 2013; 168: 3273-3279
        • Farooq V.
        • van Klaveren D.
        • Steyerberg E.W.
        • Meliga E.
        • Vergouwe Y.
        • Cheiffo A.
        • et al.
        Anatomical and clinical characteristics to guide decision making between coronary artery bypass surgery and percutaneous coronary intervention for individual patients: development and validation of SYNTAX score II.
        Lancet. 2013; 381: 639-650
        • Afilalo J.
        • Mottillo S.
        • Eisenberg M.J.
        • Alexander K.P.
        • Noiseux N.
        • Perrault L.P.
        • et al.
        Addition of frailty and disability to cardiac surgery risk scores identifies elderly patients at high risk of mortality or major morbidity.
        Circ Cardiovasc Qual Outcomes. 2012; 5: 222-228
        • Amrock L.G.
        • Deiner S.
        The implication of frailty on preoperative risk assessment.
        Curr Opin Anaesthesiol. 2014; 27: 330-335
        • Engleman D.T.
        • Adams D.H.
        • Byrne J.G.
        • Aranki S.F.
        • Collins Jr., J.J.
        • Couper G.S.
        • et al.
        Impact of body mass index and albumin on morbidity and mortality after cardiac surgery.
        J Thorac Cardiovasc Surg. 1999; 118: 866-873