Elsevier

The Spine Journal

Volume 21, Issue 10, October 2021, Pages 1679-1686
The Spine Journal

Clinical Study
Focus: Artificial Intelligence and Machine Learning
Updated external validation of the SORG machine learning algorithms for prediction of ninety-day and one-year mortality after surgery for spinal metastasis

https://doi.org/10.1016/j.spinee.2021.03.026Get rights and content

Highlights

  • Patient survival is a key consideration when determining treatment for spinal metastasis.

  • Physician predictions for life expectancy in the setting of metastatic cancer are often inaccurate.

  • The SORG machine learning algorithms accurately predict risk of 90-day and 1-year mortality in patients undergoing surgery for spinal metastasis.

  • With an independent cohort of 298 patients treated between 2004 and 2020, we report successful external validation of the SORG algorithms for preoperative risk prediction of 90-day and 1-year mortality after surgery for spinal metastasis.

ABSTRACT

BACKGROUND CONTEXT

Surgical decompression and stabilization in the setting of spinal metastasis is performed to relieve pain and preserve functional status. These potential benefits must be weighed against the risks of perioperative morbidity and mortality. Accurate prediction of a patient's postoperative survival is a crucial component of patient counseling.

PURPOSE

To externally validate the SORG machine learning algorithms for prediction of 90-day and 1-year mortality after surgery for spinal metastasis.

STUDY DESIGN/SETTING

Retrospective, cohort study

PATIENT SAMPLE

Patients 18 years or older at a tertiary care medical center treated surgically for spinal metastasis

OUTCOME MEASURES

Mortality within 90 days of surgery, mortality within 1 year of surgery

METHODS

This is a retrospective cohort study of 298 adult patients at a tertiary care medical center treated surgically for spinal metastasis between 2004 and 2020. Baseline characteristics of the validation cohort were compared to the derivation cohort for the SORG algorithms. The following metrics were used to assess the performance of the algorithms: discrimination, calibration, overall model performance, and decision curve analysis.

RESULTS

Sixty-one patients died within 90 days of surgery and 133 died within 1 year of surgery. The validation cohort differed significantly from the derivation cohort. The SORG algorithms for 90-day mortality and 1-year mortality performed excellently with respect to discrimination; the algorithm for 1-year mortality was well-calibrated. At both postoperative time points, the SORG algorithms showed greater net benefit than the default strategies of changing management for no patients or for all patients.

CONCLUSIONS

With an independent, contemporary, and geographically distinct population, we report successful external validation of SORG algorithms for preoperative risk prediction of 90-day and 1-year mortality after surgery for spinal metastasis. By providing accurate prediction of intermediate and long-term mortality risk, these externally validated algorithms may inform shared decision-making with patients in determining management of spinal metastatic disease.

Introduction

The spinal column is the most common site of bony metastasis, with approximately 5% to 10% of patients with cancer developing metastatic epidural compression [1]. As survival rates of many malignancies increase in an aging population, the prevalence of spinal metastatic disease is similarly expected to increase. After a prospective study by Patchell and colleagues in 2005 demonstrated the superiority of surgical decompression with radiotherapy versus radiotherapy alone, the rate of surgery for spinal metastases has also increased [2,3]. Surgical decompression and stabilization are typically performed to alleviate pain from mechanical instability or to relieve neurologic compromise [4]. Yet spinal surgery in this patient population is not without risk. Complications associated with surgery for spinal metastatic disease are a significant source of morbidity and mortality and include wound infections, neurologic impairment, venous thromboembolism, and instrumentation failure [5,6]. Patients with metastatic disease generally have multiple medical comorbidities with associated immunosuppression and are less likely to recover from postoperative complications.

Given these considerations, the benefits of alleviating pain and improving ambulation must be weighed against the risk of complications. It is thus important to obtain accurate short-term and long-term prognoses when determining a treatment plan. Unfortunately, physicians’ clinical prediction for life expectancy in the setting of metastatic cancer is often not accurate [[7], [8], [9]]. Scoring systems and prognostic calculators in the literature largely fail to estimate survival probability at different time points and perform inconsistently across different tumor types [4,[10], [11], [12], [13], [14], [15], [16], [17], [18], [19]].

SORG previously developed machine learning (ML) algorithms to preoperatively predict risk of 90-day and 1-year mortality in patients undergoing surgery for spinal metastasis; these have been made available in an open access web application [20]. These algorithms are accurate with high discriminatory capability and are internally valid. For these tools to be of clinical utility, however, they must be externally validated in multiple independent populations. We aim to evaluate the performance of these algorithms on an independent contemporary patient cohort from a geographically distinct institution and assess their external validity.

Section snippets

Patient sample

Our institutional review board approved retrospective review of electronic medical records. Individual patient consent was waived as the study was limited to retrospective review of medical charts. Inclusion criteria for the study were: (1) age greater than 18 years; (2) initial decompression and/or fusion for spinal metastatic disease; (3) surgery between 2004 and 2020. Patients who underwent surgical intervention for primary spinal tumors as well as those who had revision procedures for

Results

Overall, 298 patients underwent surgery for spinal metastatic disease. Two hundred and twelve (71.1%) were treated in 2010 or later, and 124 (41.6%) were treated since 2015. The majority of patients (57.4%) were male. The median age was 61 years. With respect to primary tumor histology groups, 34.9% were slow growth, 38.9% were moderate growth, and 26.2% were rapid growth. One hundred and four patients (35.0%) had presence of lung or liver metastasis and 23 patients (7.8%) had metastatic

Discussion

With the increasing prevalence of malignancy in the aging population and advances in medical and radiation oncology, the incidence of spinal metastatic disease is likely to increase. Expected postoperative survival is a critical consideration when weighing the risks and benefits of operative intervention for spinal metastasis. There have been multiple scoring systems proposed to predict survival of patients and determine appropriateness of surgery, yet there remains little consensus as to which

Declarations of Competing Interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References (43)

  • AV. Karhade et al.

    External validation of the SORG 90-day and 1-year machine learning algorithms for survival in spinal metastatic disease

    Spine J

    (2020)
  • M Bongers et al.

    Does the SORG algorithm generalize to a contemporary cohort of patients with spinal metastases on external validation

    Spine J

    (2020)
  • T Azad et al.

    Geographic variation in the surgical management of lumbar spondylolisthesis: characterizing practice patterns and outcomes

    Spine J

    (2018)
  • Y Vergouwe et al.

    Substantial effective sample sizes were required for external validation studies of predictive logistic regression models

    J Clin Epidemiol

    (2005)
  • DM Sciubba et al.

    Diagnosis and management of metastatic spine disease: a review

    J Neurosurg Spine

    (2010)
  • HM Carl et al.

    Risk factors for wound-related reoperations in patients with metastatic spine tumor

    J Neurosurg Spine

    (2018)
  • A Vigano et al.

    The relative accuracy of the clinical estimation of the duration of life for patients with end of life cancer

    Cancer

    (1999)
  • C Clement-Duchene et al.

    How accurate are physicians in the prediction of patient survival in advanced lung cancer?

    Oncologist

    (2010)
  • Tomita K, Kawahara N, Kobayashi T, Yoshida A, Murakami H, Akamaru T. Surgical strategy for spinal metastases. Spine...
  • Chen H, Xiao J, Yang X, Zhang F, Yuan W. Preoperative scoring systems and prognostic factors for patients with spinal...
  • A Hernandez-Fernandez et al.

    External validity of the Tokuhashi score in patients with vertebral metastasis

    J Cancer Res Clin Oncol

    (2012)
  • Cited by (21)

    • Development and external validation of predictive algorithms for six-week mortality in spinal metastasis using 4,304 patients from five institutions

      2022, Spine Journal
      Citation Excerpt :

      We previously developed algorithms for prediction of 3-month and 1-year mortality in patients undergoing surgery for spinal metastatic disease [28]. These algorithms have been externally both domestically and internationally [29–32]. Existing studies for shorter postoperative survival suffer from a lack of disease-specific variables such as primary tumor histology, relatively low sample size, or lack of assessment of prediction model metrics such as calibration, decision curve analysis, and Brier score [33–35].

    • Prognostic significance of lab data and performance comparison by validating survival prediction models for patients with spinal metastases after radiotherapy

      2022, Radiotherapy and Oncology
      Citation Excerpt :

      SORG-MLA exhibited the best performance among the three SPMs across all performance metrics, including discrimination, calibration, and decision curve analyses. The performance was in accordance with that observed during previous external validations, and this is the first study to support the use of SORG-MLA for patients receiving radiotherapy [30–33]. Compared with the other two competing SPMs, SORG-MLA has some advantages.

    • A machine learning algorithm for predicting prolonged postoperative opioid prescription after lumbar disc herniation surgery. An external validation study using 1,316 patients from a Taiwanese cohort

      2022, Spine Journal
      Citation Excerpt :

      Furthermore, decision-curve analysis shows this model is suitable for clinical use. External validation is an often-overlooked element in the process of incorporating prediction models in current clinical practice [44,46–48,51–58]. While there has been an abundance of ML prediction models in orthopedics as of late [47,59–62], a study by Groot et al [63].

    View all citing articles on Scopus

    FDA device/drug status: Not applicable.

    Author Disclosures: AAS: Nothing to disclose. AVK: Nothing to disclose. HYP: Nothing to disclose. WLS: Nothing to disclose. LJM: Nothing to disclose. RGE: Nothing to disclose. ANS: Nothing to disclose. DYP: Consluting: Seaspine (C), Globus (B), Nuvasive (B), Stryker (C). JHS: Scientific Advisory Board: Chordoma Foundation (Nonfinancial); Speaking and/or Teaching Arrangements: AO Spine (Travel Expense Reimbursement, Outside 12-Month Requirement), Stryker Spine (B, Outside 12-Month Requirement). FJH: Nothing to disclose.

    Funding disclosure: There was no financial support for this study.

    View full text