Skip to main content

Advertisement

SpringerLink
Log in

External validation of a prediction model for pain and functional outcome after elective lumbar spinal fusion

  • Original Article
  • Published:
European Spine Journal Aims and scope Submit manuscript

Abstract

Objective

Patient-reported outcome measures following elective lumbar fusion surgery demonstrate major heterogeneity. Individualized prediction tools can provide valuable insights for shared decision-making. We externally validated the spine surgical care and outcomes assessment programme/comparative effectiveness translational network (SCOAP-CERTAIN) model for prediction of 12-month minimum clinically important difference in Oswestry Disability Index (ODI) and in numeric rating scales for back (NRS-BP) and leg pain (NRS-LP) after elective lumbar fusion.

Methods

Data from a prospective registry were obtained. We calculated the area under the curve (AUC), calibration slope and intercept, and Hosmer–Lemeshow values to estimate discrimination and calibration of the models.

Results

We included 100 patients, with average age of 50.4 ± 11.4 years. For 12-month ODI, AUC was 0.71 while the calibration intercept and slope were 1.08 and 0.95, respectively. For NRS-BP, AUC was 0.72, with a calibration intercept of 1.02, and slope of 0.74. For NRS-LP, AUC was 0.83, with a calibration intercept of 1.08, and slope of 0.95. Sensitivity ranged from 0.64 to 1.00, while specificity ranged from 0.38 to 0.65. A lack of fit was found for all three models based on Hosmer–Lemeshow testing.

Conclusions

The SCOAP-CERTAIN tool can accurately predict which patients will achieve favourable outcomes. However, the predicted probabilities—which are the most valuable in clinical practice—reported by the tool do not correspond well to the true probability of a favourable outcome. We suggest that any prediction tool should first be externally validated before it is applied in routine clinical practice.

Graphic abstract

These slides can be retrieved under Electronic Supplementary Material.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Steinmetz MP, Mroz T (2018) Value of adding predictive clinical decision tools to spine surgery. JAMA Surg. https://doi.org/10.1001/jamasurg.2018.0078

    Article  PubMed  Google Scholar 

  2. Khor S, Lavallee D, Cizik AM et al (2018) Development and validation of a prediction model for pain and functional outcomes after lumbar spine surgery. JAMA Surg. https://doi.org/10.1001/jamasurg.2018.0072

    Article  PubMed  PubMed Central  Google Scholar 

  3. Siccoli A, de Wispelaere MP, Schröder ML, Staartjes VE (2019) Machine learning–based preoperative predictive analytics for lumbar spinal stenosis. Neurosurg Focus 46:E5. https://doi.org/10.3171/2019.2.FOCUS18723

    Article  PubMed  Google Scholar 

  4. Staartjes VE, de Wispelaere MP, Vandertop WP, Schröder ML (2018) Deep learning-based preoperative predictive analytics for patient-reported outcomes following lumbar diskectomy: feasibility of center-specific modeling. Spine J Off J North Am Spine Soc. https://doi.org/10.1016/j.spinee.2018.11.009

    Article  Google Scholar 

  5. Janssen DMC, van Kuijk SMJ, d’Aumerie B, Willems P (2019) A prediction model of surgical site infection after instrumented thoracolumbar spine surgery in adults. Eur Spine J. https://doi.org/10.1007/s00586-018-05877-z

    Article  PubMed  Google Scholar 

  6. Janssen DMC, van Kuijk SMJ, d’Aumerie BB, Willems PC (2018) External validation of a prediction model for surgical site infection after thoracolumbar spine surgery in a Western European cohort. J Orthop Surg. https://doi.org/10.1186/s13018-018-0821-2

    Article  Google Scholar 

  7. Senders JT, Staples PC, Karhade AV et al (2018) Machine learning and neurosurgical outcome prediction: a systematic review. World Neurosurg 109:476–486.e1. https://doi.org/10.1016/j.wneu.2017.09.149

    Article  PubMed  Google Scholar 

  8. Brusko GD, Kolcun JPG, Wang MY (2018) Machine-learning models: the future of predictive analytics in neurosurgery. Neurosurgery 83:E3–E4. https://doi.org/10.1093/neuros/nyy166

    Article  PubMed  Google Scholar 

  9. Cabitza F, Rasoini R, Gensini GF (2017) Unintended consequences of machine learning in medicine. JAMA 318:517–518. https://doi.org/10.1001/jama.2017.7797

    Article  PubMed  Google Scholar 

  10. Collins GS, Ogundimu EO, Le Manach Y (2015) Assessing calibration in an external validation study. Spine J 15:2446–2447. https://doi.org/10.1016/j.spinee.2015.06.043

    Article  PubMed  Google Scholar 

  11. Debray TPA, Vergouwe Y, Koffijberg H et al (2015) A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol 68:279–289. https://doi.org/10.1016/j.jclinepi.2014.06.018

    Article  PubMed  Google Scholar 

  12. Tetreault LA, Côté P, Kopjar B et al (2015) A clinical prediction model to assess surgical outcome in patients with cervical spondylotic myelopathy: internal and external validations using the prospective multicenter AOSpine North American and international datasets of 743 patients. Spine J 15:388–397. https://doi.org/10.1016/j.spinee.2014.12.145

    Article  PubMed  Google Scholar 

  13. Riley RD, Ensor J, Snell KIE et al (2016) External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. BMJ 353:i3140. https://doi.org/10.1136/bmj.i3140

    Article  PubMed  PubMed Central  Google Scholar 

  14. Collins GS, de Groot JA, Dutton S et al (2014) External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol 14:40. https://doi.org/10.1186/1471-2288-14-40

    Article  PubMed  PubMed Central  Google Scholar 

  15. Collins GS, Ogundimu EO, Altman DG (2016) Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Stat Med 35:214–226. https://doi.org/10.1002/sim.6787

    Article  PubMed  Google Scholar 

  16. Staartjes VE, de Wispelaere MP, Schröder ML (2019) Improving recovery after elective degenerative spine surgery: 5-year experience with an enhanced recovery after surgery (ERAS) protocol. Neurosurg Focus 46:E7. https://doi.org/10.3171/2019.1.FOCUS18646

    Article  PubMed  Google Scholar 

  17. Schröder ML, Staartjes VE (2017) Revisions for screw malposition and clinical outcomes after robot-guided lumbar fusion for spondylolisthesis. Neurosurg Focus 42:E12. https://doi.org/10.3171/2017.3.FOCUS16534

    Article  PubMed  Google Scholar 

  18. Staartjes VE, Schröder ML (2018) Effectiveness of a decision-making protocol for the surgical treatment of lumbar stenosis with grade 1 degenerative spondylolisthesis. World Neurosurg 110:e355–e361. https://doi.org/10.1016/j.wneu.2017.11.001

    Article  PubMed  Google Scholar 

  19. Collins GS, Reitsma JB, Altman DG, Moons KGM (2015) Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ 350:g7594

    Article  PubMed  Google Scholar 

  20. Staartjes VE, Siccoli A, de Wispelaere MP, Schröder ML (2018) Patient-reported outcomes unbiased by length of follow-up after lumbar degenerative spine surgery: do we need 2 years of follow-up? Spine J Off J North Am Spine Soc. https://doi.org/10.1016/j.spinee.2018.10.004

    Article  Google Scholar 

  21. Van Hooff ML, Spruit M, Fairbank JCT et al (2015) The Oswestry Disability Index (version 2.1a): validation of a Dutch language version. Spine 40:E83–E90. https://doi.org/10.1097/BRS.0000000000000683

    Article  PubMed  Google Scholar 

  22. Steyerberg EW, Vickers AJ, Cook NR et al (2010) Assessing the performance of prediction models: a framework for some traditional and novel measures. Epidemiol Camb Mass 21:128–138. https://doi.org/10.1097/EDE.0b013e3181c30fb2

    Article  Google Scholar 

  23. Hosmer DW, Lemeshow S, Sturdivant RX (2013) Assessing the fit of the model. In: Hosmer DW Jr, Lemeshow S, Sturdivant RX (eds) Applied logistic regression. Wiley, Hoboken, pp 153–225

    Chapter  Google Scholar 

  24. Brier GW (1950) Verification of forecasts expressed in terms of probability. Mon Weather Rev 78:1–3. https://doi.org/10.1175/1520-0493(1950)078%3c0001:VOFEIT%3e2.0.CO;2

  25. Van Hoorde K, Van Huffel S, Timmerman D et al (2015) A spline-based tool to assess and visualize the calibration of multiclass risk predictions. J Biomed Inform 54:283–293. https://doi.org/10.1016/j.jbi.2014.12.016

    Article  PubMed  Google Scholar 

  26. Core Team R (2018) R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna

    Google Scholar 

  27. Falavigna A, Dozza DC, Teles AR et al (2017) Current status of worldwide use of patient-reported outcome measures (PROMs) in spine care. World Neurosurg 108:328–335. https://doi.org/10.1016/j.wneu.2017.09.002

    Article  PubMed  Google Scholar 

  28. Glassman SD, Schwab F, Bridwell KH et al (2009) Do 1-year outcomes predict 2-year outcomes for adult deformity surgery? Spine J Off J North Am Spine Soc 9:317–322. https://doi.org/10.1016/j.spinee.2008.06.450

    Article  Google Scholar 

  29. van Niftrik CHB, van der Wouden F, Staartjes VE et al (2019) Machine learning algorithm identifies patients at high risk for early complications after intracranial tumor surgery: registry-based cohort study. Neurosurgery. https://doi.org/10.1093/neuros/nyz145

    Article  PubMed  Google Scholar 

  30. Durand WM, DePasse JM, Daniels AH (2018) Predictive modeling for blood transfusion after adult spinal deformity surgery: a tree-based machine learning approach. Spine 43:1058. https://doi.org/10.1097/BRS.0000000000002515

    Article  PubMed  Google Scholar 

  31. Ehlers AP, Roy SB, Khor S et al (2017) Improved risk prediction following surgery using machine learning algorithms. eGEMs. https://doi.org/10.13063/2327-9214.1278

    Article  PubMed  PubMed Central  Google Scholar 

  32. Kim JS, Merrill RK, Arvind V et al (2018) Examining the ability of artificial neural networks machine learning models to accurately predict complications following posterior lumbar spine fusion. Spine. https://doi.org/10.1097/BRS.0000000000002442

    Article  PubMed  PubMed Central  Google Scholar 

  33. van Rein EAJ, van der Sluijs R, Voskens FJ et al (2019) Development and validation of a prediction model for prehospital triage of trauma patients. JAMA Surg. https://doi.org/10.1001/jamasurg.2018.4752

    Article  PubMed  PubMed Central  Google Scholar 

  34. Janssen KJM, Moons KGM, Kalkman CJ et al (2008) Updating methods improved the performance of a clinical prediction model in new patients. J Clin Epidemiol 61:76–86. https://doi.org/10.1016/j.jclinepi.2007.04.018

    Article  CAS  PubMed  Google Scholar 

  35. Niculescu-Mizil A, Caruana R (2005) Predicting Good Probabilities with Supervised Learning. In: Proceedings of the 22nd international conference on machine learning. ACM, New York, pp 625–632

  36. Staartjes VE, Schröder ML (2018) Letter to the editor. Class imbalance in machine learning for neurosurgical outcome prediction: are our models valid? J Neurosurg Spine. https://doi.org/10.3171/2018.5.SPINE18543

    Article  PubMed  Google Scholar 

  37. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP (2002) SMOTE: synthetic minority over-sampling technique. J Artif Intell Res 16:321–357. https://doi.org/10.1613/jair.953

    Article  Google Scholar 

  38. Goldstein CL, Phillips FM, Rampersaud YR (2016) Comparative effectiveness and economic evaluations of open versus minimally invasive posterior or transforaminal lumbar interbody fusion: a systematic review. Spine 41(Suppl 8):S74–S89. https://doi.org/10.1097/BRS.0000000000001462

    Article  PubMed  Google Scholar 

  39. Goldstein CL, Macwan K, Sundararajan K, Rampersaud YR (2016) Perioperative outcomes and adverse events of minimally invasive versus open posterior lumbar fusion: meta-analysis and systematic review. J Neurosurg Spine 24:416–427. https://doi.org/10.3171/2015.2.SPINE14973

    Article  PubMed  Google Scholar 

  40. Schröder ML, de Wispelaere MP, Staartjes VE (2018) Are patient-reported outcome measures biased by method of follow-up? Evaluating paper-based and digital follow-up after lumbar fusion surgery. Spine J Off J North Am Spine Soc 2:2. https://doi.org/10.1016/j.spinee.2018.05.002

    Article  Google Scholar 

Download references

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Author information

Authors and Affiliations

  1. Center for Neuroscience, Queens University, Kingston, ON, Canada

    Ayesha Quddusi

  2. Department of Neurosurgery, Bergman Clinics, Naarden, Rijksweg 69, 1411 GE, Naarden, Amsterdam, The Netherlands

    Hubert A. J. Eversdijk, Anita M. Klukowska, Marc L. Schröder & Victor E. Staartjes

  3. School of Medicine, University of Nottingham, Nottingham, UK

    Anita M. Klukowska

  4. Department of Clinical Informatics, Bergman Clinics, Amsterdam, The Netherlands

    Marlies P. de Wispelaere

  5. Department of Neurosurgery, Faculty of Medicine, RWTH Aachen University, Aachen, Germany

    Julius M. Kernbach

  6. Amsterdam UMC, Neurosurgery, Amsterdam Movement Sciences, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands

    Victor E. Staartjes

  7. Machine Intelligence in Clinical Neuroscience Lab, Department of Neurosurgery, University Hospital Zurich, Clinical Neuroscience Centre, University of Zurich, Zurich, Switzerland

    Victor E. Staartjes

Authors
  1. Ayesha Quddusi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hubert A. J. Eversdijk
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Anita M. Klukowska
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Marlies P. de Wispelaere
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Julius M. Kernbach
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Marc L. Schröder
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Victor E. Staartjes
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Victor E. Staartjes.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

586_2019_6189_MOESM1_ESM.pptx

Supplement 1 Statistical Code. The code was executed in R Version 3.5.4 (The R Foundation for Statistical Computing, Vienna, Austria) on a machine running Windows 10 (Microsoft Corp., Redmond, WA, USA). (PPTX 1304 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Quddusi, A., Eversdijk, H.A.J., Klukowska, A.M. et al. External validation of a prediction model for pain and functional outcome after elective lumbar spinal fusion. Eur Spine J 29, 374–383 (2020). https://doi.org/10.1007/s00586-019-06189-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00586-019-06189-6

Keywords

Search

Navigation