Letter to the Editor: Rasch Analysis and High Value Spinal Endoscopy—Another Perspective

There are lies, damn lies, and statistics.

—Benjamin Disraeli (as attributed by Mark Twain)

To the Editor: Given this provocative introductory quote, I feel that a series of disclaimers is appropriate before leaping into a review of this special issue, “Perspectives on High-Value Endoscopic Surgeries.” First, I am not a statistician. I am a spine surgeon with a fair experience in literature review and medical education, including the teaching and evaluation of neurosurgery residents, medical students, and spine faculty. Second, I have, in the past, been accused of being in general both a skeptic and a Luddite. While I can neither confirm nor deny those charges, I suspect them to be true. Third, I have no agenda whatsoever regarding the relative value of endoscopic techniques vs other techniques in spine surgery. While endoscopic techniques are commonly used at my institution by several of my partners, I personally have no clinical experience with these techniques. With these disclaimers, I offer the following comments.

The authors have presented a series of articles describing the results of a 4-part webinar series dedicated to educating surgeons on the value and safety of endoscopic spine techniques. The series of webinars was sponsored by a manufacturer of endoscopes, and the authors used a survey methodology to assess surgeons’ attitudes regarding the use of endoscopic techniques for a variety of indications. They surveyed participants before and after the webinars to see what sort of impact the webinars had on the participant’s attitudes. To interpret the results of the surveys, the authors used a technique called Rasch analysis.

A Rasch analysis is a psychometric tool used to more correctly interpret the results of tests and surveys.1 Despite my lack of experience in Rasch analysis, I believe most spine surgeons with research experience would agree that this technique allows for the users to influence metrics values by rank ordering them before analysis such that items may vary in terms of difficulty or when values on an ordinal scale are not actually ordinal. The American Board of Neurological Surgeons, for example, has used another technique for the evaluation of the written examination such as secondarily taking into account grading biases.

An example of an ordinal scale that is not really ordinal can be found in the American Board of Neurological Surgeons’ oral examination. To paraphrase, examiners are instructed to grade candidates on a 4-point scale in which a score of 1 indicates unsafe practice, a score of 2 indicates marginal practice, a score of 3 indicates safe if not ideal practice, and a score of 4 indicates exemplary practice. It is usually the case that the assignment of more than 1 or 2 “2’s” in a session is likely to lead to failure of the candidate. Therefore, in practice, the gap between a 2 and a 3 is a much more important gap than that between a 3 and a 4 because examiners are generally reluctant to cause a candidate to fail the examination. Therefore, it takes a greater discrepancy in practice safety to drop a candidate’s score from a 3 to a 2 than from a 4 to a 3.

Another area where the Rasch analysis has been applied in neurosurgery is in the interpretation of surveys used to establish the relative difficulty of different procedures for billing purposes. Richard Florin, one of our neurosurgical representatives to the American Medical Association’s Relative Value Update Committee, used the technique to help describe the relative difficulty and work required between different techniques for the purposes of establishing reimbursement standards in a rational way (www.rasch.org, accessed on 27 September 2024). An advantage of this analysis is that similar procedures can be analyzed simultaneously and rank-ordered by perceived difficulty and risk. Dr Florin was able to demonstrate that several codes were either under- or overvalued compared with related procedures.

In the series of articles published in this special issue, the authors surveyed participants prior to and after the webinar series to assess shifts in attitudes related to the content presented. There are some methodological concerns regarding who filled out the surveys. For example, in the first webinar, 1311 surgeons participated, and only 42 submitted completed surveys. Are these 42 surgeons truly representative of the larger group? Is a 3.2% participation rate adequate to make any conclusions? Similar but less dramatic drop-offs were seen in the other three webinars with response rates of 9.1%, 15%, and 13.5%, respectively. I am in no position to critique the statistical analysis beyond this concern as the performance of the analysis is outside my field of expertise. The authors present their results, which appear to indicate a shift toward a more favorable attitude toward endoscopic techniques after participants were exposed to the content of the webinars.

This finding is not surprising. Those participating in the webinars participated because they were curious about the new technology. Presenters are known developers and advocates for these techniques and most likely would have been positive with their messaging. As a personal anecdote, I recently visited Portugal and was invited to go surfing. I had never been surfing and, being in my upper 50s, had some significant reservations about potential injury. That said, I was curious and attended a lesson. This lesson shifted my attitude toward surfing to such an extent that I went ahead and gave it a try. If the authors had concluded that exposure to an educational/promotional webinar improved participant knowledge and attitudes regarding endoscopic techniques, I would endorse this conclusion (although I would likely question the rationale for publication). However, the authors conclude in Part 3 that their findings should drive “future clinical guidelines and training programs to align with evolving endoscopic techniques,” and in the final summary article that

• “The [International Society for the Advancement of Spine Surgery] webinar series has significantly impacted surgeons’ education and contributed to the identification of high-value endoscopic spine surgery practices that may serve as a cornerstone for surgeon training standards, policy, and guidelines development. Ongoing research on technological advancements and expansions of clinical indications combined with systematic review is expected to refine the recommendations on high-value endoscopic spinal surgeries recommended for enhanced reimbursement.”

These conclusions are not supported or entirely validated by the data supplied. The lofty, aspirational statements are subjective, and the reader should consider them as such.

For the record, despite my attitude shift, I sucked at surfing.

• This manuscript is generously published free of charge by ISASS, the International Society for the Advancement of Spine Surgery. Copyright © 2024 ISASS. To see more or order reprints or permissions, see http://ijssurgery.com.