Measurement Performance of a Computer Assisted Vertebral Motion Analysis System
===============================================================================

* Reginald J. Davis
* David C. Lee
* Chip Wade
* Boyle Cheng

## Abstract

**Background** Segmental instability of the lumbar spine is a significant cost within the US health care system; however current thresholds for indication of radiographic instability are not well defined.

**Purpose** To determine the performance measurements of sagittal lumbar intervertebral measurements using computerassisted measurements of the lumbar spine using motion sequences from a video-fluoroscopic technique.

**Study design** Sensitivity, specificity, predictive values, prevalence, and test-retest reliability evaluation of digitized manual versus computer-assisted measurements of the lumbar spine.

**Patient sample** A total of 2239 intervertebral levels from 509 symptomatic patients, and 287 intervertebral levels from 73 asymptomatic participants were retrospectively evaluated.

**Outcome measures** Specificity, sensitivity, negative predictive value (NPV), diagnostic accuracy, and prevalence between the two measurement techniques; Measurements of Coefficient of repeatability (CR), limits of agreement (LOA), intraclass correlation coefficient (ICC; type 3,1), and standard error of measurement for both measurement techniques.

**Methods** Asymptomatic individuals and symptomatic patients were all evaluated using both the Vertebral Motion Analysis (VMA) system and fluoroscopic flexion extension static radiographs (FE). The analysis was compared to known thresholds of 15% intervertebral translation (IVT, equivalent to 5.3mm assuming a 35mm vertebral body depth) and 25° intervertebral rotation (IVR).

**Results** The VMA measurements demonstrated greater specificity, % change in sensitivity, NPV, prevalence, and reliability compared with FE for radiographic evidence of instability. Specificity was 99.4% and 99.1% in the VMA compared to 98.3% and 98.2% in the FE for IVR and IVT, respectively. Sensitivity in this study was 41.2% and 44.6% greater in the VMA compared to the FE for IVR and IVT, respectively. NPV was 91% and 88% in the VMA compared to 62% and 66% in the FE for IVR and IVT, respectively. Prevalence was 12.3% and 11.9% for the VMA compared to 6.1% and 5.4% for the FE in IVR and IVT, respectively. Intra-observer IVR and IVT had a CR of 2.49 and 2.62, respectively. Inter-observer IVR and IVT had a CR of 1.99 and 2.81, respectively. Intra-subject (test/retest) CR were 2.49 and 3.11 for IVR and IVT, respectively.

**Conclusions** The VMA system showed greater measurement performance in the detection of radiographic instability compared with FE radiographs.

*   Lumbar Instability
*   Flexion/Extension Radiographs
*   Motion Analysis

## Introduction

Segmental instability of the lumbar spine is suggested to be a major cause of low back pain and a significant cost within the US health care system.1–3 Segmental instability of the spine has been studied *in vivo* with radiographic methods since 1944 yet continues to be controversial in its determination.4 Over the years several quantitative definitions of lumbar instability based on radiograph-derived measurements have been proposed,5–8 however current radiographic criteria for instability, and consequential surgical and non-surgical patient selection criteria, are not well defined.

As recent May 2014 the North American Spine Society (NASS) taskforce for developing treatment guidelines for lumbar fusion posited that “there are no accepted radiologic criteria by which the change in alignment on flexion-extension views can be considered instability”. Additionally the taskforce suggest that the commonly-used criteria for clinical instability were developed in order to assist physicians in evaluating traumatic instability using static radiographs and were not intended for the determination of clinical instability in the degenerative population.9 

Common coverage guidelines for the coverage of fusion surgery for the treatment of lumbar intervertebral instability (InterQual, McKesson, San Francisco, CA, USA) advocate intervertebral rotation (IVR) larger than 22° and intervertebral translation (IVT) greater than 3 mm as criteria for fusion. The American Medical Association (AMA) proposes a range of 4.5–5 mm or 10–15% of the vertebra body width in IVT, or IVR greater than 15° at L1-2, L2-3 or L3-4, 20° at L4-5 or 25° at L5-S1. As it relates to postoperative pseudarthrosis, the US Food and Drug Administration (FDA) indicates IVR of less than 5° and IVT less than 3 mm as a successful fusion.10 Given the wide range of proposed quantified definitions of radiographic instability, a better understanding of spinal motion characteristics through standardized, accurate, and reliable functional analysis is clearly necessary.

While there is not a consensus on the quantification of radiographic instability, the diagnostic imaging modalities of standard roentgenograms (X-rays), computed tomography (CT), and magnetic resonance imaging (MRI) have widely been accepted in the evaluation of spinal pathologies. In evaluating spinal instability, dynamic end-range flexion/extension radiographs are employed to measure intervertebral range of motion (ROM), comprised of intervertebral rotation and translation (IVR and IVT), through manual identification of vertebral segments across multiple radiographic images using a ruler and protractor or computer-assisted imaging software. In recent years, there have been marked improvements to imaging modalities such as MRI and CT in capturing diagnostically-useful static images of the spine, yet these lack the capability of providing detailed information about spinal motion.

Recently, there have been improvements to the measurement evaluation techniques used for interpreting the flexion/extension radiographs (FE). Traditional manual measurements are performed by manually drawing lines or overlaying protractors on film radiographs. 11, 12 Computer-assisted manual measurements are performed manually by the user by drawing lines or identifying landmarks in digital radiographs using computer software which utilizes this user input to derive measurements of ROM.13–15 In general, the precision and reliability of measurements are improved when moving towards computerized measurements, as they standardize and minimize user interaction and therefore result in less measurement variability. 16–19 

The variability associated with the process for deriving measurements of ROM from radiographs is therefore well understood, and at least partiallyaddressable through computerized measurement techniques. In contrast, the variability associated with patient specific bending in FE radiographs is less detailed in the literature. FE methods can be characterized as uncontrolled, non-standardized across patients, and involving bending at a patientselected rate and range.20 There is an inherent level of variability involved when patients are unconstrained in how they bend, both across patients and within a patient over time. In contrast, it has been proposed that FE methods that involve controlled patient bending via the use of a handling device that assists patients through a standard arc of lumbar bending have the potential to reduce variability in patient bending during FE.21–24 Concerns about this high level of variability have driven the various approaches intended to control bending and/or ensure consistent patient effort. Esola et al.22 showed that there are measureable differences in pelvic rotation during flexion in healthy subjects compared with patients with back pain. The findings of Esola et al. (1996) support the use of a handling device for pelvic fixation, such as that used by Takayanagi et al. (2001), to avoid unintended confounding of measurements from varying pelvic motion.

Since the 1940's, bending motions for radiographic imaging has been categorized based on the bending posture (i.e. standing, seated, etc.) as well as the bending method (i.e. maximum voluntary bending, assisted bending, etc.) (Table 1). Researchers and clinicians have utilized a number of postures and methods to elicit segmental motion during imaging with varying results. These postures include a standing uncontrolled posture, a posture with hips flexed or seated, standing while the pelvis is fixed, lying on one's side, and traction and compression; whereas the methods of the bending include maximum voluntary flexion/extension, the patient pulling oneself through a ROM, a clinician pulling the patient through a ROM or a given standard bending range. Wood et al. (1994) acknowledged that studies may exaggerate stress on the lumbar spine beyond that expected under physiologic conditions in an attempt to maximize the chances of detecting potentially abnormal translational movement. 

View this table:
[Table 1](http://ijssurgery.com//content/9/36/T1)

Table 1 
Bending posture and bending method classifications for classifying studies reviewed as part of the review of medical literature.

There is a breadth of research that describes standing uncontrolled bending protocols that have the patient standing upright and unrestrained, and attempting to bend only at the lumbar spine while keeping the hips from flexing.10, 26–30 In standing upright, the knees are kept in a fully extended position. It is possible for hip flexion to occur, but through only patient-voluntary efforts this can be minimized. Standing uncontrolled bending may be further classified as involving either: (I) maximum voluntary end range angles, or (IV) standardized end range angles.

Hips flexed bending or seated protocols accomplish very large ranges of bending through combined hip and lumbar flexion, which may additionally involve flexion of the knees. They can be classified as: (I) having no pulling forces, (II) having pulling forces generated by the patient, or (III) having pulling forces generated by an imaging technician. Research has described these postures as being performed while sitting off the end of a tables edge,22, 31 while sitting in a chair and pulling oneself to a maximum range,32 and while standing upright and being pulled or pushed in extension or flexion, respectively by a radiology technician.24 

Bolstering the pelvis while the patient stands upright keeps the hips in full extension and the pelvis upright. This type of bending is utilized to elicit a lumbar motion while reducing pelvic girdle and hip motion during bending.22 For the hips to flex, the pelvis has to rotate forward during bending. Using a bolster minimizes pelvic rotation (and therefore hip flexion), which maximizes the amount of trunk bending that derives from the lumbar joints (as opposed to the hips). Bolsters may involve padded devices or harnesses. Bronfort (1994) and Pearcy (1985) had subjects stand in an apparatus which limited the movement of the pelvis and hip. Okawa (1988) supported the patient based on an apparatus at the pubis and sacrum while bending; whereas Teyhen (2006) minimized pelvic and hip motion by restricting the subject in a “mountain climbing” type harness to restrict motion.

Side-lying flexion / extension has been reported in several studies as well.28, 30, 36 Each of these studies examined flexion and extension in the recumbent position at maximum flexion extension points. Lying lumbar imaging, in general, is commonly used in current US medical practice (not necessarily with bending views) to determine the extent of spondylolisthesis reduction occurring between standing and lying lateral views.

Traction / compression was studied and reported in the 1980s and 1990s.37–39 In the tension method patients hung from their hands from a bar perpendicular to the film plane, while compression was elicited by an upright posture while wearing a backpack full of 20kg of sand. This method found in increase in the rate of radiographic instability of 4.8%, however it never became widely adopted and there were no references to any studies involving traction/compression after the Pitkanen (1997) study.

Given the various methodologies of bending and associated postures during lumbar imaging, the evidence suggests a challenge in determining applicable methodologies for accurate lumbar imaging. Anecdotal evidence suggest many of the aforementioned methodologies do not fall within the standard clinical workflow or accepted modalities which are widely used today. More importantly these modalities may present a challenge to the clinician and patient to perform based on either logistical or safety concerns from the authors experience. To overcome the high measurement variability experienced in the FE imaging modality, other imaging methodologies have been proposed.

Non-quantitative dynamic fluoroscopy methods have been proposed for the dynamic visualization of segmental motion; however to the author's knowledge no studies examining accuracy and repeatability of these methods have been published. With respect to quantified approaches for assessing ROM, research has examined accuracy and reliability of quantitative fluoroscopy (QF) and digital video-fluoroscopy (DVF) and has demonstrated repeatability comparable to that of standard computer-assisted X-ray techniques. 24, 40, 41 Several additional studies have evaluated intra- and inter-observer reliability and find that computer-assisted processing methods significantly improve intervertebral motion measurements.10, 42 However there is a dearth of literature examining key aspects of the diagnostic efficacy of the clinical application of quantitative fluoroscopy (QF), including its diagnostic sensitivity and specificity in assessing radiographic instability as well as the test/retest (intrasubject) repeatability of ROM measurements produced via QF.

When assessing clinical efficacy, diagnostic tests should be reliable, accurate, and beneficial.43 By definition a test is considered reliable if the identical test conducted on the same patient continuously derives then same outcome.44 A test is accurate if it results in the correct outcome in identifying pathology when pathology is present. Sensitivity and specificity are the fundamental measures of accuracy. At present, lumbar spinal instability is diagnosed on the basis of findings on physical exam plus FE radiographs, but there is no consensus regarding the radiographic criteria for instability, and the clinical definition of lumbar spinal instability is therefore ambiguous.8, 9 Although many studies have described clinical examination measures for the diagnosis of lumbar spinal instability, few of them have investigated the sensitivity and specificity of using FE to detect radiographic instability. Fritz evaluated patients with low-back pain referred for FE radiographs due to suspicion of lumbar instability and reported lumbar flexion ROM measures were a significant predictor of instability and that lumbar intervertebral motion testing was beneficial for predicting instability.45 

Despite the common acceptance of FE radiographs as a key clinical indicator of radiographic instability, the role of FE radiographs remains debated due to several important limitations including poor diagnostic accuracy and reliability.46 In a recent study, Yeagar et al.10 demonstrated that reliability of intervertebral measurements by the Vertebral Motion Analysis (VMA) (Ortho Kinematics Austin, TX, USA) was significantly greater when compared with a digitized manual technique. This study assesses the reliability of the computer-assisted vertebral motion analysis system in comparison to FE through intra-and interobserver reliability, and intra-subject reliability. This study further assesses the diagnostic sensitivity, specificity, prevalence, and NPV of the computerassisted vertebral motion analysis system in comparison to FE, in detecting radiographic instability. All assessments in this study were produced using a prospective analysis of collected image data.

## Materials and Methods

### Image acquisition

The VMA system assesses intervertebral motion through the use of: (1) a patient handling device that assists patients through a prescribe-able arc of lumbar bending in both an upright (Figure 1) and recumbent (Figure 2) posture (flexion/extension arcs were 70 and 60 degrees as measured by the rotation of the VMA for upright and recumbent, respectively); (2) video fluoroscopy imaging of the lumbar spine during bending (capturing images at 8 frames per second) utilizing a standard 12-inch surgical C-Arm (OEC 9800 Radiographs, General Electric, Fairfield, CT, USA; 2006 Phillips BV Pulsera, Andover, MA, USA); (3) console mounted computer equipped with data acquisition hardware (Accustream Express As205A, Foresight Imaging, Chelmsford, MA, USA) to digitize fluoroscopic video signals and (4) proprietary image processing software capable of assisting the user in the semi-automatic frame-to-frame registration and tracking of vertebral bodies across the sequence of video-fluoroscopic images to derive measurements of intervertebral rotation and translation (Ortho Kinematics, Austin, TX, USA). Previous research has reported the accuracy and repeatability of the VMA system.10, 24 

![Fig. 1](http://ijssurgery.com//http://www.ijssurgery.com/content/ijss/9/36/F1.medium.gif)

[Fig. 1](http://ijssurgery.com//content/9/36/F1)

Fig. 1 
VMA standing flexion/extension position.

![Fig. 2](http://ijssurgery.com//http://www.ijssurgery.com/content/ijss/9/36/F2.medium.gif)

[Fig. 2](http://ijssurgery.com//content/9/36/F2)

Fig. 2 
VMA recumbent flexion/extension position.

The standardized imaging protocol consist of a standard FE radiographs of uncontrolled bending under fluoroscopy (Figure 3) and VMA radiographs captured during controlled bending with the patient positioned in the patient handling device in both the upright and recumbent configuration (Figure 1, Figure 2). The upright motion platform “guided” active lumbar bending, under normal physiologic weight-bearing conditions, while constricting flexion extension to the sagittal plane and bolstering the pelvis to minimize pelvic motion and maximize lumbar bending. The individual follows the VMA rotation on their own using the VMA as a proprioceptive guide. The recumbent platform “controlled” passive lumbar bending, which minimized the gravitational and muscular forces that present during standing radiographs. The individual lies on the VMA table and the table, through a servo motor driven actuator, actively moves the individual through the flexion extension motion. Flexion/extension angles spanned a range of 70o arc for standing and recumbent motion. The flexion/extension ranging was preset on the VMA hardware to allow for 20° of extension and 50° flexion for upright and 35° flexion and 25° of extension in recumbent of VMA motion. This compensated for the reduced capability of lumbar extension resulting from extended hips in an erect posture.

![Fig. 3](http://ijssurgery.com//http://www.ijssurgery.com/content/ijss/9/36/F3.medium.gif)

[Fig. 3](http://ijssurgery.com//content/9/36/F3)

Fig. 3 
Flexion extension radiographs (FE) at a self-selected pace and position.

Upon initiation of the test movement, a fluoroscopic sequence of lumbar motion was captured at 8 pulses per second on high level flouro (i.e boost) and 4 frames of data per second were analyzed. Flexion and extension were captured as separate sequences, which began in a neutral position, progressed to the predefined maximum angle, and returned to neutral. In the upright position, a total of 28 seconds of data were collected (neutral to flexion-10 seconds, flexion to neutral-10 seconds, neutral to extension-4, extension to neutral-4) for a total of 112 frames of data. In the recumbent position, a total of 24 seconds of data were collected (neutral to flexion-7 seconds, flexion to neutral-7 seconds, neutral to extension-5, extension to neutral-5) for a total of 96 frames of data.

### Experimental Design

This study combines image data from 4 separate neurosurgery facilities that utilized the VMA system. Five hundred and nine (509) symptomatic patients (44% female, 56% male; mean age, 53.2 ± 17.8 years) contributed to 2,239 intervertebral levels. One hundred and sixty six (166) asymptomatic individuals (43% female, 57% male; mean age, 48.1 ± 15.4 years) contributed to 705 intervertebral levels. All individuals completed all imaging sequences in a single visit. Image processing software generated IVR and IVT measurements. In the case of FE measurements, this was done via the manual land-marking of an image pair and the subsequent calculation of IVR and IVT by the software. In the case of VMA measurements, this was done via an automated tracking process synchronized to the fluoroscopic sequences.10, 24 The IVR were defined as the change in angle between the inferior endplate of the superior vertebra and the superior endplate of the inferior vertebra in the motion segment, while the IVT were expressed as a percentage of inferior vertebral body depth.10 

The symptomatic patient set was collected via a consecutive case enrollment design, wherein all patients seen in the surgeon's clinic and prescribed VMA testing over a specific period of time were included. The exact timeframes varied by clinic, but overall the consecutive case series data were collected between January, 2011 and February, 2014. It is important to delineate the VMA as the “index test” and the FE radiographs as a commonly accepted “reference test”. Problems with the reference standard based on a “reference test “ are evident in diagnostic accuracy studies. The outcome of the reference standard may not be available in all patients, it may be unreliable, it may be inaccurate or there could be no acceptable reference standard at all.47 Lord et al. (2006) reported that measurements of specificity of a new diagnostic test through a cross-sectional descriptive study design are satisfactory evidence to recommend substitution of the reference test only when the new test has a sensitivity similar to that of the reference test and when the new test is safer or is more specific than the reference test.48 

Multiple data sets were constructed to assess reliability and accuracy of the VMA system. One distinct data set (Table 2) was constructed based on the complete sample of asymptomatic individuals and symptomatic patients to evaluate the system specificity, % change in sensitivity, negative predictive value (NPV), and the prevalence of radiographic instability while a second data set (Table 3) on only asymptomatic individuals was constructed to evaluate inter and intra observer agreement and intra subject (test/ retest) agreement. 

View this table:
[Table 2](http://ijssurgery.com//content/9/36/T2)

Table 2 
Average angles and translations measured for both VMA and FE imaging radiographs (N = 2239 intervertebral levels).

View this table:
[Table 3](http://ijssurgery.com//content/9/36/T3)

Table 3 
Contributions of intervertebral levels comprising the dataset.

## Statistical Methods

### Specificity, Prevalence, and NPV

A standard methodology was employed to calculate specificity, NPV, and the prevalence of radiographic instability. In evaluating the diagnostic test performance every individual being tested either has or does not have radiographic instability. Overall the diagnostic test may or may not match the individual's actual status resulting in a true positive (TP), false positive (FP), true negative (TN), or false negative (FN); the current analysis focuses on TP, FP, and TN results. Individuals in the current analysis were either symptomatic or asymptomatic. Individuals in the current analysis were classified by both the VMA and FE tests for radiographic instability based on measurements of IVR and IVT. For IVR, the thresholds used to define radiographic instability (excessive rotation) were 22°. For IVT, the thresholds used to define radiographic instability (excessive translation) were 15% of the vertebra body width, which corresponds to a threshold of roughly 5.3 mm of IVT assuming a standard vertebral body depth of 35 mm. For all analyses presented herein, individuals within the symptomatic patient population were categorized as TP whenever there was evidence of radiographic instability according to the thresholds listed above. Similarly, subjects from within the asymptomatic patient population were categorized as FP when IVR or IVT was observed to be above these thresholds.

### Inter-observer, Intra-observer, Intra-subject (test/retest) Reliability

Inter- and intra-observer agreement and intrasubject agreement was evaluated using the Bland-Altman method. The differences between each observer's first and second round measurements (A1–A2, B1–B2) for each intervertebral level were used to establish the intra-observer agreement. The differences between each pair of observer's first round measurements (A1–B1, A2–B2,) were used to establish the inter-observer agreement. Measurements were conducted with 1 week between measurements. Intra-subject (test/retest) agreement was measured by a single observer evaluating measurements (At1-At2) taken from 2 separate imaging sessions performed on the same subject 6 weeks apart. The mean (d) and standard deviation (SD) of each set of differences were calculated. The coefficient of repeatability (CR), upper and lower limits of agreement (LOA), and corresponding standard error of measurement (SEM) were defined as follows: Calculations were performed using SAS 9.3 (version 19; SAS Institutes, Cary, NC, USA).

## Results

Reporting on the ROM values from the VMA compared to the FE radiographs, Table 2 and Table 3 present summary data and performance data respectively. Table 2 presents the average IVR and IVT by level for FE, standing VMA and lying VMA for 2,239 intervertebral levels separately between the VMA and the FE radiographs. A student t test univariate analysis was conducted to show statistical differences across modes and level. Table 3 provides a level by level comparison which constituted the complete data set, additionally the contribution of each level to the overall motion of the lumbar body.

Table 4 details the accuracy measurements of specificity and NPV between the VMA and the FE radiographs for IVR and IVT measurements in detecting radiographic instability, and also provides the prevalence of radiographic instability within the symptomatic population as detected by both VMA and FE radiographs. Specificity was 0.3% (VMA 99.4%, FE 98.3%) and 0.9% (VMA 99.1%, FE 98.2%) greater in the VMA compared to the FE in intervertebral rotation and translation, respectively. Further, data indicates a 25% and 26% greater NPV in the VMA compared to the FE radiographs in IVR (VMA 91%, FE 66%) and IVT (VMA 88%, FE 62%) respectively. 

View this table:
[Table 4](http://ijssurgery.com//content/9/36/T4)

Table 4 
Specificity and NPV of the VMA and FE lumbar imaging radiographs in detecting radiographic instability across levels, and the prevalence of VMA- and FE-detectable radiographic instability.

Percent increases or decreases to the prevalence of radiographic instability as defined above of VMA relative to FE can be shown to be exactly equivalent to the magnitude of percent increases or decreases to the sensitivity of VMA relative to FE. This can be demonstrated algebraically. Sensitivity is defined as TP / (TP + FN), and prevalence of radiographic instability is defined as TP / (TP + FN + CNsymptomatic), where CNsymptomatic is defined as the number of “Condition Negative” patients within the symptomatic patient population (i.e. those who have pain but do not have radiographic instability). The sum (TP + FN) is equivalent to the total number of “condition positive” patients within the symptomatic patient population, which along with CNsymptomatic, were constant in the patient population and therefore do not vary between VMA and FE, even though their sizes were not directly measurable in this study. Any percent increase or decrease in the prevalence of radiographic instability for VMA relative to FE is shown to be exactly equivalent to the percent increase or decrease in sensitivity for VMA relative to FE, as both are defined by the same variable quantity (TP) divided by a constant. As such, sensitivity in this study was 41.2% and 44.6% greater in the VMA compared to the FE in intervertebral rotation and translation, respectively. Although many studies have described clinical examination measures for the diagnosis of lumbar spinal instability, few have investigated the sensitivity and specificity of these examinations in detecting radiographic instability.

The reliability measurement comparison of the VMA to the FE radiographs is presented in Table 5 and Table 6. Table 5 presents the average IVR and IVT by level for FE, standing VMA and lying VMA for 705 (inter- and intra-observer agreement) intervertebral levels and the 287 (intra-subject agreement) intervertebral levels, respectively, separately between the VMA and the FE radiographs. A student t test univariate analysis was conducted to show statistical differences across modes and level. Table 6 provides a level by level comparison of the inter- and intraobserver dataset (n = 705 levels) as well as the intrasubject dataset (n = 287 levels). 

View this table:
[Table 5](http://ijssurgery.com//content/9/36/T5)

Table 5 
Average angles and translations measured for both VMA and FE imaging radiographs in the agreement dataset.

View this table:
[Table 6](http://ijssurgery.com//content/9/36/T6)

Table 6 
Contributions of intervertebral levels comprising the agreement dataset.

Table 7 presents the summary statistics for inter- and intra-observer agreement and intra-subject agreement VMA measurement. Inter- and intra-observer variability of the VMA is comparable to that present by Yeagar et al.10. The inter observer CR of IVR and IVT is reports as 2.49 and 2.62 respectively, while the ICC was 0.974 and 0.926 respectively. Intra observer CR of IVR and IVT is reports as 1.99 and 2.81 respectively, while the ICC was 0.939 and 0.946 respectively. To our knowledge this study is the first to report intra-subject variability on a computer assisted vertebral motion analysis system. The intra-subject CR of IVR and IVT are reported as 2.49 and 2.57 respectively, while the ICC was 0.919 and 0.923 respectively. 

View this table:
[Table 7](http://ijssurgery.com//content/9/36/T7)

Table 7 
Intra and inter observer variability and intra subject variability in VMA lumbar imaging radiographs.

## Discussion

The current study demonstrates improved performance of the VMA radiographs compared to the standard FE radiographs in measurement performance. While there have been significant improvements to imaging modalities and imaging analysis a lack of consensus around segmental and clinical instability still exist. Panjabi (2003) suggested that the ambiguity in consistent definition of clinical instability partially a result of increased variability in the patient bending and the limited accuracy of *in vivo* methods for measuring motion. In this study, the VMA system addresses the first two primary concerns by providing a controlled patient bending ROM and accuracy inherent to the automatic frameto-frame registration and tracking algorithm. The current study provides evidence that there is a marked improvement in measurement performance of the VMA compared to the FE radiographs while also exhibiting statistically greater reliability. This is especially notable because both modalities (VMA and FE) had relatively high specificity, and the specificity for VMA was if anything slightly higher than FE. The fact that there were increase in sensitivity with no corresponding decrease in specificity demonstrates that VMA testing detects radiographic instability to the same extent as that detected with FE.

Diagnostic misclassification errors have received increasing attention as these errors constitute significant sources of liability claims as well as increased risk of morbidity and mortality.29, 48 Misclassification errors resulting from the methodology and reporting of radiological examinations are common place across modalities.49 These errors can originate in multiple ways from testing to reporting. Misclassification errors can result from incomplete or inappropriate evaluation of medical history, physical examinations, or diagnostic testing; interpretation of diagnostic test results; or therapeutic or surgical decisions based on diagnostic test results.49, 50 The accuracy and test/retest reliability of diagnostic test can be a principal confounding factor is diagnostic error. The current study shows that compared to the FE radiographs the VMA is 31% to 35% more efficient in identifying radiologic instability, based on the current thresholds.

The reliability of the VMA system has been reported by Yeagar et al.10 who evaluated inter- and intraobserver reliability between the VMA and digital manual measurements. Their study demonstrates that marked improvements in intra- and interobserver precision are achieved by VMA analysis compared with a digitized manual technique. Further, Yeagar et al.10 suggests that a physician making repeated measurements on the same image (intra observer) can be 95% confident they will fall within a range of ±1.53° and ±2.20% of the actual value in IVR and IVT respectively. Further they found that different physicians making a repeated measurement on the same image (inter-observer), the CR 2.15° and 3.90%, respectively. The current study extended the Yeagar et al.10 analysis be volume of vertebral levels analyzed as well as by evaluating intra-subject reliability.

In comparing the current study's CR to that of Yeagar et al.10 the results were quantitatively similar. The current findings indicate inter-observer reliability (95% confident) falls within a range of ±2.49° and ±2.62% of the actual value in IVR and IVT respectively. Further the current study reports a quantitatively similar inter-observer reliability CR of 1.99° and 2.81% in IVR and IVT, respectively. To our knowledge the current study is the first to evaluate intra-subject reliability in a computer assisted vertebral motional analysis system. The current findings extend those of Yeagar et al.10 and provide evidence of the test/retest reliability of the VMA system.

## Conclusion

Many clinical tests are used to confirm or refute the presence of a disease or further the diagnostic process. A primary clinical test for detecting spinal instability is the dynamic radiograph imaging modality and have been so for nearly a decade. However, there still seems to be a lack of consistency in the utilization of dynamic radiographs specifically in determining an applicable standard threshold for rotational and translational radiographic instability in the lumbar spine. The current technique (i.e. VMA) has the potential to provide greater measurement performance by utilizing the combination of controlled motion and a semi-automated measurement technique to enhance the clinician's ability to identify radiographic instability of the lumbar spine.

Ideally diagnostic tests correctly identify all patients with the disease, and similarly correctly identify all patients who are disease free. Ideally, a perfect test would never be positive in a patient who is disease free and would never be negative in a patient who is in fact diseased. Even though it remains controversial, FE radiographs are a principal diagnostic modality for identifying radiographic instability of the lumbar spine. More importantly the quantitative ROM thresholds for detecting radiographic instability have been debated for half a century with little consensus. Unfortunately there have been few advances in assessing for radiographic instability of the lumbar spine in recent years. Through computer assisted vertebral motion analysis as well as the use of patient bending control devices and fluoroscopy, the ability improve upon current methods for detecting radiographic instability is clinically relevant. The current study demonstrates the improved measurement performance of the VMA system as a diagnostic modality in assessing for and detecting radiographic instability of the spine both in rotation and translation.

## Disclosures

Chip Wade is a consultant for and owns stock in Orthokinematics; David Lee is a consultant for and sits on the Scientific Advisory Board for Orthokinematics. All other authors have no relevant financial disclosures.

## Acknowledgements

The authors would like to acknowledge the participation of the following physicians as contributing investigators in collecting asymptomatic individual data: Morgan Lorio, Neurospine Solutions, Bristol, TN; Hyun Bae, The Spine Institute Comprehensive Spine Center, Santa Monica, CA; Antonio Castellvi, Florida Orthopedics Institute, Tampa, FL; and David Harris, Center for Healing and Regenerative Medicine, Austin, TX. Additionally, we acknowledge Will Malpica and Kristin Madl of Ortho Kinematics in assistance with data acquisition.

*   Copyright © 2015 ISASS - This manuscript is generously published free of charge by ISASS, the International Society for the Advancement of Spine Surgery

## References

1.  Dagenais S, Galloway EK, Roffey DM (2014) A systematic review of diagnostic imaging use for low back pain in the United States. The Spine Journal 14:1036–1048.
    
    

2.  Haldeman S, Dagenais S (2008) A supermarket approach to the evidence- informed management of chronic low back pain. Spine J 8:1–7.
    
    [CrossRef](http://ijssurgery.com//lookup/external-ref?access_num=10.1016/j.spinee.2007.10.009&link_type=DOI) 
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=18164448&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 
    
    [Web of Science](http://ijssurgery.com//lookup/external-ref?access_num=000252601300001&link_type=ISI) 

3.  Deyo RA, Mirza SK, Martin BI (2006) Back pain prevalence and visit rates: estimates from U.S. national surveys, 2002. Spine 31:2724–7.
    
    [CrossRef](http://ijssurgery.com//lookup/external-ref?access_num=10.1097/01.brs.0000244618.06877.cd&link_type=DOI) 
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=17077742&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 
    
    [Web of Science](http://ijssurgery.com//lookup/external-ref?access_num=000241857900013&link_type=ISI) 

4.  Panjabi MM (2003) Clinical spinal instability and low back pain. Journal of Electromyography and Kinesiology 13:371–379.
    
    [CrossRef](http://ijssurgery.com//lookup/external-ref?access_num=10.1016/S1050-6411(03)00044-0&link_type=DOI) 
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=12832167&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 
    
    [Web of Science](http://ijssurgery.com//lookup/external-ref?access_num=000184038500007&link_type=ISI) 

5.  Dupuis PR, Yong-Hing K, Cassidy JD, Kirkaldy-Willis WH (1985) Radiologic diagnosis of degenerative lumbar spinal instability. Spine 10:262–76.
    
    [CrossRef](http://ijssurgery.com//lookup/external-ref?access_num=10.1097/00007632-198504000-00015&link_type=DOI) 
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=3992347&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 
    
    [Web of Science](http://ijssurgery.com//lookup/external-ref?access_num=A1985AGP6800014&link_type=ISI) 

6.  Farfan HF, Gracovetsky S (1984) The nature of instability. Spine 9(7):714–719.
    
    [CrossRef](http://ijssurgery.com//lookup/external-ref?access_num=10.1097/00007632-198410000-00011&link_type=DOI) 
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=6505842&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 
    
    [Web of Science](http://ijssurgery.com//lookup/external-ref?access_num=A1984TR52400011&link_type=ISI) 

7.  Pope MH, Frymoyer JW, Krag MH (1992) Diagnosing instability. ClinOrthopRelat Res 279:60–67.
    
    

8.  American Medical Association (2000) Guides to the evaluation of permanent impairment (American Medical Association, Chicago, IL), 5th ed, p 379.
    
    

9.  North American Spine Society (NASS) (2014) Lumbar fusion: defining appropriate coverage positions, pp 1–18.
    
    

10. Yeager MS, Cook DJ, Boyle C, Cheng BC (2014) Reliability of computer-assisted lumbar intervertebral measurements using a novel vertebral motion analysis system. The Spine Journal 14:274–281.
    
    

11. Diab K, Sevastik J, Hedlund R, et al. (1995) Accuracy and applicability of measurement of the scoliotic angle at the frontal plane by Cobb's method, by Ferguson's method and by a new method. Eur Spine J 4:291–295.
    
    [CrossRef](http://ijssurgery.com//lookup/external-ref?access_num=10.1007/BF00301037&link_type=DOI) 
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=8581530&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 

12. De Carvalho A, Vialle R, Thomsen L, et al. (2007) Reliability analysis for manual measurement of coronal plane deformity in adolescent scoliosis. Are 30 x 90 cm plain films better than digitized small films? Eur Spine J 16:1615–1620.
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=17619912&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 

13. Tanure M, Pinheiro A, Oliveira A (2010) Reliability assessment of Cobb angle measurements using manual and digital methods. Spine J 10:769–774.
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=20359959&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 

14. Cheung J, Wever D, Veldhuizen A, et al. (2002) The reliability of quantitative analysis on digital images of the scoliotic spine. Eur Spine J 11:535–542.
    
    [CrossRef](http://ijssurgery.com//lookup/external-ref?access_num=10.1007/s00586-001-0381-7&link_type=DOI) 
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=12522710&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 

15. Stokes IAF, Aronsson DD (2006) Computer-assisted algorithms improve reliability of King Classification and Cobb angle measurement of scoliosis. Spine 31:665–670.
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=16540871&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 

16. Gstoettner M, Sekyra K, Walochnik N, et al. (2007) Inter- and intraobserver reliability assessment of the Cobb angle: manual versus digital measurement tools. Eur Spine J 16:1587–1592.
    
    [CrossRef](http://ijssurgery.com//lookup/external-ref?access_num=10.1007/s00586-007-0401-3&link_type=DOI) 
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=17549526&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 
    
    [Web of Science](http://ijssurgery.com//lookup/external-ref?access_num=000250205600007&link_type=ISI) 

17. Shea KG, Stevens PM, Nelson M, et al. (1998) A comparison of manual versus computer-assisted radiographic measurement: intraobserver measurement variability for Cobb angles. Spine 23:551–555.
    
    [CrossRef](http://ijssurgery.com//lookup/external-ref?access_num=10.1097/00007632-199803010-00007&link_type=DOI) 
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=9530786&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 
    
    [Web of Science](http://ijssurgery.com//lookup/external-ref?access_num=000072395000007&link_type=ISI) 

18. Carman DL, Browne RH, Birch JG (1990) Measurement of scoliosis and kyphosis radiographs: intraobserver and interobserver variation. J Bone Jt Surg Am 72:328–333.
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=2312528&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 

19. Mok JM, Berven SH, Diab M, et al. (2008) Comparison of observer variation in conventional and three digital radiographic methods used in the evaluation of patients with adolescent idiopathic scoliosis. Spine 33:681–686.
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=18344863&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 

20. Breen A, Muggleton J, Mellor F (2006) An objective spinal motion imaging assessment (OSMIA): Reliability, accuracy and exposure data. BMC Musculoskeletal Disorders 7:1–10.
    
    

21. Knutsson F (1944) The instability associated with disc degeneration in the lumbar spine. Acta Radiology 25:593–608.
    
    [CrossRef](http://ijssurgery.com//lookup/external-ref?access_num=10.3109/00016924409136488&link_type=DOI) 

22. Esola MA, McClure PW, Fitzgerald GK, Siegler S (1996) Analysis of lumbar spine and hip motion during forward bending in subjects with and without a history of low back pain. Spine (Phila Pa 1976) 21:71–78.
    
    

23. Takayanagi K, Takahashi K, Yamagata M, Moriya H, Kitahara H, Tamaki T (2001) Using cineradiog raphy for continuous dynamic-motion analysis of the lumbar spine. Spine (Phila Pa 1976) 26:1858–1865.
    
    

24. Wood KB, Popp CA, Transfeldt EE, Geissele AE (1994) Radiographic evaluation of instability in spondylolisthesis. Spine (Phila Pa 1976) 19:1697–1703.
    
    

25. Muggleton J, Kondracki M, Allen R (2000) Spinal fusion for lumbar instability: Does it have a scientific basis? J Spinal Dis 13:200–204.
    
    

26. Boden SD, Wiesel SW (1990) Lumbosacral Segmental Motion in Normal Individuals Have We Been Measuring Instability Properly? Spine 15(6):571–576.
    
    [CrossRef](http://ijssurgery.com//lookup/external-ref?access_num=10.1097/00007632-199006000-00026&link_type=DOI) 
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=2402698&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 
    
    [Web of Science](http://ijssurgery.com//lookup/external-ref?access_num=A1990DT93400026&link_type=ISI) 

27. Iguchi T, Kanemura A, Kasahara K, Sato K, Kurihara A, Yoshiya S, Doita M (2004) Lumbar instability and clinical symptoms: which is the more critical factor for symptoms: sagittal translation or segment angulation? Journal of Spinal Disorders & Techniques 17(4):284–290.
    
    

28. Penning L (1992) Functional pathology of lumbar spinal stenosis. Clinical Biomechanics 7:3–17.
    
    [CrossRef](http://ijssurgery.com//lookup/external-ref?access_num=10.1016/0268-0033(92)90002-L&link_type=DOI) 

29. Stokes IA, Frymoyer JW (1987) Segmental motion and instability. Spine 12(7):688–691.
    
    [CrossRef](http://ijssurgery.com//lookup/external-ref?access_num=10.1097/00007632-198709000-00009&link_type=DOI) 
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=2961083&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 
    
    [Web of Science](http://ijssurgery.com//lookup/external-ref?access_num=A1987K527600008&link_type=ISI) 

30. Wood KB, Popp CA, Transfeldt EE, Geissele AE (1994) Radiographic evaluation of instability in spondylolisthesis. Spine 19(15):1697–1703.
    
    [CrossRef](http://ijssurgery.com//lookup/external-ref?access_num=10.1097/00007632-199408000-00008&link_type=DOI) 
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=7973963&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 

31. Tallroth K, Ylikoski M, Landtman M, Santavirta S (1994) Reliability of radiographical measurements of spondylolisthesis and extension-flexion radiographs of the lumbar spine. European Journal of Radiology 18(3):227–231.
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=7957297&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 

32. Hipp JA, et al. (2008) Motion Preservation Surgery of the Spine, Quantitative motion analysis of motion-preserving and fusion technologies of the spine (Saunders Elsevier, Philadelphia), pp 85–96.
    
    

33. Bronfort G (1994) Proc Intern Conference on spinal manipulation (June), Chiropractic spinal manipulative therapy and exercise versus pharmacological treatment and exercise for chronic low back pain, pp 35–37.
    
    

34. Pearcy M, Portek IA, Shepherd J (1984) Threedimensional x-ray analysis of normal movement in the lumbar spine. Spine 9(3):294–297.
    
    [CrossRef](http://ijssurgery.com//lookup/external-ref?access_num=10.1097/00007632-198404000-00013&link_type=DOI) 
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=6374922&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 

35. Teyhen DS, Flynn TW, Childs JD, Abraham LD (2007) Arthrokinematics in a subgroup of patients likely to benefit from a lumbar stabilization exercise program. Physical Therapy 87(3):313–325.
    
    [Abstract/FREE Full Text](http://ijssurgery.com//lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToicHRqb3VybmFsIjtzOjU6InJlc2lkIjtzOjg6Ijg3LzMvMzEzIjtzOjQ6ImF0b20iO3M6MTU6Ii9panNzLzkvMzYuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

36. Tanz SS (1953) Motion of the lumbar spine; a roentgenologic study. The American Journal of Roentgenology, Radium Therapy, and Nuclear Medicine 69(3):399–412.
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=13016843&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 

37. Friberg O (1987) Lumbar instability: a dynamic approach by traction-compression radiography. Spine 12(2):119–129.
    
    [CrossRef](http://ijssurgery.com//lookup/external-ref?access_num=10.1097/00007632-198703000-00007&link_type=DOI) 
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=2954216&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 
    
    [Web of Science](http://ijssurgery.com//lookup/external-ref?access_num=A1987G977200007&link_type=ISI) 

38. Kalebo P, Kadziolka R, Sward L (1990) Compression-traction radiography of lumbar segmental instability. Spine 15(5):351–355.
    
    [CrossRef](http://ijssurgery.com//lookup/external-ref?access_num=10.1097/00007632-199005000-00002&link_type=DOI) 
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=2363064&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 

39. Pitkänen M, Manninen HI, Lindgren KA, Turunen M, Airaksinen O (1997) Limited usefulness of traction-compression films in the radiographic diagnosis of lumbar spinal instability: comparison with flexion-extension films. Spine 22(2):193–197.
    
    [CrossRef](http://ijssurgery.com//lookup/external-ref?access_num=10.1097/00007632-199701150-00012&link_type=DOI) 
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=9122800&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 

40. Champain S, Benchikh K, Nogier A, et al. (2006) Validation of new clinical quantitative analysis software applicable in spine orthopaedic studies. Eur Spine J 15:982–91.
    
    [CrossRef](http://ijssurgery.com//lookup/external-ref?access_num=10.1007/s00586-005-0927-1&link_type=DOI) 
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=15965708&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 

41. Schulze M, Trautwein F, Vordemvenne T, et al. (2011) A method to perform spinal motion analysis from functional X-ray images. J Biomech 44:1740–6.
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=21501842&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 

42. Pearson AM, Spratt KF, Genuario J, et al. (2011) Precision of lumbar intervertebral measurements: does a computer-assisted technique improve reliability? Spine 36:572–80.
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=21217439&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 

43. 1.  Hulley S
    
    Newman T, Browner W, Cummings S (2001) in Designing clinical research, Designing studies of medical tests, ed Hulley S (Lippincott Williams & Wilkins, Philadelphia), pp 175–1931.
    
    

44. Sackett DL, Haynes RB, Guyatt GH, Tugwell P (1991) A basic science for clinical medicine, Interpretation of diagnostic data, in Clinical Epidemiology (Little Brown and Company, Boston/Toronto/London), pp 69–152.
    
    

45. Fritz JM, Piva SR, Childs JD (2005) Accuracy of the clinical examination to predict radiographic instability of the lumbar spine. European Spine Journal 14:743–750.
    
    [CrossRef](http://ijssurgery.com//lookup/external-ref?access_num=10.1007/s00586-004-0803-4&link_type=DOI) 
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=16047211&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 
    
    [Web of Science](http://ijssurgery.com//lookup/external-ref?access_num=000232530600005&link_type=ISI) 

46. Deyo RA, Nachemson A, Mirza SK (2004) Spinalfusion surgery—the casefor restraint. N Engl J Med 350:722–6.
    
    [CrossRef](http://ijssurgery.com//lookup/external-ref?access_num=10.1056/NEJMsb031771&link_type=DOI) 
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=14960750&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 
    
    [Web of Science](http://ijssurgery.com//lookup/external-ref?access_num=000188869500015&link_type=ISI) 

47. Bossuyt PM, Irwig L, Craig J, Glasziou P (2006) Comparative accuracy: assessing new tests against existing diagnostic pathways. BMJ 332:1089–92.
    
    [FREE Full Text](http://ijssurgery.com//lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEzOiIzMzIvNzU0OS8xMDg5IjtzOjQ6ImF0b20iO3M6MTU6Ii9panNzLzkvMzYuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

48. Lord SJ, Irwig L, Simes RJ (2006) When is measuring sensitivity and specificity sufficient to evaluate a diagnostic test, and when do we need randomized trials? Ann Intern Med 144:850–855.
    
    [CrossRef](http://ijssurgery.com//lookup/external-ref?access_num=10.7326/0003-4819-144-11-200606060-00011&link_type=DOI) 
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=16754927&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 
    
    [Web of Science](http://ijssurgery.com//lookup/external-ref?access_num=000238057000008&link_type=ISI) 

49. Weinstein JN, Lurie JD, Olson PR, et al. (2006) United States’ trends and regional variations in lumbar spine surgery: 1992-2003. Spine 31:2707–14.
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=17077740&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 
    
    [Web of Science](http://ijssurgery.com//lookup/external-ref?access_num=000241857900011&link_type=ISI) 

50. Leone A, Guglielmi G, Cassar-Pullicino VN, et al. (2007) Lumbar intervertebral instability: a review. Radiology 245:62–77.
    
    [CrossRef](http://ijssurgery.com//lookup/external-ref?access_num=10.1148/radiol.2451051359&link_type=DOI) 
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=17885181&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 
    
    [Web of Science](http://ijssurgery.com//lookup/external-ref?access_num=000249577500009&link_type=ISI) 

51. Newman-Toker D, Pronovost P (2009) Diagnostic errors: The next frontier for patient safety. Journal of American Medical Association 301:1060–1062.
    
    [CrossRef](http://ijssurgery.com//lookup/external-ref?access_num=10.1001/jama.2009.249&link_type=DOI) 
    
    [PubMed](http://ijssurgery.com//lookup/external-ref?access_num=19278949&link_type=MED&atom=%2Fijss%2F9%2F36.atom) 
    
    [Web of Science](http://ijssurgery.com//lookup/external-ref?access_num=000264015800031&link_type=ISI) 

52. Tan E, Ting B, Jia K, Skolasky R, McFarland E (2012) Examinations for orthopedic trauma patients. American Journal of Medical Quality, 1, 9.