Reliability and Validity Evidence of Multiple Balance Assessments in Athletes With a Concussion

Multiple tests have been developed to assess balance dysfunction in individuals with concussions. Balance assessments range from simple clinical sideline tests to complex laboratory testing. The most common assessments discussed in the concussion literature are the Clinical Test of Sensory Organization and Balance (CTSIB), the Sensory Organization Test (SOT), the Balance Error Scoring System (BESS), and the Romberg test or scale.^6,10  The CTSIB, BESS, and Romberg test are low-technology assessments of balance that are subjective and rely upon trained rater interpretations of balance deficits.⁶  The SOT uses laboratory-grade equipment to assess balance performance. However, the SOT is not cost-effective or easily accessible to the average clinician.^6,11  Because of limitations in the current balance assessments, the National Collegiate Athletic Association has adopted the Wii Fit (Nintendo, Redmond, WA) as an alternative method to assess postural control after a possible concussion and during the return-to-play process.¹²  Yet limited evidence exists to support the reliability and validity of the Wii Fit as a balance-assessment instrument in athletes with concussions. Also, limited reliability and validity evidence supports the use of the other balance-assessment instruments in individuals with concussions.

Therefore, the purpose of our systematic review was to examine the reliability and validity evidence for the Wii Fit balance game compared with traditional balance assessment by the CTSIB, the SOT, the BESS, and the Romberg test in athletes with concussions.

Reliability is the ability of a test to demonstrate the same results over time.¹³  Reliability coefficients, specifically interrater and intrarater, that range from 0 to 1 (0 = no reliability, 1 = perfect reliability) are the most commonly reported methods.¹³  For this review, we considered studies that assessed and contained reliability coefficients.

Validity refers to the ability of a test to be accurate and measure what it is supposed to measure.¹³  There are many forms of validity, including diagnostic validity, which is critical in the evaluation of an injury or disease. Correct clinical diagnosis relies upon the ability of a test to indicate if the patient has or does not have the condition.¹³  Sensitivity (true positive) and specificity (true negative) are the statistical indexes most commonly used for diagnostic validity of a test within the health sciences.¹³  Therefore, we reviewed sensitivity and specificity measures for each test of balance dysfunction in individuals with concussions.

Articles relating to the history, reliability, and validity of the current clinical balance assessments were searched using the key words (1) Clinical Test of Sensory Organization and Balance AND reliability OR validity OR sensitivity AND specificity, (2) CTSIB AND reliability OR validity OR sensitivity AND specificity, (3) CTSIB AND concussion, (4) Sensory Organization Test AND reliability OR validity OR sensitivity OR specificity, (5) SOT AND reliability OR validity OR sensitivity OR specificity, (6) SOT AND concussion, (7) Balance Error Scoring System AND reliability OR validity OR sensitivity OR specificity, (8) BESS AND reliability OR validity OR sensitivity OR specificity, (9) BESS AND concussion, (10) Romberg, (11) Romberg AND balance, (12) sharpened Romberg AND balance, (13) Romberg AND reliability, (14) Romberg AND validity, and (15) Romberg AND concussion.

The literature search was limited to the dates January 1, 1990, to January 31, 2013. We included a publication if it contributed to the understanding of the CTSIB, SOT, BESS, or Romberg test and was used in concussion research or research on neurologic conditions similar to a concussion, specifically traumatic brain injury or Parkinson disease. A publication was excluded if it did not meet the inclusion criteria. Publications posted to PubMed, PsycINFO, and CINAHL that used the CTSIB, SOT, BESS, or Romberg test in balance studies and all reliability and validity evidence of each test were examined. We selected these databases because they reflect the major contributing literature in the allied health sciences without potential overlap of citations. Our search identified and retrieved 58 articles that satisfied the inclusion criteria and pertained to 1 of the balance assessments, with an emphasis on concussion, reliability, or validity (Table 1 and Figure 1).

Articles related to the Wii Fit history, reliability, and validity were searched using the key words (1) Wii Fit, (2) Wii Fit AND, (3) Wii Fit AND reliability, (4) Wii Fit AND validity, and (5) Wii Fit AND concussion. The search was limited to the dates January 1, 1990, to January 31, 2013. We identified publications that addressed the use of the Wii Fit in concussion research or research on a similar neurologic condition to a concussion. A publication was excluded if it did not meet the inclusion criteria. PubMed, PsycINFO, and CINAHL were reviewed for information regarding the use of the Wii Fit with balance or general reliability and validity data across all demographics. We included these databases because they comprise the major contributing literature of the allied health sciences without potential overlap. Our search revealed 14 articles that were related to the purposes of the study; 6 relevant articles from the search were retrieved and included in the review (Figure 2). Specific inclusion and exclusion criteria are found in Table 1.

We identified 63 relevant articles for consideration in the review. Of the 63 articles, 28 were considered appropriate for inclusion in the review and 35 were excluded (Figures 1 and 2). Articles were rejected because they were duplicates, had no relationship to concussion or similar neurologic deficits (Methods section for exact criteria), or did not present statistically significant reliability or validity data. For example, some authors reported that the balance test was reliable (without coefficients) and valid (without sensitivity or specificity). We rejected these studies because of the lack of statistically significant reliability or validity data. Following the PRISMA 2009 guidelines¹⁴  for including significant measurement results, study strength, and weaknesses, we rated articles on an alphabetical scale (A–F). Articles rated A had a robust sample size (30+), had reliability and validity coefficients with appropriate magnitude calculations, provided power analysis, involved quantitative study methods, and were easy to read. Articles rated B had a sample size of ≥15, supplied reliability and validity coefficients (lacking magnitude calculations), and were based on quantitative study methods. Articles rated C had a low sample size (<15), gave reliability and validity coefficients, or were questionnaire-based studies. Articles rated D lacked a sufficient sample size (≤15), did not include reliability or validity coefficients, and involved qualitative study methods. Articles rated F were considered pilot studies (<10 sample size) or unpublished research and did not include reliability or validity coefficients.

Articles rated F were deemed inappropriate for inclusion and were excluded from this systematic review (Table 1). Articles with a rating of D were also rejected unless they contributed solely to the historical context of the tests in question; we included only 1 article with a D rating in this review. All of the articles included in this review are described in Table 2. The citations in the evidence table include the above-mentioned information in alphabetical order by the first author's last name. Opinion articles, consensus statements, handbooks, and other nonoriginal research included in the evidence table were not rated but were included in the study because they made a contribution to the development of the argument we present. All authors contributed to identifying and screening the retrieved articles and making recommendations as to which articles should be included; however, the first author (N.M.) made the final judgments regarding inclusion or exclusion of the articles identified.

We found that no reliability or validity data currently support the use of the CTSIB, the SOT, the Romberg test, or the Wii Fit balance game for evaluating balance dysfunction in concussion injuries. The BESS was the only balance assessment that had supporting reliability and validity evidence for use in this population.^9,11,16,24  In a single study¹⁸  of reliability, high reliability of the CTSIB was noted in older normal adults, but no reliability or validity evidence was seen in the concussion group (Table 2). The SOT had 2 articles^25,32  supporting moderate to high reliability in healthy young and older adults. No reliability and validity evidence was demonstrated for the SOT in patients with concussions. Two studies^9,11  supported the BESS's moderate to high reliability, and in a single study,¹⁶  high specificity and low sensitivity values were observed. A single study³⁴  of reliability showed moderate to high reliability (0.86) of the Romberg test for use in diagnosing balance dysfunction in individuals with Parkinson disease.

In the Wii Fit literature search, we found 2 articles^25,31  that supported the use of the Wii Fit for static balance activities and a single article²⁶  that did not support the use of the Wii Fit in dynamic balance tasks (Table 2). No currently published research studies have examined the validity of the Wii Fit.

The purpose of our systematic review was to examine the reliability and validity evidence for the CTSIB, the SOT, the BESS, the Romberg test, and the Wii Fit balance game for detecting balance disturbance in athletes with concussions. Based on the results of this review, no reliability and validity data currently exist in support of the use of the CTSIB, SOT, Romberg test, or Wii Fit for detecting concussion-related balance disturbances in all demographics. The BESS has high reliability but low validity (specifically sensitivity and specificity values) to assess balance disturbances in athletes with concussions. Because of the limitations in the literature, other neurologic disorders similar to concussion could provide relatable reliability and validity data to support the use of the CTSBI, SOT, Romberg, or Wii Fit tests for detecting balance dysfunction.

The CTSIB was originally developed by Shumway-Cook and Horak²⁸  in 1986 and involves 6 major scenarios that systematically remove or conflict sensory inputs.^6,28  The modified CTSIB, which involves more technically complex equipment measures such as a force platform, is used more commonly in research. The assessment procedures for the CTSIB, regardless of modification, include the following phases: alternating between standing on high-density foam or the ground, with eyes open, closed, or looking into an object.²⁸  Balance is measured using a scale of 1 to 4: a lower value indicates the least amount of postural sway, and a value of 4 indicates the potential for falling.²⁸ 

Guskiewicz et al¹⁹  administered the modified CTSIB and found that participants with concussions had decreased postural stability compared with baseline scores during the initial 3 days after injury. These results suggest that the balance deficits were related to sensory integration interaction errors. The CTSIB is a basic clinical test and can be administered using clinical measures or laboratory-grade technology, but it is considered a lower standard of assessment.⁶  Whitney et al³⁰  measured the reliability of the unmodified CTSIB in older adults and reported an interclass correlation coefficient (ICC) of 0.98, indicating that the CTSIB is a useful test to assess balance deficits in older adults (Table 3). However, no reliability, sensitivity, or specificity measures have been reported in the literature for healthy young individuals or individuals with concussions. Therefore, generalizability of the CTSIB is limited.

The SOT has gained considerable support in recent years and has proved to be a viable measure for assessing balance after concussion.¹¹  The SOT uses a force platform, a reference point, and a harness cage for safety. The SOT systematically disrupts afferent sensory information by reducing spatial awareness cues via somatosensory or visual components.¹¹  Individuals must attempt to maintain a steady quiet standing posture during each trial. Six different combinations of 3 trials each that alter the afferent information for 20 seconds provide clinicians with a composite balance score and an equilibrium performance score. Scoring is based on a 100-point scale, and higher scores correlate with better balance. For example, a participant whose eyes are covered will attempt to remain stable as the force platform translates in the anterior-posterior directions. In another combination, the environmental cage is moved in reference to the body's sway without movement of the force platform.

A limited number of authors investigated the reliability of instruments designed to assess balance. For example, the SOT is limited in terms of its reliability. Ford-Smith et al¹⁸  assessed the test-retest reliability of the SOT in 40 noninstitutionalized adults over the age of 65 years. The ICCs ranged from 0.26 to 0.64 during a week-to-week average of the 3 trials of the SOT balance test (Table 3). This range corresponds to reliability that is highly variable within and across conditions. Wrisley et al³²  assessed the reliability of the SOT over a 2-week period in 15 healthy young adults. The mean composite ICC score was 0.64. The reported differences between the young and older adults could be attributed to age-related deficits in functional stability, age-related changes in neuromuscular strategies to maintain posture, or the differences in multiple testing scenarios. Overall, the SOT shows a moderate level of reliability in healthy individuals, but reliability tests have not been conducted in injured populations; therefore, generalizability is limited.

The BESS was developed at the University of North Carolina–Chapel Hill to provide a low-technology, cost-effective, sport-related concussion assessment for balance.⁶  This test requires a foam platform and a stopwatch and can be used as both a sideline and a clinical test. Individuals perform a series of stances that include double, single, and tandem (legs in a heel-to-toe formation) leg support. The starting and reference position, for grading of balance errors, requires the participant to place the hands upon the iliac crest and close the eyes. A 20-second clinical observation period begins once the participant's eyes are closed. Performance is assessed by the number of deviations or errors from the original position, with a maximum of 10 possible errors per trial. Two trials are assessed for each foot position, resulting in 6 total trials.

The BESS is currently the most widely used postconcussion balance test because it is a low-technology form of analysis.⁶  In a study of the interrater reliability between the BESS and SOT measurements in athletes with concussions, the ICCs ranged between 0.78 and 0.96.¹⁰  However, when balance was evaluated in individuals with concussions, the sensitivity value for the BESS at baseline was 0.34 and the specificity values range from 0.91 to 0.96 across days 1–7 after the injury (Table 3).²⁵ 

The BESS is generally widely accepted by researchers as an appropriate test for assessing balance deficiencies in individuals with concussions.¹⁵  The BESS has been thoroughly examined and systematically reviewed in many populations, including young adults, old adults, athletes of different sports, and persons with concussions¹⁵ ; furthermore, it is emerging as the current gold standard in nonlaboratory settings to evaluate balance deficits.¹⁵  Though the BESS has wide popularity and has been clinically validated, limitations remain. Because of its reported low sensitivity statistic (0.34) and inability to detect balance dysfunction at day 7 after an initial concussion, the BESS may best be used as a prescreening test to augment sideline measurements of a suspected concussion.⁶  Lastly, the BESS relies heavily upon rater interpretation, resulting in lower interrater reliability.¹⁵ 

The Romberg test was initially developed in the late 19th century to detect sensory impairments in individuals with tabes dorsalis.²⁴  The purpose of the Romberg test is to assess balance when individuals experience reduced visual sensory input. With reduced visual input, reliance on the vestibular and somatosensory systems for balance and postural modulations is exaggerated. Commonly used by physicians and other health care professionals, the Romberg test evaluates potential balance impairments after a neurologic incident such as a concussion.^6,23  Postural control can be examined via the Romberg test by having the individual stand as quietly as possible without deviating, with the feet placed together (left and right medial malleoli touching) or in the heel-to-toe position, with differing visual sensory conditions.^10,11,16  The visual sensory conditions are eyes open and eyes closed.^10,11,16  The Romberg sign is positive if the individual begins to sway involuntarily during the reduced-vision condition.²⁹  If abnormal movements or sway are detected during the Romberg test, then a neurologic deficit is believed to exist within the higher-functioning sensory systems.²⁴  It has been suggested that the sensory dysfunction (associated with tabes dorsalis or concussion) can be related to myelopathies or neuropathies of the sensory system.^23,24  If myelopathy, neuropathy, or short-term reduction in flow of sensory information to higher centers exists, balance or postural control will be reduced.

Reliability and validity data of the Romberg test are limited. We are aware of no reliability or validity data that have been reported for this test in individuals who have experienced concussions. However, reliability evidence does exist for the Romberg test in other neurologic dysfunctions. The test-retest intraclass correlation for individuals with Parkinson disease was 0.84 in the reduced-vision condition and 0.86 in the normal-vision condition²⁹  (Table 3). These ICC values quantify the reliability of the Romberg test as excellent in individuals with Parkinson disease.

Therefore, regardless of the similarities in the mechanism underlying postural instability, no reliability or validity data currently exist for the use of the Romberg test in assessing concussion injuries.⁷  It is possible that a relationship exists between the sensory and balance deficits observed in a concussion and in Parkinson disease. Specifically, both concussion and Parkinson disease are associated with a reduction in the capacity of the basal ganglia to process sensory input, resulting in insufficient sensory integration and aberrant motor control for postural maintenance.³⁴ 

Over the past years, the Romberg test has evolved because of its subjectivity.²²  The Romberg test of standing balance on firm and compliant support surfaces is more commonly used to identify vestibular dysfunction.²²  The Romberg test of standing balance on firm and compliant support surfaces expands the traditional standing test with the addition of a compliant support surface similar to that used in the BESS.^6,22  The compliant surface emphasizes reliance upon the proprioceptive or vestibular sensory systems, effectively testing the sensory systems most commonly responsible for balance dysfunction.

The sensitivity and specificity values of the Romberg test of standing balance on firm and compliant support surfaces reported in individuals with vestibular dysfunction were 0.55 and 0.64, respectively. In individuals over the age of 40 years, the sensitivity and specificity values were 0.61 and 0.58, respectively²²  (Table 3). Thus, the Romberg test accurately detects balance dysfunction in approximately 60% of cases and should not be used as a screening measure for vestibular impairment.²² 

The Wii Fit balance game uses visual display to transpose lower body ground reaction force data to meet certain goals based upon the game's requirements. The Wii Fit has been demonstrated to be a reliable measure for static standing balance assessments of center of pressure when compared with the gold standard laboratory-grade force platform.¹⁷  The ICC was 0.94 for results within the device and 0.89 between the Wii Fit and a laboratory-grade force platform (Table 3).¹⁷  Furthermore, the Wii Fit balance game improves anterior-posterior sway during quiet stance in older adults when used as a treatment instrument.¹⁷ 

Examining the use of the Wii Fit in patients with Parkinson disease could provide additional insight into reliability and validity data for those with concussions. In a comparable static standing condition, center-of-pressure assessments revealed ICCs ranging between 0.92 and 0.98 when a force platform was compared with the Wii Fit (Table 3).²⁰  This evidence supports the suggestion that the Wii Fit can be used as an effective measure of static balance and is comparable with the gold-standard force platform.^17,20  However, the Wii Fit may be less accurate during active balance tasks that involve sudden dynamic movements.

During active movements of participants with lower extremity injury, the Wii Fit had poor concurrent validity relative to center of pressure and Star Excursion Balance Test measurements.³¹  The intersession correlation coefficient ranged from 0.39 to 0.80, and the intrasession reliability scores ranged from 0.29 to 0.74.³¹  Given the large ranges of reliability scores during active movements, the Wii Fit appears to exhibit poor reliability and validity in individuals with lower extremity injuries and in dynamic movements.

The Wii Fit is currently being used by fitness and rehabilitation clinicians to assess and aid in the recovery from injury. The range of Wii Fit games challenges a variety of motor-control strategies to accomplish the predetermined goals of the game.²⁶  Therefore, it is essential for researchers to identify which type of Wii Fit game appropriately assesses balance. In community-dwelling older adults, dual-task walking and dual-task timed up-and-go maneuvers have been significantly correlated with the basic Wii Fit step test.³³  The Wii Fit basic step test is a replication of the basic aerobic step test, which is considered a functional health assessment for submaximal and maximal cardiorespiratory fitness.³⁵  However, the validity and reliability measures of balance assessment in individuals with concussions have not been established for the Wii Fit. Additionally, no researchers have demonstrated a clear relationship between the Wii Fit and the other standard clinical assessments of balance or commonly used postural-control assessments in concussion management.

The findings of our systematic review suggest that no published reliability or validity data exist for the CTSIB, the SOT, the Romberg test, or Wii Fit in assessing balance in athletes with concussions. The BESS has high reliability and low validity evidence (specifically sensitivity) to support its use in assessing balance in an athlete with a concussion. The Romberg test and Wii Fit are reliable tools in Parkinson patients. Further reliability and validity data are needed to determine if the CTSIB, SOT, Romberg test, and Wii Fit can accurately assess balance in athletes with concussions. Additionally, more studies that evaluate the relationship between current clinical or standard balance assessments and the Wii Fit are needed.