Invariance Testing of the Disablement in the Physically Active Scale Short Form-10

Participants were recruited from 22 athletic training clinics and 2 outpatient rehabilitation clinics across the United States. The individuals were physically active, and those with chronic pain were excluded (Table 1).^1,2,4  All participants provided written informed consent, and the study was approved by our university's institutional review board.

Participants completed paper versions of the DPA SF-10 and a demographic questionnaire. The DPA SF-10 was completed at 3 visits; the time of completion depended on the injury type (Table 1) and was based on the methods used to create the DPA scale.¹  Recruits who were healthy or had sustained an acute or subacute injury completed the DPA SF-10 at the initial visit (time point 1), 3 to 5 days after the initial visit (time point 2), and 6 to 10 days after the initial visit (time point 3). Those who had a persistent injury completed the DPA SF-10 at the initial visit (time point 1), 7 to 10 days after the initial visit (time point 2), and 3 weeks after the initial visit or at their discharge visit if it occurred before this time (time point 3).

The deidentified participant demographic data collected included injury category, athletic status, age, sex, sport, general injury location, specific injury location, and type of injury. The collected DPA SF-10 and demographic data were input into Qualtrics (Qualtrics XM) by the collecting athletic trainer.

Data were downloaded from Qualtrics for analyses using SPSS (version 25.0; IBM Corp) and Analysis of Moment Structure (version 25.0; IBM Corp). Individuals who did not respond to the DPA SF-10 items at all 3 time points and those who did not respond to at least 90% (ie, 9 of 10) of the DPA SF-10 items were removed from the data set. Normality was assessed using histograms, skewness, and kurtosis values, and multivariate outliers were evaluated using a Mahalanobis distance set at P < .01.³ 

Data from the full sample were included in a confirmatory factor analysis (CFA) on the DPA SF-10 by time point using the Analysis of Moment Structure software to assess the scale structure and verify the underlying constructs of DIS and QOL. A hierarchical CFA was performed, grouping the disablement variables IMP and FL to create a second-order variable, DIS, that would then covary with the first-order variable QOL. To assess correlations among the 3 latent variables (ie, IMP, FL, QOL), we conducted an additional first-order CFA. Model fit indices were evaluated on the basis of a priori values; the comparative fit index (CFI; ≥0.95), Tucker-Lewis Index (TLI; ≥0.95), root mean square error of approximation (RMSEA; ≤0.06), and Bollen Incremental Fit Index (IFI; ≥0.95) were computed to assess overall goodness of fit.^3,5,6  The likelihood ratio statistic (χ²) was also assessed though it was not used as a primary assessment of model fit.

To ensure that individuals were interpreting the items and meanings of the items similarly across time (ie, across time points 1, 2, and 3), we performed invariance testing to assess measurement and structural invariance of the DPA SF-10 across the 3 time points. Longitudinal testing involved assessing the measurement parameters (ie, equal form, loadings, and intercepts), and if the model passed the invariant criteria, substantive parameters were also assessed (ie, variances, covariances, and means). The same criteria used for the CFAs were used to assess model fit.^3,7  Invariance was evaluated using the CFI difference test (CFI_DIFF), with a cutoff of <0.01 and the χ² difference test (χ²_DIFF), with a cutoff of P = .01.^3,7  Given the sensitivity of the χ²_DIFF test to sample size, the CFI_DIFF test held greater weight in decisions regarding invariance testing of model fit.⁷ 

A total of 315 individuals participated in the study; 13 people did not complete the DPA SF-10 items at all 3 time points and were removed from the data set. A total of 23 participants were identified as univariate and multivariate outliers and were removed from the analysis. When examining distributional properties (ie, skewness and kurtosis values) of the sample, we found that only 1 item had a nonnormal distribution (ie, kurtosis ≥ 3.4 but ≤ 4.00). Transformation of the data was considered; however, the data transformation was not completed because it was unlikely to lead to substantial differences in results or interpretations.⁸  The sample comprised 279 individuals (age = 23.6 ± 8.9 years; age range, 18–78 years; median age = 21.0 years), with females accounting for 53.4% (n = 149; Table 2). A total of 57 (20.4%) individuals were healthy, and 222 (79.6%) were injured; injuries were classified as persistent (n = 95, 34.1%), acute (n = 66, 23.7%), or subacute (n = 61, 21.9%). Most individuals were competitive athletes (n = 169, 60.6%) and had a high level of activity (n = 157, 56.3%).

The CFA of the DPA SF-10 model at time 1 indicated acceptable fit to the sample data (Figure). All factor loadings were different (P values ≤ .001), and goodness-of-fit indices met recommended values for CFI (0.978), TLI (0.969), and IFI (0.978) but slightly exceeded the RMSEA (0.073) cutoff. Five path coefficients exceeded 0.91. Inspection of the first-order model indicated moderate to high correlations between first-order latent variables: IMP and FL (r = 0.82, P < .001), IMP and QOL (r = 0.46, P < .001), and FL and QOL (r = 0.38, P < .001). Modification indices demonstrated meaningful cross-loadings.

The CFA of the DPA SF-10 model at time 2 indicated acceptable fit to the sample data. All factor loadings were different (P values ≤ .001), and goodness-of-fit indices met recommended values for CFI (0.970), TLI (0.958), and IFI (0.971) but exceeded the recommended RMSEA value (0.082). Five path coefficients exceeded 0.90. Inspection of the first-order model revealed moderate to high correlations between first-order latent variables: IMP and FL (r = 0.89, P < .001), IMP and QOL (r = 0.48, P < .001), and FL and QOL (r = 0.44, P < .001). Modification indices demonstrated meaningful cross-loadings, as well as specification among a number of error-term covariances.

The CFA of the DPA SF-10 model at time 3 indicated acceptable fit to the sample data. All factor loadings were different (P values ≤ .001), and goodness-of-fit indices met recommended values for CFI (0.973), TLI (0.962), and IFI (0.973) but exceeded the recommended RMSEA value (0.078). Four path coefficients were ≥0.90, with 1 exceeding 1.0. Inspection of the first-order model demonstrated moderate to high correlations between first-order latent variables: IMP and FL (r = 0.94, P < .001), IMP and QOL (r = 0.44, P < .001), and FL and QOL (r = 0.36, P < .001). Similar to time points 1 and 2, modification indices demonstrated meaningful cross-loadings, as well as specification between error-term covariances.

The full sample (N = 279) was included in longitudinal invariance testing. The initial model (configural) demonstrated acceptable model fit (CFI = 0.970, \(\def\upalpha{\unicode[Times]{x3B1}}\)\(\def\upbeta{\unicode[Times]{x3B2}}\)\(\def\upgamma{\unicode[Times]{x3B3}}\)\(\def\updelta{\unicode[Times]{x3B4}}\)\(\def\upvarepsilon{\unicode[Times]{x3B5}}\)\(\def\upzeta{\unicode[Times]{x3B6}}\)\(\def\upeta{\unicode[Times]{x3B7}}\)\(\def\uptheta{\unicode[Times]{x3B8}}\)\(\def\upiota{\unicode[Times]{x3B9}}\)\(\def\upkappa{\unicode[Times]{x3BA}}\)\(\def\uplambda{\unicode[Times]{x3BB}}\)\(\def\upmu{\unicode[Times]{x3BC}}\)\(\def\upnu{\unicode[Times]{x3BD}}\)\(\def\upxi{\unicode[Times]{x3BE}}\)\(\def\upomicron{\unicode[Times]{x3BF}}\)\(\def\uppi{\unicode[Times]{x3C0}}\)\(\def\uprho{\unicode[Times]{x3C1}}\)\(\def\upsigma{\unicode[Times]{x3C3}}\)\(\def\uptau{\unicode[Times]{x3C4}}\)\(\def\upupsilon{\unicode[Times]{x3C5}}\)\(\def\upphi{\unicode[Times]{x3C6}}\)\(\def\upchi{\unicode[Times]{x3C7}}\)\(\def\uppsy{\unicode[Times]{x3C8}}\)\(\def\upomega{\unicode[Times]{x3C9}}\)\(\def\bialpha{\boldsymbol{\alpha}}\)\(\def\bibeta{\boldsymbol{\beta}}\)\(\def\bigamma{\boldsymbol{\gamma}}\)\(\def\bidelta{\boldsymbol{\delta}}\)\(\def\bivarepsilon{\boldsymbol{\varepsilon}}\)\(\def\bizeta{\boldsymbol{\zeta}}\)\(\def\bieta{\boldsymbol{\eta}}\)\(\def\bitheta{\boldsymbol{\theta}}\)\(\def\biiota{\boldsymbol{\iota}}\)\(\def\bikappa{\boldsymbol{\kappa}}\)\(\def\bilambda{\boldsymbol{\lambda}}\)\(\def\bimu{\boldsymbol{\mu}}\)\(\def\binu{\boldsymbol{\nu}}\)\(\def\bixi{\boldsymbol{\xi}}\)\(\def\biomicron{\boldsymbol{\micron}}\)\(\def\bipi{\boldsymbol{\pi}}\)\(\def\birho{\boldsymbol{\rho}}\)\(\def\bisigma{\boldsymbol{\sigma}}\)\(\def\bitau{\boldsymbol{\tau}}\)\(\def\biupsilon{\boldsymbol{\upsilon}}\)\(\def\biphi{\boldsymbol{\phi}}\)\(\def\bichi{\boldsymbol{\chi}}\)\(\def\bipsy{\boldsymbol{\psy}}\)\(\def\biomega{\boldsymbol{\omega}}\)\(\def\bupalpha{\bf{\alpha}}\)\(\def\bupbeta{\bf{\beta}}\)\(\def\bupgamma{\bf{\gamma}}\)\(\def\bupdelta{\bf{\delta}}\)\(\def\bupvarepsilon{\bf{\varepsilon}}\)\(\def\bupzeta{\bf{\zeta}}\)\(\def\bupeta{\bf{\eta}}\)\(\def\buptheta{\bf{\theta}}\)\(\def\bupiota{\bf{\iota}}\)\(\def\bupkappa{\bf{\kappa}}\)\(\def\buplambda{\bf{\lambda}}\)\(\def\bupmu{\bf{\mu}}\)\(\def\bupnu{\bf{\nu}}\)\(\def\bupxi{\bf{\xi}}\)\(\def\bupomicron{\bf{\micron}}\)\(\def\buppi{\bf{\pi}}\)\(\def\buprho{\bf{\rho}}\)\(\def\bupsigma{\bf{\sigma}}\)\(\def\buptau{\bf{\tau}}\)\(\def\bupupsilon{\bf{\upsilon}}\)\(\def\bupphi{\bf{\phi}}\)\(\def\bupchi{\bf{\chi}}\)\(\def\buppsy{\bf{\psy}}\)\(\def\bupomega{\bf{\omega}}\)\(\def\bGamma{\bf{\Gamma}}\)\(\def\bDelta{\bf{\Delta}}\)\(\def\bTheta{\bf{\Theta}}\)\(\def\bLambda{\bf{\Lambda}}\)\(\def\bXi{\bf{\Xi}}\)\(\def\bPi{\bf{\Pi}}\)\(\def\bSigma{\bf{\Sigma}}\)\(\def\bPhi{\bf{\Phi}}\)\(\def\bPsi{\bf{\Psi}}\)\(\def\bOmega{\bf{\Omega}}\)\({\rm{\chi }}_{339}^2\) = 585.637, RMSEA = 0.051; Table 3), indicating equal form across repeated measures. The metric model (ie, equal loadings) passed both the CFI_DIFF and χ²_DIFF tests. Because the model satisfied the invariance criteria, an equal latent variance model was conducted. Both CFI_DIFF and χ²_DIFF noninvariant criteria were exceeded, indicating the variances were not equal across repeated measures. Variance ranged from 0.63 to 1.04 for the first-order latent variable IMP, from 0.54 to 0.95 for FL, and from 0.15 to 0.66 for QOL. At each time point, variance for all latent variables decreased, indicating that the DPA SF-10 scale appeared to capture less variability across the sample with repeated administrations.

The scalar model (ie, equal loadings and intercepts) passed both the CFI_DIFF and the χ²_DIFF tests. The invariant scalar model results allowed comparison of reported levels of the latent variables across repeated measures. All means for first-order latent variables IMP, FL, and QOL were different across repeated measures. The means of the latent variables decreased, indicating that across repeated measures, participants reported less IMP and FL and better QOL.

The psychometric properties of the DPA SF-10 were assessed at 3 time points. Overall, the DPA SF-10 met many model fit recommendations, which indicated it may be a suitable tool for measuring patient disablement. Although overall goodness of fit (ie, model fit indices) was met, a number of localized areas of strain in the solution indicated potential ill fit (eg, high path coefficients).⁷  For example, 1 local area of concern involved the high (0.99, 0.99, >1.00, respectively) path coefficients from DIS to IMP found across repeated measures (ie, time points 1–3). Standardized path loadings of this magnitude typically indicate the presence of multicollinearity in the data and model misspecification.⁷  In addition, the high correlations (≥0.82) between IMP and FL suggested that multicollinearity was present and that IMP and FL either were not measuring unique constructs or were being interpreted similarly by respondents. Last, model misspecification may be evidenced by the large modification indices; the indices revealed meaningful cross-loadings and suggested specification between error-term covariances.⁷  The presence of multicollinearity within factors, as well as the potential model misspecification, has also been noted in previous research.²  Our findings, when combined with those reported earlier, support the need to further remove items or reduce factors to produce a parsimonious and psychometrically sound scale.^3,7 

Longitudinal invariance was conducted to determine measurement and structural invariance of the DPA SF-10 across multiple measures. An invariant solution implies that participants, across repeated measures, interpret the questions and underlying latent factors (IMP, FL, QOL) in the same way.^3,7  Therefore, when a model is invariant, comparisons among repeated measures, levels, and groups of individuals are possible. Clinically, providers would be able to assess change over repeated measures and conclude that the measured change was a true change in the patient's perceived disablement instead of measurement error. Our invariant solution allowed us to compare sample variances and means for IMP, FL, and QOL across repeated measures. Overall, the variance in scores decreased, and participants reported improved scores for IMP, FL, and QOL, implying that treatment improved an individual's overall disablement and QOL across repeated measures.

Although the invariant model offers a promising tool for tracking a patient's disablement, overall model fit and potential multicollinearity raise concerns. Although it can be argued that scales used to assess disablement should be multidimensional,^1,9,10  scale subdimensions must uniquely contribute without substantial overlap to provide a psychometrically sound and precise measure.^3,7  Although the DPA SF-10 meets many fit requirements and offers a lower response burden than that of the original DPA scale, our data suggested the multicollinearity and model misspecification found in a previous study²  were still present among a sample of individuals who responded only to the 10 included items. Furthermore, our results confirm earlier findings^1,2  that demonstrated the QOL and DIS latent variables were unique constructs. The uniqueness of these constructs suggests that the same phenomenon is not being measured and that these responses should not be summed as a score of DIS if 1 construct that is different from QOL is already labeled disablement. Our results supported previous results²  that suggested clinicians should score and assess the individual construct scores to evaluate patient health status across physical (ie, DIS) and mental (ie, QOL) health constructs.¹¹  When considered in the context of previous findings,^2,11  our overall findings suggest further alteration of the DPA SF-10 is necessary before it can be used in clinical practice and research.

We recruited a diverse population from sites across the United States; however, participants were primarily from a small group of clinics and consisted of younger individuals. Therefore, future investigators should assess the measurement properties of the scale in samples that include active pediatric, older adult, and geriatric individuals to ensure that the instrument is psychometrically sound. Furthermore, we did not establish the responsiveness of the DPA SF-10; future authors should include this step if the scale is to be used to assess the effectiveness of clinical practice. Previous researchers have also indicated the model fit concerns in the DPA SF-10 may be adequately addressed using an 8-item scale solution (ie, DPA scale Short Form-8 [SF-8]) that removes items 4 (ie, Stability) and 6 (ie, Skill Performance) from the DPA SF-10.²  The DPA SF-8 solution fully addressed the multicollinearity present in both the DPA scale and the DPA SF-10; therefore, the DPA SF-8 may be an appropriate instrument to use if its psychometric properties can be validated with a new sample of participants who respond only to the 8 items included in the DPA SF-8. Future investigators should use similar methods as in our study to confirm the psychometric properties of the DPA SF-8 while also conducting invariance testing to ensure that the scale can be used to assess change with repeated measures or differences across groups among a sample of individuals who answer only those 8 items.

Psychometric properties of the DPA SF-10 were assessed at and across repeated measures. The CFA procedures indicated the DPA SF-10 met some goodness-of-fit indices and invariant criteria; however, a number of localized concerns previously identified in the scale were confirmed among a sample of individuals who answered only the items included in the DPA SF-10. Whereas the DPA SF-10 has improved the measurement properties and reduced the response burden compared with that of the original DPA scale, adoption into clinical practice is not recommended until further psychometric testing and model alteration confirm the most psychometrically sound instrument.