Implementation of a Standardized Dataset for Collecting Information on Patients With Spinal Cord Injury

Twelve ISCIDS modules that were available in 2012 were translated from English into Dutch. The modules are as follows: core,4 lower urinary tract function,5 bowel function,6 cardiovascular function,7 pulmonary function,8 male sexual function,9 female sexual and reproductive function,10 urodynamic,11 pain,12 skin and thermoregulation function,13 and quality of life.14 

Additionally, the International Standards to document remaining Autonomic Function after Spinal Cord Injury (ISAFSCI), which was developed through a collaboration between the American Spinal Injury Association (ASIA) and the International Spinal Cord Society (ISCoS),15 was translated.

Translation of the data sets was performed according to the recommendations suggested by Biering-Sørensen et al.3 In the first step, two persons (H.S. [rehabilitation physician] and S.vL. [physiotherapist]) with extensive knowledge on SCI independently translated the data sets into Dutch and consulted two other persons (J.N. and M.P. [researchers]) when necessary. After translating the full core data set, including the introducing text and the training cases, it was decided to translate only the variable names, variable definitions, and ISCoS comments regarding details of the other data sets. Background and classification rules were translated only for the core data set and not for the other 11 data sets. This was decided because all health professionals working with the data sets are able to read and understand English. In the next step, the translated data sets were discussed by a group of 20 DuFSCoS rehabilitation physicians. The translation of the core data set was extensively read by 11 of these professionals, the other data sets were checked by two of the health professionals each. The health professional checking the translation was asked whether the translation was correct, and, if not, what was incorrect in his or her opinion. With the comments we received, the final translations of the data sets were established.

After the ISCIDS were translated into Dutch, they were combined to form a joint data set. The same 20 rehabilitation physicians were asked in a paper-and-pencil round to judge each item by whether they thought the item should be incorporated into the NDD or not and per data set whether there were items missing that should be included or items that should be changed. When 70% (14 rehabilitation physicians) or more of the professionals agreed on the inclusion of an item in the NDD, it was incorporated.16 If less than 30% agreed with inclusion in the NDD, an item was excluded. In a plenary consensus meeting, suggested changes, additional items, and items for which 30% to 70% of the professionals agreed to include in the NDD were discussed until consensus was reached on the inclusion or exclusion of the item.

Most adaptations were made in the core data set. For example, next to the date of acute care hospital admission, an item on the date of admission to the rehabilitation center was added. Other adaptations to the core data set were as follows:

• The place of stay before admission to the rehabilitation center (home, hospital [including general or academic hospital or hospital abroad], other, and unknown) was added, because these centers do not provide acute care (after onset of SCI) in the Netherlands.

• Date of final inpatient discharge was called date of final discharge from the rehabilitation center.

• Adaptations were made in the examples of places of discharge particular to the Dutch situation. For example, condominium was left out, and other rehabilitation center was included.

• An item was added as to whether the place of discharge is temporary or not. In some cases, place of discharge is temporary, for example, until adaptations to the patient's house are finished.

• Specification of the type of non-traumatic causes of SCI (vascular, oncologic, inflammation, degenerative, congenital, other) was added. Specification was based on the ISCIDS on non-traumatic SCI,17 

• Pattern of onset of the SCI was added, based on the ISCIDS on non-traumatic SCI17 (within 1 day, more than 1 day but within 7 days, between 7 days and 1 month, more than a month).

• Type of associated injury was specified (moderate to severe traumatic brain injury, non-vertebral fractures requiring surgery, severe facial injuries affecting sense organs, major chest injury requiring chest tube or mechanical ventilation, traumatic amputations of an arm or leg, severe hemorrhaging, or damage to any internal organ requiring surgery).

• Type of spinal surgery was specified (laminectomy, spinal fusion, other, unknown).

Some of these changes were adopted to the International SCI Core Data Set version 2.0 (adding date of rehabilitation admission, extending causes of SCI with 6 non-traumatic causes).18 

Another data set with major adaptations was the Pain Data Set, which was shortened to reduce the time to complete the data set. Instead of asking how many pain problems a patient experiences and describing the three major pain problems, the data set now focuses on the presence or absence of pain. If pain is reported, it is asked whether nociceptive, neuropathic, or other pain was present. When present, the location and type of pain is specified in detail (eg, visceral, musculoskeletal, or other for nociceptive pain and at and/or below level and other for neuropathic pain) for each type of pain separately. Additionally, the intensity of the pain during the last week, how much the pain limits the activities of the patient, and treatment of the pain were registered. Items concerning duration of the pain and most items on pain interference were omitted.

Duplicate items that are covered by several data sets were included only once. For example, only the items on transpiration and temperature were adopted from the data set to document remaining autonomic function. The items on heart rate and blood pressure were taken from the data set on cardiovascular function, items on the lower urinary tract function and urodynamic evaluation were covered by the data sets on bowel, bladder, sexual, and urodynamic functioning instead of from the autonomic function data set. In the skin and thermoregulations data set, only the presence and characteristics of pressure ulcers were included in the NDD, as thermoregulation was covered by the items from the data set on remaining autonomic function.

Other adaptations to the data sets included adding a question comparing quality of life before the SCI with the quality of life at the present moment to the data set on quality of life and omitting a question about changes in urological symptoms within the last year from the data set on bladder function. No changes were made in the data sets on bowel function, cardiovascular function, pulmonary function, male and female sexual function, and urodynamics. In addition to the ISCIDS, the International Standard for Neurological Classification of Spinal Cord Injury (ISNCSCI) and three standardized measurement instruments were included in the NDD: the Walking Index for SCI (WISCI II), Spinal Cord Independence Measure (SCIM III), and the Hoffer classification of functional mobility.19–22 

After reaching consensus, the interrater reliability of the NDD was examined in 30 patients; three rehabilitation centers tested 10 patients each. Characteristics of the patients are presented in Table 1. All three centers provided two rehabilitation physicians specialized in the rehabilitation of patients with SCI. The centers were instructed to include inpatients who were medically stable, to include a representative sample of patients with more men than women, and to include patients with tetraplegia and paraplegia. For each patient, the NDD was administered twice, by a different physician, with one or more days in between (median, 4 days [range, 1–41]). The physicians collected the information directly from the patient; in some cases, minor additions were extracted from the medical record of the patient. Percentage of agreement was defined as the percentage of patients for which both raters gave the same answer on an item. Cohen's kappa was used to calculate the interrater reliability of dichotomous and categorical variables, and intraclass correlation coefficient (ICC) was used in the case of continuous variables. Per data set, the mean percentage of agreement and mean kappa over the items is presented.

The study included 23 men and seven women, with a mean age of 45.1 years (SD = 16.5 years). Most of them (22) had a traumatic SCI (Table 1). Two of the included women did not want to talk about their sexual functioning with both of the physicians, and another woman did not want to talk about her sexual functioning with one of the physicians administrating the NDD. So, for female sexual functioning, insufficient data were available to determine the interrater reliability. Interrater reliability also could not be established for urodynamics, as not all patients have had urodynamic examination at the moment of administration of the NDD; as a result, there was too much missing data. Finally, interrater reliability could not be established for the Skin and Thermoregulation Data Set because of missing data: in only 4 of the 30 patients were responses available on the questions on the location and severity of pressure ulcers.

As Table 2 shows, for all data sets the agreement ranges between 50% and 100%, with a mean agreement per data set of at least 66.4%. Cohen's kappa was 0.46 or higher, representing at least moderate agreement, corresponding to moderate to almost perfect agreement.23 As an example, the interrater reliability of the items on bowel management are shown in Table 3. In some of the items with a very high agreement, kappa was very low, due to a high prevalence of one of the answer categories. This was the case for the items on anal dysfunction, supplementary methods of defecation, and the presence of perianal problems. In these cases, only the percentage of agreement was reported. The variable “supplementary method of defecation” was registered by both physicians for two patients and by one of the physicians for five patients; for the other 23 patients, the variable was not answered by both rehabilitation physicians. Although it could be stated that there was agreement between the raters for the 23 patients in whom the variable was not registered and no agreement in the five patients in whom the variable was recorded by one of the raters, it was decided to include only people in whom the variable was recorded by both rehabilitation physicians. The variable “need to wear a pad or plug” was handled the same way. In this variable, two of the answer categories are “unknown” and “not applicable.” In 14 cases, one of the raters responded with unknown or not applicable to this variable, whereas the other rater did not register this variable. Although these missing answers could have been explained as unknown as well, it was chosen to exclude these 14 people in the analyses of the variable “need to wear a pad or plug.”

Determination of the functional mobility and walking ability was measured reliable as well. The percentage of agreement for the Hoffer was 85.7%, with a weighted kappa of 0.74, whereas the ICC for the WISCI III was 0.84.

Table 4 shows the results of the interrater reliability of the SCIM III. For all subscales, the mean percentage of agreement on item level was higher than 80% and the interrater reliability was good with an ICC of 0.83 or higher.

In addition to the reliability study, the participating physicians were asked for each dataset whether there were any doubts on what to answer on the items in the NDD. When they stated that they had any doubt on what to answer, it was asked whether they had checked the manual of the NDD and if the manual was clear in these cases. It appeared that the rehabilitation physicians had hardly any doubt on what to answer on the items of the NDD. In only 14 of the 720 items was there any doubt on what to answer. The manual was consulted in only three cases by the physician, and the manual was clear enough to help them out.

The NDD is administered in all eight rehabilitation centers in the Netherlands specialized in the rehabilitation of patients with an SCI. Data collection is facilitated by a web-based data entry platform and database. The platform is accessible with a personal access code. Patients can be shared by several professionals within a center or can be transferred to another center when applicable. Authorized persons are able to export the data of their center to a CSV file and some coordinating people are able to export the data of all centers together. Which data and measurement moments will be exported can be chosen. In all participating centers, one or more persons, for example a nurse practitioner, physician assistant, or resident, takes care of the data collection at admission and discharge of each patient. In some centers, one person is responsible for the data collection, whereas in other centers more professionals are involved in administering the NDD. The data are partly retrieved directly from the patient and partly retrieved from the medical record (eg, medication for bladder regulation).

Four months after the implementation of the NDD at all participating centers, a questionnaire was sent to the centers to review their experiences with the NDD. The questionnaire evaluated several topics, including the opinion on (1) continuing or stopping the NDD, (2) using the data for multicenter comparisons, (3) managing of the data, (4) topics missing in the NDD, and (5) redundant topics in the NDD. One of the eight centers did not respond to the questionnaire. In the centers where multiple professionals administered the NDD, each professional received a questionnaire. When there was a discrepancy in the answers given by the professionals within a center, the most negative answer was used in the analysis.

All centers responded positive to the question on continuing to use the NDD as a national database. They felt an instrument like the NDD fits well with international developments and is a useful instrument to gain insight into some epidemiologic characteristics of the population with SCI within the Dutch rehabilitation centers. Comments were given regarding the lack of a link between the platform in which the NDD is built and the electronic patient files of the centers. Ideally, those are integrated to avoid duplication of data entry. All centers pointed out that there should be clear agreements on using the data for analyses. The majority of the respondents (6 out of 7) agreed that requests for using the national data, for example, for research purposes or benchmarking, should be controlled by the DuFSCoS and that the data should be analyzed anonymously, both on the level of the individual patient and on the level of the participating rehabilitation centers.

Six respondents indicated that there were topics in the NDD that they found difficult to discus with the patient at admission or which they postponed until a later date. This was especially the case for sexuality (mainly female sexuality), quality of life, and pain. One of these respondents did not want to discuss sexuality with the female patients at all and skipped this part of the NDD. Four of the respondents missed topics in the NDD, whereas three stated that they did not miss any topic. Missed topics were on the lung function, such as air-stacking and vaporizing, medical history, comorbidity, spasticity, muscle strength, and psychiatry. Three respondents noted they would remove some of the topics. Two of them stated that the data set on urinary tract function was too comprehensive, especially on surgical procedures that are in most of the cases not relevant during the first (inpatient) rehabilitation period after the onset of SCI. One person thought bowel function was examined too extensively. One person would remove the topic on (female) sexuality, and it was suggested that quality of life not be examined at admission, but only at discharge. Summarizing the results of the survey, all respondents were willing to continue using the NDD and to collect the data in the national database. Main obstacles for succeeding were thought to be the absence of a link between the local patient files and the national database, which leads to double data entry, and a lack of time to fill in the NDD. Completing the NDD can take about 30 to 60 minutes.

In accordance with the results of the user evaluation survey, a contract has been made in which the council of the DuFSCoS is the owner of the NDD. A coordinating committee has been founded to stimulate data collection and to coordinate the use of multicenter data for research and quality improvement purposes. Once a year, the committee organizes a meeting for the persons taking care of the data collection. In these meetings, they discuss the obstacles they face and, when necessary, the NDD will be adapted. Also, the committee discusses with the DuFSCoS whether newly available data sets should be incorporated in the NDD and, if so, in what period.

The NDD is collaboratively developed by all eight rehabilitation centers with a specialty in SCI in the Netherlands. With the NDD, it is possible to collect important data on the majority of persons with SCI in the Netherlands. The reliability study showed that, although the results are based on a small sample size and the study procedure was somewhat different from the normal clinical situation, the use of the NDD for the registration of medical information is at least sufficient. Experiences from the participating centers were mainly positive as well. It is thought that an instrument like the NDD fits well with international developments and is a useful instrument to gain insight into some epidemiologic characteristics of the population with SCI within the Dutch rehabilitation centers.

Whereas most SCI databases include only people with traumatic SCI,24 the NDD is administered in people with traumatic and non-traumatic SCI. Another strength of the NDD is the broad coverage of topics and the incorporation of the ISCIDS. There are, however, also some challenges for succeeding. At the moment, the NDD is not incorporated in the electronic patient file most of the centers use, which results in double data entry. In the future, a connection between systems or another way to centrally collect the data is recommended to prevent double data entry and to guarantee continuation of administration of the NDD.

Another constraint is the lack of external funding for the project. The project depends on the willingness of the participating centers to provide manpower, and it is difficult to check the data continuously on completeness and accuracy. Very recently the first data collected with the NDD were analyzed.25 These analyses showed that completeness of the data collection varied strongly among the different topics of the NDD. Data sets including sensitive topics like sexuality and quality of life were incomplete. This is in line with what we found in the reliability and evaluation study; some participating centers reported that they were reluctant to address these topics, especially shortly after admission. One of the recent suggestions of the coordinating committee of the NDD is to reduce the data collection at admission to improve the completeness of the data and to adapt the way in which the ISNCSCI is administered. To compute the level of injury and the American Spinal Injury Association Impairment Scale (AIS), the NDD contains an algorithm, however ongoing bugs hindered the registration of the ISNCSCI. It is decided to register only the level of injury and AIS, which are calculated using one of the other algorithms available online.26 Additionally, the committee will submit a proposal to the DuFSCoS on how to reduce the NDD administered at admission to the rehabilitation center.

Despite the fact that the NDD is not administered completely in most of the patients, there are important data available for the majority of the patients admitted for (inpatient) rehabilitation after the onset of SCI. Overall, the NDD is a useful instrument to collect information on the functioning of patients with SCI in a standardized way.

The development of the NDD was supported by Rehabilitation Netherlands (Revalidatie Nederland), Innovation in Rehabilitation program, grant IPR2011-12. Many thanks to all rehabilitation physicians participating in the consensus rounds and to all people involved in the data collection for sharing their experiences on the implementation of the NDD. The authors declare no conflicts of interest.