Objective

The main objective of this study was to report results of the practice analysis survey and to provide insights into the average levels of performance and the importance of professional tasks executed by chiropractic nutritionists. In addition, this study informs the chiropractic community of the changes made to the Chiropractic Board of Clinical Nutrition diplomate exam.

Methods

Seventy-eight practicing chiropractic nutritionists responded to the practice analysis survey. Their responses were analyzed, and conclusions about frequency and importance of performance tasks were reached. A panel of subject matter experts provided a qualitative review of the survey responses. The quantitative and qualitative analyses of the survey responses indicated that minor changes to the test plan were needed.

Results

Descriptive statistical techniques were employed to analyze the survey responses. The qualitative panel suggested reducing the number of domains on the nutrition exam from 7 to 6 by combining Laboratory and Nutrition-Specific Testing and Imaging and Other Special Studies domains. Additionally, the panel decided on the final distribution of weights combining the quantitative results with qualitative perspectives.

Conclusion

The practice analysis is a first step in the definition of the skills required for practicing chiropractic nutritionists. The analysis becomes one of the references and a decision-making tool used by the board for developing and administrating quality assessments.

The Chiropractic Board of Clinical Nutrition (CBCN) collaborated with the National Board of Chiropractic Examiners (NBCE) to develop, administer, and score the CBCN exam. The CBCN was founded in 2009 as a credentialing and certifying agency for chiropractic nutritionists. The goal of the specialty group is to advance clinical nutrition while at the same time enhancing the health of chiropractic patients.

A diplomate of the Chiropractic Board of Clinical Nutrition is a licensed chiropractic physician who has successfully completed a postdoctoral program in nutrition of at least 300 credit hours from a chiropractic college or university, institution, foundation, or agency whose program is approved by an accrediting agency recognized by the US Department of Education. In addition, the diplomate has successfully passed the certifying examination administered jointly by the CBCN and NBCE.1 

The CBCN assessment is a 200-question exam administered electronically in various testing centers across the United States.2  The exam was first developed in 2010 and administered in 2011. The original test plan encompassed 7 domains: Case History, Nutrition-Related Physical and Orthopedic/Neurologic Examinations, Laboratory and Nutrition-Specific Tests, Imaging and Special Studies, Diagnosis or Clinical Impression, Treatment and Specific Nutritional Interventions, and Case Management.

This study presents methodology and reports the results of the recent practice analysis survey conducted by the NBCE for the purpose of validity reaffirmation of the CBCN exam. The NBCE is responsible to its clients for the preparation of psychometrically sound and legally defensible licensure examinations. The periodic performance of practice analysis (ie, job analysis) studies assists the NBCE in evaluating the validity of the test plan, which guides content distribution of the licensure examination. Furthermore, practice analysis studies have long been recognized by measurement and testing professions as important sources of validity evidence for licensure examinations.2 

Literature Review

Exam Validity

Assessment is an important component for ensuring public protection in professional licensing and certification. A model for test development assumes that test creators will ensure the validity of developed exams for the intended populations of test takers. The validation process begins with the formulation of detailed trait or construct definitions derived from theory, prior research, or systematic observation and analyses of the relevant behavior. Test items are then prepared to fit the construct definitions.3  Since the 1980s, test validity has increasingly become a significant concern, and evidence of validity in high-stakes assessment programs is now expected. Cronbach4  states that “validation was once a priestly mystery, a ritual behind the scenes, with the professional elite as witness and judge. Today it is a public spectacle combining the attractions of chess and mud wrestling.”

Kane5  points out that measurement is an inferential process in which limited samples are used to draw conclusions about people or organizational units. “To validate an interpretation or use of measurements is to evaluate the rationale, or argument, for the claims being made, and this in turn requires a clear statement of the proposed interpretations and uses and a critical evaluation of these interpretations and uses.”

The Standards for Educational and Psychological Testing require a verifiable close connection between the content of a test and the construct it intends to measure. “Evidence based on test content can include logical or empirical analyses of the adequacy with which the test content represents the content domain and of the relevance of the content domain to the proposed interpretation of test scores.”2 

Practice/Job Analysis

Practice analysis performs a fundamental role in developing valid licensure and certification assessment tools.6  Knapp and Knapp7  define practice analysis as a systematic collection of data describing the knowledge, skills, and/or competencies required to practice a profession. The practice analysis is conducted to inquire about the work performed by professionals to document tasks essential to practice.2,8  As a process of obtaining information about professions, practice analysis is the most widely used organizational data collection technique.9  Practice analysis helps to establish test validity by creating a profile of the profession, tracking trends in professional practice, and providing information vital to the development and refinement of professional programs. It also identifies current practice standards while anticipating future change.

Legal Standards for Practice Analysis

The legal recognition of practice/job analysis started in the area of employee selection. Legal cases related to job analysis are relevant to licensing and professional testing because “they characterize which types of procedures are viewed by the court as being appropriate for defining professional responsibilities and knowledge.”7  The Uniform Guidelines on Employee Selection Procedures provide guidance for employee selection procedures, helping employers to comply with federal laws pertaining to Title VII of the Civil Rights Act (1964). The guidelines provide many references to practice/job analyses. Specifically, Section 5 provides standards for validity studies demanding that selection procedures will have established evidence of criterion-related content and construct validity. The section requires that licensing tests, which are used to allow entrance into a profession, should be consistent with generally accepted professional standards for evaluating standardized tests and that validity studies should not be conducted by the test user.10 

The Uniform Guidelines were adopted by 5 federal agencies: the Equal Employment Opportunity Commission, the Office of Personnel Management, the Department of Labor, the Department of Justice, and the Department of Treasury.11  The leitmotif of the guidelines is the need for a close connection between a selection instrument (test) and the requirements of the profession for which the test is being used.12 

Although there is no federal law that requires a practice analysis to be conducted,13  there have been several court cases that directly referred to the procedure. In 1971, a case was debated before the US Supreme Court, which ruled against a public utility corporation that required a high school diploma for its higher-paid jobs. The Court ruled that “tests must be reasonably related” to the job for which the test is required.14  Later, in 1983, in Kirkland vs New York Department of Correctional Services, the Court stated that “identification of the relative importance of the skills and tasks involved in a job and the competency required for the various aspects of a position are essential functions of a job analysis.” Further, “the cornerstone in the construction of a content valid examination is the job analysis.”15 

In 1975, employees of the Albemarle Paper Co. claimed to have suffered from racially discriminatory hiring and promoting practices. The Court ruled that “job relatedness cannot be proven through vague and unsubstantiated hearsay” and that “limiting job analysis to selected jobs, that are unrepresentative of the full range of work performed, is inadequate for test development.”16 

Professional Standards

Testing specialists in the field of educational measurement are very particular about the requirements for a high-stakes assessment. The requirements that pertain to practice analyses include Standards for Educational and Psychological Testing2  and the Principles for the Validation and Use of Personnel Selection Procedures.17  Although the standards and principles are not legal documents, they frequently have been used by the courts to determine the appropriateness of validation procedures.18  Therefore, many licensing agencies have elected to adhere to the standards in their test development procedures.

The standards emphasize that job analysis is the primary basis for determining the content and assessing the validity of licensure examinations. A practice analysis study conducted in conjunction with exam development efforts should establish the frequency and importance of core professional tasks. For an exam that serves a purpose of public protection, based on practice analysis, test developers should be able to isolate and focus on professional core competencies that are important for the protection of the public.2 

Surveys are often constructed to collect job-related data where respondents are asked to report the frequency and the importance of the tasks they perform at work. The collected data are analyzed and numerical estimates calculated for frequency of performance, importance to public protection, and necessity to develop competency in a task at the time of initial licensure.6 

Current Study

The principal purpose of this study was to report results of the practice analysis survey and to provide insights into the average levels of performance and importance of professional tasks executed by chiropractic nutritionists. Coinciding with this purpose were more specific examinations of content validity in each domain of the exam. Finally, based on the quantitative results obtained by the survey and qualitative panel review of the test domains, the CBCN test plan was slightly modified. The final objective of this study was to inform the chiropractic community of these modifications.

Several methodological approaches exist for conducting practice analyses, including observational studies, qualitative interviews, reviews of clinical incidents, surveying, review of job diaries, review of checklists, and others.19  When selecting a methodology for practice analysis, several aspects are usually considered, including cost, practicality, purpose, analyst experience, and the profession to be analyzed.11  Nevertheless, a task survey questionnaire is the most common method of collecting information from practicing professionals.6  Therefore, surveying was used as the data collection methodology for this practice analysis.

Ethics

This study was approved by the NBCE Institutional Review Board. The board granted an exemption since the research involves survey procedures and the survey responses are anonymous.

The Survey

The 2019 survey relied on previously collected information to inform about trends in chiropractic nutrition and health care and to anticipate possible changes in the future of practice. A diverse group of chiropractic nutritionists developed and approved the content of the initial survey, which was administered in 2010.

A panel of subject matter expert (SMEs), in order to assist with the practice analysis study, was created internally at the NBCE. The panel included 3 doctors of chiropractic who also have extensive experience in test construction, a psychometrician, and a psychometric intern. One of the panel members is also a diplomate of the CBCN.

The CBCN board members and the NBCE panel of SMEs had thoroughly reviewed the survey administered in 2010, weighing the relevance of questions and providing necessary modifications. Additionally, the Role Description for a Diplomate of the CBCN was reviewed and confirmed as up to date and accurate by the CBCN Board.

The primary purpose of the 2019 survey was to inform and provide validity evidence for the content of the CBCN diplomate examination. Specifically, those responsible for test development should have empirical evidence to inform the selection of test content. The 2019 survey of clinical nutritionists instructed respondents to indicate the frequency with which they perform professional functions and specific nutritional interventions; respondents were also asked to provide an opinion of the risk to a patient's health or safety if the function or care was omitted or poorly performed.

Participants

The NBCE contacted a list of chiropractic nutritionists provided by the CBCN with a request to complete the survey. The original list contained 168 contacts; 78 survey responses were returned to the NBCE, which makes the survey response rate 46%. A variety of descriptive data were collected from the participants (Table 1).

Table 1-

Descriptive Statistics for Demographic Variables Included in the Study

Descriptive Statistics for Demographic Variables Included in the Study
Descriptive Statistics for Demographic Variables Included in the Study

Measures

The survey contained 122 items; 28 items inquired about demographic characteristics of the respondents of the patients they see. The bulk of the 2019 survey of clinical nutritionists collected information from respondents regarding the frequency with which they perform professional functions and specific nutritional interventions (frequency of professional functions). Additionally, respondents were asked to provide an opinion of the risk to a patient's health or safety if the function or care was omitted or poorly performed (risk factor).

The professional functions were presented in a logical order in the survey, beginning with the clinical nutritionist initially obtaining a case history, followed by performing examinations, then performing or ordering additional nutrition-specific studies and tests and interpreting the results. Next, respondents considered the following professional functions: developing differential diagnoses, determining a probable prognosis, and formulating a treatment plan regarding the types of nutritional interventions available, giving special consideration to interactions between herbs, medications, and other nutrients. They also reported on case management functions, such as providing nutritional counseling on lifestyle habits, diets, exercise, and overall well-being. Nutrition-specific considerations related to documentation and objective outcome measures were evaluated.

Scales

The frequency of professional functions section of the survey asked clinical nutritionists to indicate how frequently during the past 12 months they had performed each of the professional functions and how frequently they had utilized various specific nutritional interventions for patient care. The frequencies were reported on a 5-point scale ranging from never to routinely. The risk factor portion presented a context where the respondent was asked to consider a patient who needed a clinical nutritionist to perform each function. The respondent was asked to assess the risk to a patient's health or safety if a clinical nutritionist omitted or poorly performed the function. The respondent was instructed to assess risk independently of how frequently they may perform the function. Risk was assessed on a 5-point scale of no risk to severe risk.

Importance Index

The creation of the importance index was done by multiplying the frequency of responses in professional functions by the risk. The professional functions responses were dichotomized in the following way:
formula
where Responseij is the survey response i, i = 1, 2, …, 78, on a professional function item j, j = 1, 2, …, 88. The risk responses were dichotomized in the following way:
formula
where Responseik is the survey response i, i = 1, 2, …, 78 on a risk item k, k = 1, 2, …, 88.
Then the importance index was calculated for each combination of professional function and risk in the following way:
formula
where n(Responseij = 1) is the count of “routinely” or “frequently” responses on professional function items and n(Responseik = 1) is the count of “severe risk” or “significant risk” responses on risk items.

Calculation of Category Weight

The calculation of the category weights on the CBCN exam was performed in 2 steps. The first step was to compute the overall risk for the original 7 categories represented in the exam: 1 = Case History, 2 = Physical and Nutrition Examination, 3 = Laboratory and Nutrition-Specific Testing, 4 = Imaging and Other Specific Studies, 5 = Diagnosis or Clinical Impression, 6 = Treatment and Specific Nutritional Interventions, and 7 = Case Management. The following is a formulaic representation of the calculation:
formula
where Importancetc is the total importance for exam category c, c = 1, 2, …, 7, and Importanceic is the importance index for item i = 1, 2, …, Ic, in category c.
The overall importance for the exam was calculated in the following way:
formula
where ImportanceTE is the importance index for the entire exam. Finally, the category percentage (weight) was calculated in the following way:
formula
where weightc is the percentage of category c on the exam.

Content Validity

The concept of content validity is a concern with the extent to which the content of items in a scale adequately represents the complete range or breadth of the construct under consideration.20  The concept of content validity addresses the comprehensiveness and representativeness of an exam. Content validity is the representativeness of sampling adequacy of the content of a measuring instrument. It is not possible to draw a random sample of items from the hypothetical “universe” of content. Such a universe exists only theoretically; therefore, content validity is judgmental. Each item on a test must be studied, and each domain must be weighted for its presumed representativeness of the content universe.21  Lawshe22  developed a quantitative approach to content validation. In the original study, the participants were asked to rate each item on a 3-point scale, where each item is judged to be either essential, important but not essential, or unimportant. The number of essential responses was counted, and the content validity ratio (CVR) was computed in the following way:
formula
where ne is the number of panelists who rated the item as being essential and N is the overall number of panelists. The CVR index can range between −1 and +1 with values closer to 1 indicating better agreement.23 

For the purposes of current research, we calculated the CVR values for each domain of the CBCN exam. For this, we referred to the calculations previously made (see the section “Importance Index”) using n for items classified as “routinely” or “frequently” as ne.

Lawshe22  had never reported the critical values for CVR and how they relate to various sample sizes. Ayre and Scally24  evaluated critical values for CVR, producing a table of exact values including the minimum number of panel members required for the agreement to be above chance. They reported the critical CVR = .30 for a panel size of N = 40, the highest sample size considered in that study.

According to the survey, the specialty of chiropractic nutrition appears to remain a field dominated by white (92%) males (69%). The majority of the respondents indicated that they have been in practice for more than 15 years (88%), while 12% reported to be in practice for 5 to 15 years. The descriptive statistics associated with the demographic variables included in the study are presented in Table 1.

Case History (α = .95, CVR = .60)

It is noted that alpha is a reliability coefficient ranging from 0 to 1.0. and that the CVR is a coefficient of agreement between the judges on how essential a particular item or group of items is. Ratings of case history activities relating to the frequency and risk appear in Table 2. Chiropractic nutritionists reported routinely obtaining personal health history, obtaining a list of prescription and nonprescription drugs and supplements, reviewing the nervous system, reviewing the musculoskeletal system, obtaining a family health history, and inquiring about exercise habits, MChiropractic History = 4.4. The average risk involved to the patient's health or safety if the tasks were poorly performed was reported to be significant, MRisk Chiropractic History = 3.3.

Table 2-

Case History Means for Activity Frequency and Risk

Case History Means for Activity Frequency and Risk
Case History Means for Activity Frequency and Risk

Additionally, chiropractic nutritionists routinely inquire about the use of alcohol, review alimentary and gastrointestinal systems, inquire about the use of caffeine, obtain vital signs, review endocrine and metabolic systems, and review symptoms of skin conditions and allergies. Inquiring about religious or ethnic dietary restrictions and obtaining a 3-day or 7-day diet diary is sometimes performed by chiropractic nutritionists, and there is some risk to the health and safety of a patient if omitted or poorly performed.

Physical and Nutritional Examination (α = .90, CVR = .32)

Ratings of physical and nutritional activities relating to the frequency and risk are detailed in Table 3. Survey respondents reported occasional performance of all aspects of physical and nutritional examinations, MPhysical and Nutritional Examination = 3.6. Some risk was reported if the tasks were poorly performed, MRisk Physical and Nutritional Examination = 3.0.

Table 3-

Physical and Nutritional Examination Means for Activity Frequency and Risk

Physical and Nutritional Examination Means for Activity Frequency and Risk
Physical and Nutritional Examination Means for Activity Frequency and Risk

Laboratory and Nutrition (α = .72, CVR = .53)

Respondents reported that they frequently obtain and interpret blood and urine laboratory test results and that they interpret hair, saliva, stool, or other laboratory tests. The overall lower category average of 3.1 reflects the low frequency of genetic testing and live cell analysis performed by chiropractic nutritionists. Ratings of laboratory and nutrition activities relating to the frequency and risk are presented in Table 4.

Table 4-

Laboratory and Nutrition Means for Activity Frequency and Risk

Laboratory and Nutrition Means for Activity Frequency and Risk
Laboratory and Nutrition Means for Activity Frequency and Risk

Chiropractic nutritionists reported occasional performance of all aspects of laboratory and nutrition, MLaboratory and Nutrition = 3.1. There is some risk involved to the patient's health or safety if the assessment tasks are poorly perform, MRisk Laboratory and Nutrition = 3.0.

Imaging and Other Special Studies (α = .69, CVR = .47)

Respondents reported they sometimes obtain and interpret diagnostic imaging studies (eg, bone density, x-ray, and computed tomography) to evaluate for nutritional implications and that there is significant risk to the patient's health or safety if the assessment task is poorly performed. Respondents sometimes obtain and interpret other specialized studies (eg, electrocardiogram and ultrasound) for nutritional implications, and there is significant risk to the patient's health or safety if the assessment task is poorly performed. Obtaining and interpreting an electrodermal analysis is rarely performed and has little risk to the patient's health or safety if poorly performed. Table 5 presents results of performance frequency and associated risk for imaging and other special studies activities.

Table 5-

Imaging and Other Special Studies Means for Activity Frequency and Risk

Imaging and Other Special Studies Means for Activity Frequency and Risk
Imaging and Other Special Studies Means for Activity Frequency and Risk

Survey participants reported rare in-office performance of imaging and other special studies, MImaging and Other Special Studies = 2.6, which may be due to poor cost-effectiveness. Some risk was reported to be associated with poor performance of imaging/other special studies tasks, MRisk Imaging and Other Special Studies = 2.8.

Diagnosis or Clinical Impression (α = .78, CVR = .79)

Table 6 presents results for diagnostic or clinical impression. Chiropractic nutritionists reported that they routinely perform diagnosis or clinical impression, MDiagnosis or Clinical Impression = 4.6, and that there is significant risk to the patient's health or safety if they poorly perform these assessment tasks, MRisk Diagnosis or Clinical Impression = 3.8.

Table 6-

Diagnosis or Clinical Impression Means for Activity Frequency and Risk

Diagnosis or Clinical Impression Means for Activity Frequency and Risk
Diagnosis or Clinical Impression Means for Activity Frequency and Risk

Treatment and Specific Nutritional Interventions (α = .86, CVR = .43)

Treatment and specific nutritional intervention appear in Table 7. Respondents reported that they routinely recommend therapeutic levels of vitamins and minerals and frequently recommend or help develop special diets for various conditions (eg, diabetes, kidney disease, allergy elimination, gluten intolerance, and lactose intolerance). They frequently utilize nutritional detoxification protocols, specific amino acid therapy (eg, arginine, glutamine, and tryptophan), and phytochemical supplementation (eg, isoflavones and epicatechins). Furthermore, the respondents indicated utilization of weight loss management programs, glandular therapy, and supplementation for classic nutritional deficiency diseases (eg, iron, pernicious anemia, and rickets). Chiropractic nutritionists rarely manage anorexia and bulimia, but poor management is associated with significant risk.

Table 7-

Treatment and Specific Nutritional Interventions Means for Activity Frequency and Risk

Treatment and Specific Nutritional Interventions Means for Activity Frequency and Risk
Treatment and Specific Nutritional Interventions Means for Activity Frequency and Risk

On average, chiropractic nutritionists sometimes perform treatment and specific nutritional interventions, MTreatment and Specific Nutritional Interventions = 3.2, and there is some risk to the patient's health or safety if they poorly perform an assessment task, MRisk Treatment and Specific Nutritional Interventions = 3.2.

Case Management (α = .96, CVR = .46)

The averages for case management activities are presented in Table 8. Respondents reported that they frequently perform all aspects of case management, MCase Management = 4.2. There is some risk involved to the patient's health or safety if the assessment tasks is poorly performed, MRisk Treatment and Specific Nutritional Interventions = 3.3.

Table 8-

Case Management Means for Activity Frequency and Risk

Case Management Means for Activity Frequency and Risk
Case Management Means for Activity Frequency and Risk

Qualitative Panel and Test Plan Revisions

On July 2, 2019, a qualitative panel convened to review the survey results. The panel reviewed the quantitative methodology and suggested revisions to the test plan. The panel suggested reducing the number of domains on the nutrition exam from 7 to 6 by combining the Laboratory and Nutrition-Specific Testing and Imaging and Other Special Studies domains. Additionally, the panel decided on the final distribution of weights combining the quantitative results with qualitative perspectives. The final distribution of weights across domains on the CBCN exam is presented in Table 9.

Table 9-

Revised Test Plan for the CBCN Exam

Revised Test Plan for the CBCN Exam
Revised Test Plan for the CBCN Exam

This study detailed the practice responsibilities of chiropractic nutritionists, ascertaining the knowledge and skills required to effectively carry out the responsibilities of the practice. Among many factors contributing to the decision to readdress the content of an exam is the change in practice itself. An incongruity between the exam and the practice may pose a threat to content validity. Therefore, a time-to-time thorough review of the practice is strongly recommended.2 

The present investigation established validity evidence for the content of the CBCN examination. The results of the practice analysis survey have provided insights into the frequencies and importance of professional tasks performed by chiropractic nutritionists. Using the importance index, weights were calculated for all domains in the CBCN test plan. The outcomes led to the reduction of the number of domains on the CBCN exam from 7 to 6 by combining Laboratory and Nutrition-Specific Testing and Imaging and Other Special Studies.

The calculation of within-domain CVR indices provided further evidence of domain-level validity. The reliability of the survey instrument was assessed by estimating internal consistency indices.25 

Chiropractic nutritionists responding on the practice analysis found the items listed on the survey to be representative of the activities routinely performed in their practice. Although minor adjustments were made to the weight distribution, in general, the analyses of the responses to the survey items supported the logic of the test plan.

The present validation efforts relied on a mixed-method methodology, which, is defined as the “research in which a team of researchers combines elements of quantitative and qualitative research approaches for the broad purposes of breadth and depth of understanding and corroboration.”26  The combination of quantitative and qualitative results provides an opportunity to examine corroboration and convergence of findings across multiple data sources. Furthermore, the qualitative approach may help to augment statistical results derived by the quantitative part.27 

The quantitative elements in this study encompassed the statistical data analyses of the survey responses. The qualitative element was the panel of subject matter experts who reviewed the quantitative results and, utilizing their personal expertise, suggested the final distribution of the weights on the CBCN exam.

Limitations

Findings of this study should be considered in the context of potential limitations. Raymond28  advocates that although practice analysis serves as the primary source of evidence supporting the validity of scores from licensure and certification exams, there is surprisingly little consensus in the measurement community regarding suitable methods for conducting job analyses and translating the results into test plans. Furthermore, Dierdorff and Wilson29  conducted a meta-analysis of job analysis reliability using 46 various job analysis studies. The researchers could not reach a general conclusion, stating that “more research is clearly needed if a more accurate picture is to be drawn. Yet the investigators revealed that task data had higher interrater and intrarater reliabilities.

The practice analysis alone is not enough to fully support the content validity for a licensing exam. Therefore, it was imperative to conduct the qualitative panel. Knapp and Knapp7  explained that the practice analysis data play a key role in guiding decisions regarding the critical responsibilities, skills, and knowledge and that the consensus of the subject matter experts represents “the last word.” The subject matter experts provide a qualitative link between the practice analysis and the test plan, thus improving the content validity of the licensure examination.

From a statistical perspective, the findings may not reveal the complete picture due to the limited sample size. The nutrition specialty within the chiropractic profession includes fewer than 200 members. The NBCE reached out to all of them but was able to collect only 78 responses. We consider this the most significant limitation to the study.

The study utilized a survey as a methodology for data collection. Yet surveys are afflicted by many types of errors.30  As most surveys are sampling instruments designed to estimate responses of larger populations, the quality of data collected by surveys is usually related to the survey sampling design. Thus, the central concern of survey designers is to provide sample estimates of a population that are as accurate as possible.20 

Practice analysis provides information on the core tasks and risks that are most critical for competent performance by a chiropractic nutritionist and provides a viable and defensible content validation of the licensure examination. This study signifies a first step in the definition of the skills required for practicing chiropractic nutritionists. The 2019 NBCE survey based on the practice analysis becomes one of the references and a decision-making tool for developing and administrating quality assessments. Based on evidence and rigorous methodology, this study can be used to evaluate and support the CBCN test plan.

This work was funded by the Chiropractic Board of Clinical Nutrition. The authors have no conflicts of interest to declare relevant to this work.

1. 
Handbook for Diplomate Candidates [Internet].
Chiropractic Board of Clinical Nutrition; 2018 [cited 2019 Jul 2]. Available from: https://www.cbcn.us/assets/users/chiro/283/uploads/docs/2018/01/CBCN%20Handbook-June2018.pdf
2. 
American Educational Research Association, American Psychological Association, National Council on Measurement in Education.
Standards for Educational and Psychological Testing
.
Washington, DC
:
American Educational Research Association, American Psychological Association, National Council on Measurement in Education;
2014
.
3. 
Anastasi
A.
Psychological Testing. 6th ed
.
New York, NY
:
Macmillan;
1988
.
4. 
Cronbach
LJ.
Five perspectives on the validity argument
.
In:
Wainer
H,
Braun
HI,
eds.
Test Validity
.
Hillsdale, NJ
:
Erlbaum;
1988
:
3
17
.
5. 
Validation.
Kane M.
In:
Brennan
RL,
ed.
Educational Measurement. 4th ed
.
Westport, CT
:
Praeger Publishers;
2006
:
17
64
.
6. 
Wang
N,
Wiser
RF,
Newman
LS.
Examining reliability and validity of job analysis survey data
.
Paper presented at: National Council on Measurement in Education
;
April 19–23,
1999
;
Montreal, ON, Canada.
7. 
Knapp
JE,
Knapp
LG.
Practice analysis: building the foundation for validity
.
In:
Impara
JC,
ed.
Licensure Testing: Purposes, Procedures, and Practices
.
Lincoln, NE
:
Buros Institute of Mental Measurement and University of Nebraska–Lincoln;
1995
:
93
116
.
8. 
Kane
M.
Model-based practice analysis and test specifications
.
Appl Meas Educ
.
1997
;
10
(1)
:
5
18
.
9. 
Morgeson
FP,
Campion
MA.
Social and cognitive sources of potential inaccuracy in job analysis
.
J Appl Psychol
.
1997
;
82
(5)
:
627
655
.
10. 
Part 1607—Uniform guidelines on employee selection procedures. Title 29—Labor
.
Subtitle B Code of Federal Regulations;
1978
.
11. 
Foster
MR,
Condrey
SE.
Effective job analysis methods
.
In:
Condrey
SE,
ed.
Handbook of Human Resource Management in Government. 2nd ed
.
San Francisco, CA
:
Jossey-Bass;
2005
:
528
555
.
12. 
Levine
EL.
Everything You Always Wanted to Know About Job Analysis
.
Tampa, FL
:
Mariner;
1983
.
13. 
Brannick
MT,
Levine
EL.
Job Analysis: Methods, Research, and Applications for Human Resource Management in the New Millennium
.
Thousand Oaks, CA
:
Sage;
2002
.
14. 
Griggs v Duke Power Co
,
401 U.S. 424
(1971)
.
15. 
Kirkland v New York Department of Correctional Services
,
711 F.2d 1117 (2nd Cir.
1983)
.
16. 
Albemarle Paper Company v Moody
,
422 U.S. 405
(1975)
.
17. 
Tippins
N,
Sackett
P,
Oswald
F.
Principles for the Validation and Use of Personnel Selection Procedures. 5th ed
.
Bowling Green, OH
:
American Psychological Association;
2018
.
18. 
Harvey
RJ.
Job analysis
.
In:
Dunnette
MD,
Hough
LM,
eds.
Handbook of Industrial and Organizational Psychology. 2nd ed
.
Palo Alto, CA
:
Consulting Psychologists Press;
1991
:
71
163
.
19. 
Gael
S.
Job Analysis: A Guide to Assessing Work Activities
.
San Francisco, CA
:
Jossey-Bass;
1983
.
20. 
Crano
WD,
Brewer
MB,
Lac
A.
Principles and Methods of Social Research. 3rd ed
.
New York, NY
:
Routledge;
2014
.
21. 
Kerlinger
FN,
Lee
HB.
Foundations of Behavioral Research. 4th ed
.
Orlando, FL
:
Harcourt Brace Jovanovich College Publishers;
2000
.
22. 
Lawshe
GH.
A quantitative approach to content validity
.
Pers Psychol
.
1975
;
28
:
563
575
.
23. 
Pearl
CE,
Vasque
E
III,
Marino
MT,
et al
Establishing content validity of the quality indicators for classrooms serving students with autism spectrum disorders instrument
.
Teach Educ Special Educ
.
2018
;
41
(1)
:
58
69
.
24. 
Ayre
C,
Scally
A.
Critical values for Lawshe's content ratio: revisiting the original methods of calculation
.
Meas Eval Counseling Dev
.
2014
;
47
(1)
:
79
86
.
25. 
Cronbach
LJ.
Coefficient alpha and the internal structure of tests
.
Psychometrika
.
1951
;
16
:
297
334
.
26. 
Johnson
RB,
Onwuegbuzie
AJ,
Turner
LA.
Toward a definition of mixed methods research
.
J Mixed Methods Res
.
2007
;
1
(2)
:
112
133
.
27. 
Dempsey
TL.
Handling the Qualitative Side of Mixed Methods Research: A Multisite, Team-Based High School Education Evaluation Study
.
Research Triangle Park, NC: RTI International;
September
2018
.
28. 
Raymond
MR.
Job analysis and the specification of content for licensure and certification examinations
.
Appl Meas Educ
.
2001
;
14
(4)
:
369
415
.
29. 
Dierdorff
EC,
Wilson
MA.
A meta-analysis of job analysis reliability
.
J Appl Psychol
.
2003
;
88
(4)
:
635
646
.
30. 
Scheaffer
RL,
Mendenhal
W
III,
Ott
RL.
Elementary Survey Sampling. 6th ed
.
Belmont, CA
:
Thomson;
2006
.

Author notes

Bruce Shotts is the director of written examinations at the National Board of Chiropractic Examiners (901 54th Avenue, Greeley, CO 80634; bshotts@nbce.org). Igor Himelfarb is the director of psychometrics and research at the National Board of Chiropractic Examiners (901 54th Avenue, Greeley, CO 80634; ihimelfarb@nbce.org). Greg Crawford is an exam development specialist at the National Board of Chiropractic Examiners (901 54th Avenue, Greeley, CO 80634; gcrawford@nbce.org). Justin Harding is a psychometric intern at the National Board of Chiropractic Examiners (901 54th Avenue, Greeley, CO 80634; jharding@nbce.org). Andrew Gow is the director of practical examinations at the National Board of Chiropractic Examiners (901 54th Avenue, Greeley, CO 80634; agow@nbce.org).

Concept development: BS, IH, GC. Design: IH. Supervision: BS, AG, GC. Data collection/processing: IH, JH. Analysis/interpretation: IH, JH. Literature search: IH, BS, AG. Writing: IH, BS, GC, JH, AG. Critical review: IH, BS, GC, JH, AG.