Relationship Between Performance on Child and Adolescent Psychiatry In-Training and Certification Examinations

Subspecialty training in child and adolescent psychiatry (CAP) is 2 years long and focuses on the diagnosis and treatment of developmental, behavioral, emotional, and mental disorders of childhood and adolescence. The Child Psychiatry Resident In-Training Examination (CHILD PRITE) was developed by the American College of Psychiatrists (the College) to provide feedback to CAP fellows and program directors about the status of the fellows' knowledge as compared with others at the same level of training. The CAP Part I examination is the first step in the American Board of Psychiatry and Neurology (ABPN) certification process, which includes an oral examination (Part II) and is designed to assess core knowledge in the subspecialty. The association between the 2 examinations, which seek to assess the same achievement domain, has not been studied, and a positive correlation would provide support for their use as measures of knowledge in this psychiatric subspecialty.

Studies in other specialties have consistently yielded positive associations between performance on in-training examinations and board certification examinations, including anesthesiology,1 emergency medicine,2 family medicine,3 internal medicine,4 neurology,5 obstetrics and gynecology,6,7 ophthalmology,8 orthopedic surgery,9 otolaryngology,10 pathology,11 pediatrics,12 physical medicine and rehabilitation,13 preventive medicine,14 psychiatry,15,16 radiology,17 surgery,18 and urology.19 All of those studies focused on primary, rather than subspecialty, certification.

Most CAP training programs administer the CHILD PRITE annually as one measure of medical knowledge and to meet the Accreditation Council for Graduate Medical Education (ACGME) program requirements for an annual evaluation procedure that includes a written examination of the knowledge base.20 Hence, most CAP fellows will take the CHILD PRITE once or twice during training. Percentile ranks (percentage of examinees scoring lower than a given raw score) are reported by Year 1 fellows, Year 2 fellows, and total examinees. Percentile ranks (percentage of programs scoring lower than a given mean raw score) are also provided for program mean scores. Given its educational purpose, pass/fail standards have never been established for the CHILD PRITE.

The 2006 CHILD PRITE consisted of 200 multiple-choice questions covering the following content areas: development (15%; 30 of 200), biological science (5.5%; 11 of 200), epidemiology (2%; 4 of 200), psychopathology (20%; 40 of 200), assessment procedures (6.5%; 13 of 200), treatment (20%; 40 of 200), psychiatric aspects of pediatric disorders (5.5%; 11 of 200), issues in practice not otherwise specified (17.5%; 35 of 200), consultation other than pediatrics (6%; 12 of 200), and prevention (2%; 4 of 200). It was administered in December 2006 to 783 CAP fellows, about half of whom were in their first year of training and half of whom were in their second year. The internal consistency reliability for the total test was .85.

To qualify for ABPN certification in CAP, an individual must have satisfactorily completed ACGME-accredited fellowship training in CAP, possess an unrestricted license to practice medicine in a state, commonwealth, territory, or possession of the United States or province of Canada, and be certified in general psychiatry. The CAP candidates must first pass a multiple-choice examination (Part I) to sit for the oral examination (Part II), which consists of 2 sections, one in preschool/grade school and one in adolescence.

Many CAP graduates do not complete certification in general psychiatry, which is also a 2-step process with a multiple-choice examination followed by an oral examination, until they have completed their CAP training. Hence, there is variability in when they attempt subspecialty certification, and consequently, it was decided to include candidates who initiated CAP certification in 2007–2012.

The 2007–2012 CAP Part I examinations had 200 items covering the following content areas: development (15%; 30 of 200), biological science (5%–10%; 10–20 of 200), clinical science (5%–10%; 10–20 of 200), psychopathology/classification/differential diagnosis (15%–20%; 30–40 of 200), diagnostics (10%; 20 of 200), treatment (15%; 30 of 200), consultation/issues in practice (15%–25%; 30–50 of 200), and prevention (5%–10%; 10–20 of 200). The content areas were the same on all 6 examinations, but the percentage of items allotted to each one varied. Scores are reported as the percentage correct, and to pass, the examinees had to score at or above the pass/fail standard set for the test. The standard is set by the content experts on the test development committee and is criterion-referenced, that is, it reflects their judgment of the number of items an examinee must answer correctly to be considered minimally competent. The examinations were administered in June to the following numbers of new examinees: 279 (2007), 317 (2008), 281 (2009), 314 (2010), 392 (2011), and 381 (2012). The internal consistency reliabilities ranged from .79 to .88.

Both the CHILD PRITE and the CAP Part I examination were developed and reviewed by separate committees of content experts and test development specialists. Both tests were given under secure conditions following standardized test administration procedures, and all results, including test statistics (examinee scores, reliability, and standard errors of measurement) and item statistics (difficulty and discrimination) were reviewed. Although there is no formal relationship between the College's PRITE Commission and the ABPN, the participants in the test development committees are generally familiar with both examinations, particularly those who have served as program directors.

The College and the ABPN agreed to collaborate in this study, and CHILD PRITE examinees signed a release statement when they sat for the examination. Their scores were forwarded to the ABPN, and the names of those individuals were matched with the names of the examinees who took the CAP Part I examinations. As part of the ABPN application process, the applicants signed a statement allowing the Board to release information about examination results and examination scores, provided such data were reported in the aggregate.

Of the 783 fellows who took the 2006 CHILD PRITE, 513 (66%) agreed to participate in the study. Of those, 453 (88%) could be matched with names in the ABPN database. Of the 453, 342 (75%) have participated in the CAP certification process thus far: 14 (4%) initiated CAP certification in 2007, 77 (23%) in 2008, 105 (31%) in 2009, 72 (21%) in 2010, 46 (13%) in 2011, and 28 (8%) in 2012.

Of the 342 subjects, 5 (1%) completed CAP training in 2006, 168 (49%) completed training in 2007, 164 (48%) completed training in 2008, and 5 (1%) completed training between 2009 and 2012. For subsequent analyses, based on the year training was completed, 169 (49%) were assumed to be in their first year of CAP training when they took the 2006 CHILD PRITE and the remaining 173 (51%) were assumed to be in their second year.

Of the 111 CAP fellows not in the study, 67 (60%) are certified in general psychiatry and have not applied for CAP certification, and 44 (40%) are not yet certified in general psychiatry.

In addition to descriptive statistics, a t test for independent samples was used to compare the performance of the 2 training groups on the examinations, and Pearson correlation coefficients were computed to examine the relationship between scores on the 2 tests. Because the subjects had taken 1 of 6 different CAP Part I examinations, linear equating21 was used to equate scores on the other 5 examinations to the 2009 examination. Linear equating is used to establish equivalent scores from examinations with different means and standard deviations.

The examination results are summarized in table 1. For the 2006 CHILD PRITE, the mean raw score for Year 1 CAP fellows was 132 (SD  =  13), and for Year 2 CAP fellows, it was 142 (SD  =  12). The Year 2 CAP fellows performed significantly better on the examination (t  =  −7.13; df  =  340; P < .001). The mean raw score for the 342 study participants was 137 (SD  =  14). In 2006, the mean score for all examinees was 134 (SD  =  15). On the ABPN CAP Part I examination, the mean score percentage for Year 1 CAP fellows was 77% (SD  =  7%), and for Year 2 CAP fellows, it was 78% (SD  =  6%). The Year 2 CAP fellows performed significantly better on the examination (t  =  −2.23; df  =  340; P < .03); however, the difference in mean scores of about 1 percentage point had no practical significance. On the CAP Part I examination, the equated mean score for the study participants was 78% (SD  =  6%), and 96% passed on first attempt. The pass rate for all new examinees in 2007–2012 was 91%.

Because the Year 2 CAP fellows performed significantly better on the CHILD PRITE than did the Year 1 CAP fellows, separate correlation coefficients were computed for the 2 groups. For the Year 1 group, the correlation between the CHILD PRITE total score and the CAP Part I examination total score was .41 (P  =  .01). For the Year 2 group, the correlation was .52 (P  =  .01).

The CHILD PRITE report provided to examinees and program directors gave percentile ranks for Year 1 and Year 2 trainees, so that individual scores could be compared to the appropriate peer group. The performance of 4 percentile groups (1st–25th, 26th–50th, 51st–75th, and 76th–99th) on the 2 examinations and the pass rate on the certification examination are summarized in table 2 (Year 1 CAP fellows) and table 3 (Year 2 CAP fellows). Although the scores and pass rates on the certification examination increased across the 4 percentile groups for both groups of fellows, all groups performed quite well on that examination.

Candidates who pass the CAP Part I examination go on to the CAP Part II (oral) examination. Those who fail typically retake the multiple-choice examination the next year. At this point, 80% (273 of 342) of the study participants are certified in CAP.

This study focused on the relationship between performance on an in-training and a certification examination in a subspecialty, and significant positive correlations were found between the 2 assessments. The correlations were .41 for first-year CAP fellows and .52 for second-year CAP fellows. It was also found that the second-year fellows performed significantly better on the in-training examination than did the first-year fellows, and the correlations between the 2 examinations were higher for the Year 2 fellows than they were for the Year 1 fellows. These results are similar to those reported in many of the studies cited previously for specialty in-training and certification examinations.

Based on performance on the CHILD PRITE, the fellows who were at somewhat more risk on the certification examination were those who scored at or below the 25th percentile on the in-training examination, but most of them (> 80%) nonetheless passed the CAP Part I examination on first attempt.

The positive correlations between these 2 measures of knowledge in child and adolescent psychiatry provide evidence of the validity of the CHILD PRITE and the CAP Part I examination. The significantly better performance by the more advanced fellows on the in-training examination and the higher correlation between their in-training and certification examination scores are additional sources of validity evidence.

One limitation to the study was the relatively small sample size related to voluntary participation, difficulty in matching names of subjects across both assessments, and lack of participation in the certification process. However, this group's rate of participation in CAP certification (75%) was similar to that found in a previous study in which it was estimated that about 67% of CAP graduates sought ABPN certification in CAP.22 

Because becoming certified in a subspecialty is contingent on first achieving primary certification, there may be a restriction in range in the study sample. Those who experienced difficulty in achieving specialty certification or who delayed certification may be less able candidates, and if they were included, the correlations might have increased because of the increase in sample variability. The finding that the study participants performed somewhat better on both measures than did their peers also suggests decreased variability in the sample. In addition, the time between taking the CHILD PRITE and the CAP Part I examination may have weakened the correlation between the 2 assessments. Also, most examinees performed quite well on the certification examination.

Although the CHILD PRITE is a formative examination designed to give feedback to trainees and faculty on the relative performance of CAP fellows, and the independently developed CAP Part I examination is a component of a 2-part certification process, these results suggest that the in-training examination can provide guidance for CAP fellows and their training directors in preparing for the CAP Part I examination.