Review of Junior Resident Plain Film Reporting and Audit in Singapore

Radiology residency training in Singapore transformed in 2011, leading to accreditation from the Accreditation Council for Graduate Medical Education–International (ACGME-I). Some changes included new teaching methods and introduction of quality assurance for plain film (PF) reporting. Prior to this transformation, radiology departments relied heavily on junior physicians to report PF. They were typically allowed to report PF independently, after 1 to 3 months apprenticeship with a radiology department, if they showed satisfactory on-the-job performance. No standardized assessments or audits were conducted. Since 2011, ACGME-I accreditation required the introduction of national PF training and assessment.^1–3  Junior physicians without a residency position were no longer allowed to report PF independently.

Now, postgraduate year 1 (PGY-1) residents undergo an initial 6 months of supervised PF reporting, with a target of 600 studies and 100% over-reading by an attending. Once these requisites are met, they attempt a standardized national PF test (NPFT) in the seventh month of training.³  This test, which parallels the rapid reporting component of the Fellowship of the Royal College of Radiologists Part 2B Examination,^4,5  comprises 2 sets of 30 mixed normal and abnormal PF. Each PF must be classified as normal or abnormal, and if abnormal, the resident must correctly interpret the abnormal finding. A minimum score of 80% on both sets is required to pass. Those who fail on the first attempt continue to have 100% of their PF overread and may reattempt the test in the 11th and 12th month of PGY-1.

After passing the examination, all PF reported independently by PGY-1 to PGY-3 residents are subject to a random 50% audit generated from a radiology information system (Table 1). The auditing attending first makes their own read of the PF, before reviewing the resident's report, to assess for discrepancy. A discrepant interpretation is defined as a false-positive, false-negative, or correctly identifying a finding but drawing a wrong conclusion. It is considered a major discrepancy if it could potentially result in adverse clinical outcomes (disability or mortality) or alter clinical management. For a verdict of a major discrepancy to stand, the PF is further read by a subspecialty attending physician and there must be consensus. Immediate feedback is provided to the resident for learning and to initiate follow-up action. Audit results are submitted to the national residency training committee quarterly. The audit findings, residents' interventions and outcomes are included in the annual report to ACGME-I. A resident is placed on remediation if they exceed a 5% major discrepancy rate. The 5% threshold is benchmarked to real-time errors in daily radiology practice, which averages between 3% to 5%.^6,7  Each residency program performs timely audit for 80 to 140 PF each weekday, representing a half-day session for an attending radiologist.

This study seeks to determinate the performance in PF reporting under the current residency framework.

Junior resident reported PF audit data from 2012 to 2018 from all 3 radiology residency programs in Singapore were retrospectively reviewed. The number of residents audited, number of PF reported and audited, major discrepancy rates, and remediation rates were analyzed.

Due to constraints of the tracking system, the total number of PF read by programs A and C residents were extrapolated from the audited numbers, but the programs gave assurance that the audited percentage is very close to 50%. And in program B, additional PF read by some residents beyond the specified training load considered for auditing were included in the total number in the initial years.

This study is classified as a Clinical Audit by our Centralized Institutional Review Board, which does not require approval.

Table 2 depicts the number of junior residents in each program by year, along with audit numbers and discrepancy rates. Due to growth of the residency programs, the number of residents and thus the numbers of audited PF rose sharply in the initial years.

An average of 86 440 PF were audited annually nationwide, from an estimated 184 288 PF. Each program audited between 4355 to 50 880 PF (mean 28 813). An estimated mean of 4148 PF (range 1452–9752) were reported per junior resident per year, about 346 PF per month.

The major discrepancy rate ranged from 0.04% to 1.13% (mean 0.34%), with the differences between programs shown in Table 3. One resident required remediation in the study period.

The mean major discrepancy rate over 7 years is low at 0.34%, with further decrease over the years. These figures are below the published rates in the United States of 1.4% to 1.5%.^8,9  We postulate several factors for this trend. First, residents undergo structured and intense PF training at all 3 residency sites. Apart from one-on-one teaching, there is additional daily subspecialty PF training using focused didactic talks and training sets. The talks introduce important concepts, pathology, and variants on a predefined topic, which are then reinforced using training sets mirroring the NPFT format, containing both normal and abnormal PF. After 6 weeks, residents would have covered the entire repertoire of cases expected of them. Subsequently, they embark on at least 80 practice sets as preparation for their NPFT. Practice sets differ from training sets as the cases are drawn from the entire curriculum and not limited to each focused topic. By undergoing this graduated intense training, our residents become familiar with the test format, learn time management, and hone interpretation skills in a short period of time. Secondly, many residents enter residency with prior radiology experience. In Singapore, many junior physicians take up non-trainee positions in radiology departments before entering residency. As such, our junior residents may be able to grasp radiology concepts at a faster pace.

Program A achieved the lowest major discrepancy rate, while the highest came from Program B (Figure), with the differences being statistically significant (Table 3). The only time an annual major discrepancy rate exceeded 1% was in the first year of audit for Program B (Figure). While each residency site uses the same national curriculum described above, it is possible that local variability in the quality of attending teaching or case variety influences the discordant rate variability.

The number of PF reported and audited parallels the number of residents, but a divergent pattern emerges for the mean number of PF reported per resident. As each program matures, more residents were progressively recruited. Consequently, the mean number of PF reported per resident decreased for Programs A and B (Table 2), but the 50% audit requirement is maintained. A newly established general hospital joined Program C at the start of residency, resulting in rapid increase in the PF workload. The workload increased substantially faster than the number of residents recruited. A plateau was reached in 2016. This explains the higher mean number of PF reported per resident for Program C in the initial years.

Several limitations exist in our analysis. Initially, clear consensus on what constituted a major discrepancy was not established among programs, leading to variability and a general tendency to categorize discrepancies as major. Therefore, the declining major discrepancy rate may have been due to a norming of a more standardized classification system. We also lack published local data to validate our NPFT.

The auditing process is resource-intensive in terms of attending supervision and tracking. With such low rates of discrepancies, it may be time to reconsider the auditing requirements. Statistical calculation for audit sample size required, based on our data of an average 0.34% major discrepancy rate, 184 288 resident reported PF annually and desired accuracy range of 0.1% with a 99.9% confidence interval, is 27 552 PF annually (15%).^10,11  Selecting a larger sample size consumes additional time and resources without adding significant value.¹²  We did not look at discrepancy rate broken down by year of training. Presumably, more advanced residents would have fewer discrepancies, so perhaps fewer studies of more advanced residents need be audited. Of course, the auditing process may provide an additional educational value beyond simply ensuring minimum patient care standards.

Future work can be undertaken to standardize curricula, auditing processes, and data tracking.

Structured ACGME-I accredited training in Singapore has produced a high level of junior resident competency in PF interpretation.