ABSTRACT

Background

Graduate medical education in Singapore recently underwent significant restructuring, leading to the accreditation of residency programs by the Accreditation Council for Graduate Medical Education–International (ACGME-I). In radiology, this involved a change in teaching and quality assurance of plain film (PF) reporting. PF reported by junior residents (postgraduate year 1–3) are subject to a 50% random audit. To date, national data on junior resident performance in PF reporting have not been published.

Objective

We reviewed performance in PF reporting under the current teaching and audit framework.

Methods

Retrospective review of junior resident reported PF audit data from all 3 radiology residency programs in Singapore. The number of residents audited, number of PF reported and audited, and major discrepancy rates were analyzed.

Results

On average, 86 440 PF were audited annually nationwide from an estimated 184 288 junior resident-reported PF. Each program trained between 4 to 24 junior residents annually (mean 15), averaging about 44 each year nationwide. A mean of 28 813 PF were audited annually in each program (range 4355–50 880). An estimated mean of 4148 PF (range 1452–9752) were reported per junior resident per year, about 346 PF per month. The major discrepancy rate ranged from 0.04% to 1.13% (mean 0.34%). One resident required remediation in the study period.

Conclusions

Structured residency training in Singapore has produced a high level of junior resident competency in PF interpretation.

Introduction

Radiology residency training in Singapore transformed in 2011, leading to accreditation from the Accreditation Council for Graduate Medical Education–International (ACGME-I). Some changes included new teaching methods and introduction of quality assurance for plain film (PF) reporting. Prior to this transformation, radiology departments relied heavily on junior physicians to report PF. They were typically allowed to report PF independently, after 1 to 3 months apprenticeship with a radiology department, if they showed satisfactory on-the-job performance. No standardized assessments or audits were conducted. Since 2011, ACGME-I accreditation required the introduction of national PF training and assessment.13  Junior physicians without a residency position were no longer allowed to report PF independently.

Now, postgraduate year 1 (PGY-1) residents undergo an initial 6 months of supervised PF reporting, with a target of 600 studies and 100% over-reading by an attending. Once these requisites are met, they attempt a standardized national PF test (NPFT) in the seventh month of training.3  This test, which parallels the rapid reporting component of the Fellowship of the Royal College of Radiologists Part 2B Examination,4,5  comprises 2 sets of 30 mixed normal and abnormal PF. Each PF must be classified as normal or abnormal, and if abnormal, the resident must correctly interpret the abnormal finding. A minimum score of 80% on both sets is required to pass. Those who fail on the first attempt continue to have 100% of their PF overread and may reattempt the test in the 11th and 12th month of PGY-1.

After passing the examination, all PF reported independently by PGY-1 to PGY-3 residents are subject to a random 50% audit generated from a radiology information system (Table 1). The auditing attending first makes their own read of the PF, before reviewing the resident's report, to assess for discrepancy. A discrepant interpretation is defined as a false-positive, false-negative, or correctly identifying a finding but drawing a wrong conclusion. It is considered a major discrepancy if it could potentially result in adverse clinical outcomes (disability or mortality) or alter clinical management. For a verdict of a major discrepancy to stand, the PF is further read by a subspecialty attending physician and there must be consensus. Immediate feedback is provided to the resident for learning and to initiate follow-up action. Audit results are submitted to the national residency training committee quarterly. The audit findings, residents' interventions and outcomes are included in the annual report to ACGME-I. A resident is placed on remediation if they exceed a 5% major discrepancy rate. The 5% threshold is benchmarked to real-time errors in daily radiology practice, which averages between 3% to 5%.6,7  Each residency program performs timely audit for 80 to 140 PF each weekday, representing a half-day session for an attending radiologist.

Table 1

Radpeer Scoring

Radpeer Scoring
Radpeer Scoring

This study seeks to determinate the performance in PF reporting under the current residency framework.

Methods

Junior resident reported PF audit data from 2012 to 2018 from all 3 radiology residency programs in Singapore were retrospectively reviewed. The number of residents audited, number of PF reported and audited, major discrepancy rates, and remediation rates were analyzed.

Due to constraints of the tracking system, the total number of PF read by programs A and C residents were extrapolated from the audited numbers, but the programs gave assurance that the audited percentage is very close to 50%. And in program B, additional PF read by some residents beyond the specified training load considered for auditing were included in the total number in the initial years.

This study is classified as a Clinical Audit by our Centralized Institutional Review Board, which does not require approval.

Results

Table 2 depicts the number of junior residents in each program by year, along with audit numbers and discrepancy rates. Due to growth of the residency programs, the number of residents and thus the numbers of audited PF rose sharply in the initial years.

Table 2

Junior Resident Plain Film Audit Statistics (2012–2018)

Junior Resident Plain Film Audit Statistics (2012–2018)
Junior Resident Plain Film Audit Statistics (2012–2018)

An average of 86 440 PF were audited annually nationwide, from an estimated 184 288 PF. Each program audited between 4355 to 50 880 PF (mean 28 813). An estimated mean of 4148 PF (range 1452–9752) were reported per junior resident per year, about 346 PF per month.

The major discrepancy rate ranged from 0.04% to 1.13% (mean 0.34%), with the differences between programs shown in Table 3. One resident required remediation in the study period.

Table 3

Difference in Mean Plain Film Major Discrepancy Rates Between the 3 Residency Programs (2012–2018)

Difference in Mean Plain Film Major Discrepancy Rates Between the 3 Residency Programs (2012–2018)
Difference in Mean Plain Film Major Discrepancy Rates Between the 3 Residency Programs (2012–2018)

Discussion

The mean major discrepancy rate over 7 years is low at 0.34%, with further decrease over the years. These figures are below the published rates in the United States of 1.4% to 1.5%.8,9  We postulate several factors for this trend. First, residents undergo structured and intense PF training at all 3 residency sites. Apart from one-on-one teaching, there is additional daily subspecialty PF training using focused didactic talks and training sets. The talks introduce important concepts, pathology, and variants on a predefined topic, which are then reinforced using training sets mirroring the NPFT format, containing both normal and abnormal PF. After 6 weeks, residents would have covered the entire repertoire of cases expected of them. Subsequently, they embark on at least 80 practice sets as preparation for their NPFT. Practice sets differ from training sets as the cases are drawn from the entire curriculum and not limited to each focused topic. By undergoing this graduated intense training, our residents become familiar with the test format, learn time management, and hone interpretation skills in a short period of time. Secondly, many residents enter residency with prior radiology experience. In Singapore, many junior physicians take up non-trainee positions in radiology departments before entering residency. As such, our junior residents may be able to grasp radiology concepts at a faster pace.

Program A achieved the lowest major discrepancy rate, while the highest came from Program B (Figure), with the differences being statistically significant (Table 3). The only time an annual major discrepancy rate exceeded 1% was in the first year of audit for Program B (Figure). While each residency site uses the same national curriculum described above, it is possible that local variability in the quality of attending teaching or case variety influences the discordant rate variability.

Figure

Junior Resident Plain Film Major Discrepancy Rate Per Year by Program

Figure

Junior Resident Plain Film Major Discrepancy Rate Per Year by Program

The number of PF reported and audited parallels the number of residents, but a divergent pattern emerges for the mean number of PF reported per resident. As each program matures, more residents were progressively recruited. Consequently, the mean number of PF reported per resident decreased for Programs A and B (Table 2), but the 50% audit requirement is maintained. A newly established general hospital joined Program C at the start of residency, resulting in rapid increase in the PF workload. The workload increased substantially faster than the number of residents recruited. A plateau was reached in 2016. This explains the higher mean number of PF reported per resident for Program C in the initial years.

Several limitations exist in our analysis. Initially, clear consensus on what constituted a major discrepancy was not established among programs, leading to variability and a general tendency to categorize discrepancies as major. Therefore, the declining major discrepancy rate may have been due to a norming of a more standardized classification system. We also lack published local data to validate our NPFT.

The auditing process is resource-intensive in terms of attending supervision and tracking. With such low rates of discrepancies, it may be time to reconsider the auditing requirements. Statistical calculation for audit sample size required, based on our data of an average 0.34% major discrepancy rate, 184 288 resident reported PF annually and desired accuracy range of 0.1% with a 99.9% confidence interval, is 27 552 PF annually (15%).10,11  Selecting a larger sample size consumes additional time and resources without adding significant value.12  We did not look at discrepancy rate broken down by year of training. Presumably, more advanced residents would have fewer discrepancies, so perhaps fewer studies of more advanced residents need be audited. Of course, the auditing process may provide an additional educational value beyond simply ensuring minimum patient care standards.

Future work can be undertaken to standardize curricula, auditing processes, and data tracking.

Conclusions

Structured ACGME-I accredited training in Singapore has produced a high level of junior resident competency in PF interpretation.

References

References
1
Singapore Government Agency
.
Graduate Medical Education in Singapore
. ,
2020
.
2
Tan
BS,
Teo
LLS,
Wong
DES,
Chan
SXJM,
Tay
KH.
Assessing Diagnostic Radiology Training: The Singapore Journey
.
In
:
Hibbert
K,
Chhem
R,
van Deven
T,
Wang
S,
eds
.
Radiology Education: The Evaluation and Assessment of Clinical Competence
.
Berlin, Germany
:
Springer;
2012
:
168
179
.
3
Ministry of Health, Singapore
.
Singapore Country Addendum to the ACGME-I Advanced Specialty Requirements
. ,
2020
.
4
Royal College of Radiologists
.
Statistical Report on Fellowship Examinations for Clinical Radiology and Clinical Oncology
. ,
2020
.
5
Booth
TC,
Martins
RDM,
McKnight
L,
Courtney
K,
Malliwal
R.
The fellowship of the Royal College of Radiologists (FRCR) examination: a review of the evidence
.
Clin Radiol
.
2018
;
73
(
12
):
992
998
. doi:.
6
Brady
AP.
Error and discrepancy in radiology: inevitable or avoidable?
Insights Imaging
.
2017
;
8
(
1
):
171
182
. doi:.
7
Lee
CS,
Nagy
PG,
Weaver
SJ,
Newman-Toker
DE.
Cognitive and system factors contributing to diagnostic errors in radiology
.
AJR Am J Roentgenol
.
2013
;
201
(
3
):
611
617
. doi:.
8
Ruutiainen
AT,
Scanlon
MH,
Itri
JN.
Identifying benchmarks for discrepancy rates in preliminary interpretations provided by radiology trainees at an academic institution
.
J Am Coll Radiol
.
2011
;
8
(
9
):
644
648
. doi:.
9
Weinberg
BD,
Richter
MD,
Champine
JG,
Morriss
MC,
Browning
T.
Radiology resident preliminary reporting in an independent call environment: multiyear assessment of volume, timeliness, and accuracy
.
J Am Coll Radiol
.
2015
;
12
(
1
):
95
100
. doi:.
10
Healthcare Quality Improvement Partnership
.
Dixon
N,
Pearce
M.
Guide to Ensuring Data Quality in Clinical Audits
.
http://www.hqip.org.uk/assets/LQIT-uploads/Guidance-0212/HQIP-CA-PD-028-Guide-to-Ensuring-Data-Quality-in-Clinical-Audits-220212.pdf
.
Accessed June 30
,
2020
.
11
National Institute for Clinical Excellence
.
Principles for Best Practice in Clinical Audit
. ,
2020
.
12
University Hospitals Bristol
.
How To: Set and Audit Sample and Plan Your Data Collection
. ,
2020
.

Author notes

Funding: The authors report no external funding source for this study.

Competing Interests

Conflict of interest: The authors declare they have no competing interests.