Background

The mini-Clinical Evaluation Exercise (mini-CEX) and direct observation of procedural skills (DOPS) are reliable tools for work-based assessment of medical trainees. Tools of this type do not yet exist for evaluation of practical laboratory skills of pathology residents.

Objective

We developed and piloted a 9-item instrument for direct observation of laboratory skills (DOLS).

Methods

We used the DOLS tool with 10 hematopathology residents (PGY-1 to PGY-5) from Aga Khan University. Each resident was evaluated by 3 faculty members in the laboratory during 4 separate encounters using the DOLS instrument. We assessed construct validity, interrater reliability and G coefficient, feasibility of using DOLS, and learner satisfaction.

Results

A total of 120 encounters were observed with a mean score (±1 SD) of 56.7% (±12.44). Assessment scores moderately correlated with the number of laboratory procedures previously performed by participants (r = 0.658 and 0.641; P = .0001) and with PGY level. Interrater reliability ranged between 0.47 and 0.96. Cohen's d was 1.64. Residents accounted for a large component of estimated variance (73%), suggesting DOLS can differentiate residents' laboratory skills; variance associated with assessors was small (0.01%). Residents reported being satisfied with the tool. Mean time (±1 SD) taken for observing and feedback was 17.89 ± 5.89 minutes.

Conclusions

The new DOLS instrument could provide reliable scores for observing laboratory skills. Residents were satisfied with the tool, and rating times make the tool feasible for formative assessments.

For hematopathology residents, acquisition of laboratory skills is an essential goal, as 70% of clinical decisions are based on a laboratory diagnosis.1  Time constraints in service-heavy laboratories and lack of assessment tools are 2 barriers to good assessment.

Validated instruments are available for assessing clinical examination (the mini-Clinical Evaluation Exercise [mini-CEX]) and procedural skills (direct observation of procedural skills [DOPS]).2  Residents value opportunities to be observed and given feedback,3  yet few studies have assessed skills in pathology4  and hematology residents,5  and there currently are no validated tools.

We developed and piloted a tool for direct observation of laboratory skills (DOLS) in hematopathology residents.

This study was conducted in 2016 in the Section of Haematology and Transfusion Medicine, Department of Pathology and Laboratory Medicine, at Aga Khan University (AKU), in Karachi, Pakistan. The institution offers a 5-year hematopathology residency and a 2-year program for management of hematological disorders following 3 years of internal medicine training.

Development of the Tool

DOPS is a validated tool for assessing procedural skills, and it served as the basis for our DOLS tool. To create DOLS, we removed irrelevant items from the DOPS list and replaced them with pre-analytic steps, interpretation of test results, and biosafety. The resulting instrument was reviewed by AKU faculty, residents, and education experts. Based on feedback, we reduced the scale from 9 to 6 points. DOLS (available as online supplemental material) consists of 9 items scored from 0 to 6 (1 to 2 is below the expected level of competency, 3 is borderline, 4 meets, 5 to 6 exceeds, and 0 could not observe). The competency score is the cumulative performance on the 9 items and in case of unobserved performance, average score is provided with a 0 item score removed from both numerator and denominator. Items are broad and applicable to a variety of laboratory procedures. To ensure consistency in assessments, we developed guidelines, a procedure-specific checklist, and a feedback form (available as online supplemental material).

Selection of Procedures and Use of DOLS

Four tests with moderate complexity were selected as essential procedural skills: ABO blood typing, D (Rh) typing, prothrombin time (PT), and activated partial thromboplastin time (APTT). Each resident was observed by 3 assessors simultaneously during 4 encounters, followed by verbal and written feedback. Residents in postgraduate year 1 (PGY-1) through PGY-5 were included to observe differences in performance. Residents were classified respectively as “novice” and “experienced” if they had performed < 10 or ≥ 10 lab procedures prior to the assessment, based on the authors' observation that ≥ 10 encounters improve bench skills. Data were anonymized to conceal the identity of residents and assessors.

Ethical approval was obtained from the AKU Ethical Review Committee.

Statistical analysis used SPSS Statistics 23 (IBM Corp, Armonk, NY, USA) and STATA version 12 (StataCorp LLC, College Station, TX, USA). Total scores were computed as a percentage of maximum score possible for a 6-point scale on 8 items excluding overall competency, for a possible maximum score of 48 (8 items × 6 score). The relationship between scores and the procedures performed was assessed using Spearman correlation, and scores for novice and experienced residents were compared using the Mann-Whitney U test. The intraclass correlation coefficient (ICC) was used for measuring interrater reliability of scores. Single measures ICC scores were computed to observe how reliable DOLS would be if a single rater was used. Internal consistency was measured by Cronbach's alpha. The difference in scores for novice and experienced trainees was calculated using Cohen's d, and generalizability (G) theory was used to determine the robustness of DOLS scores. This takes into account the sources of variance in measurements.

Nine hematopathology residents, 1 clinical hematology resident, and 3 of 7 faculty members participated in the study. Mean assessor time for observing was 13.17 ± 5.76 minutes and 4.72 ± 0.66 minutes for providing feedback. Resident performance varied across tests, and tests with multiple technical steps (PT and APTT) were not performed well by PGY-1 to PGY-3 residents, whose performance was satisfactory on simpler tests (ABO and D typing). PGY-4 and PGY-5 hematopathology residents performed satisfactory in all 4 encounters, and the PGY-5 clinical hematology trainee showed borderline performance for PT and APTT.

The mean score was lowest for technical skills (2.98 ± 1.22) and highest for professionalism (3.93 ± 0.25). The mean total score was 56.74 ± 12.44 (range 27.08–66.67), and scores were lower for PGY-1 residents compared with others. Total score and overall performance correlated with the number of laboratory procedures previously performed (r = 0.658 and 0.641, respectively; P = .0001), and after excluding 2 outliers, global ratings and procedure-specific checklists showed an upward trend with advancing PGY. Median total scores are shown in the figure. ICC A1 was high at 0.95, suggesting assessors' scores correlated with each and also “agreed” on an absolute score (table 1). The ICC A1 for 6-point, 9-item DOLS having was 0.64 (95% CI 0.54–0.75), and improved to 0.67 when “professionalism” was deleted and to 0.70 when “professionalism” and “biosafety” were deleted.

figure

Median Total Scores

figure

Median Total Scores

Close modal
table 1

Summary Report of Item Scores and Interrater Reliability of Each Item on a 6-Point DOLS

Summary Report of Item Scores and Interrater Reliability of Each Item on a 6-Point DOLS
Summary Report of Item Scores and Interrater Reliability of Each Item on a 6-Point DOLS

Internal consistency calculated by Cronbach's alpha was high at 0.989 for overall competency.

Cohen's d was computed as 1.64, which is a large difference in a practical sense. A G coefficient of 0.852 was obtained when 2 procedures were evaluated by 1 assessor, although resident competency was case-dependent. Residents provided the major source of variance (73% of total variance; table 2) suggesting DOLS can differentiate residents' procedural skills, while variance associated with assessors was small (0.01%).

table 2

Estimated Variance Components for DOLS with Residents, Assessors, and Lab Tests as Random Factors

Estimated Variance Components for DOLS with Residents, Assessors, and Lab Tests as Random Factors
Estimated Variance Components for DOLS with Residents, Assessors, and Lab Tests as Random Factors

Use of DOLS was rated highly by the majority of residents. Faculty assessors suggested setting anchors for scores of 5 and 6, and reducing the procedure checklists to a few critical steps.

In our pilot study, an assessment tool for hematopathology residents (DOLS) showed a moderate correlation between experience and PGY level and scores, and a marked difference for the groups with and without prior experience. This is consistent with a systematic review that showed an association between training level and scores.6 

Technical ability had the highest ICC, and biosafety and professionalism had the lowest. G study data for DOLS showed a reliability of 0.931 when 5 laboratory procedures were observed by 1 assessor, comparing favorably with G coefficients of 0.81 and 0.80 for DOPS ratings of colonoscopy and surgical skills using 2 and 4 assessors, respectively.7,8  Our focus was on limiting burden for assessors, as the feasibility of DOLS would decline with increased faculty time and effort. Residents were the highest component of variance, while assessors were the lowest, with both suggesting the DOLS differentiated residents' technical approach with high agreement among assessors scoring the same encounter.

For feasibility, total time for assessment and feedback at 20 minutes was comparable to DOPS. Residents were satisfied with evaluation, and it was perceived as an opportunity for direct learning from faculty.

Limitations of the study include a small sample, a limited number of assessors, the single site format, and the fact that evaluators knew residents' PGY level.

DOLS is a novel tool for formative evaluation of practical skills in hematopathology residents. There is a need for validated tools to offer robust assessments and actionable feedback to trainees. Our approach to develop the DOLS tool may be useful to other specialties to develop work-based assessment tools.

1
Hallworth
MJ.
The '70% claim': what is the evidence base?
Ann Clin Biochem
.
2011
;
48
(
part 6
):
487
488
. doi:.
2
Norcini
J,
Burch
V.
Workplace-based assessment as an educational tool: AMEE Guide No. 31
.
Med Teach
.
2007
;
29
(
9
):
855
871
. doi:.
3
Cohen
SN,
Farrant
PB,
Taibjee
SM.
Assessing the assessments: UK dermatology trainees' views of the workplace assessment tools
.
Br J Dermatol
.
2009
;
161
(
1
):
34
39
. doi:.
4
Finall
A.
Trainers' perceptions of the direct observation of practical skills assessment in histopathology training: a qualitative pilot study
.
J Clin Pathol
.
2012
;
65
(
6
):
538
540
. doi:.
5
Leung
E,
Dix
D,
Ford
J,
Barnard
D,
McBride E. The pediatric hematology/oncology educational laboratory in-training examination (PHOELIX): a formative evaluation of laboratory skills for Canadian pediatric hematology/oncology trainees
.
Pediatr Blood Cancer
.
2015
;
62
(
11
):
1952
1955
. doi:.
6
Jelovsek
JE,
Kow
N,
Diwadkar
GB.
Tools for the direct observation and assessment of psychomotor skills in medical trainees: a systematic review
.
Med Educ
.
2013
;
47
(
7
):
650
673
. doi:.
7
Barton
JR,
Corbett
S,
van der Vleuten
CP;
English Bowel Cancer Screening Programme; UK Joint Advisory Group for Gastrointestinal Endoscopy. The validity and reliability of a Direct Observation of Procedural Skills assessment tool: assessing colonoscopic skills of senior endoscopists
.
Gastrointest Endosc
.
2012
;
75
(
3
):
591
597
. doi:.
8
Marriott
J,
Purdie
H,
Crossley
J,
Beard
JD.
Evaluation of procedure-based assessment for assessing trainees' skills in the operating theatre
.
Br J Surg
.
2011
;
98
(
3
):
450
457
. doi:.

Author notes

Editor's Note: The online version of this article contains Direct Observation of Laboratory Skills (DOLS) questionnaire, marking scheme for DOLS, procedure-specific checklist, and resident feedback for formative assessment.

Funding: The study was funded by the Aga Khan University Scholarship of Teaching and Learning (SoTL) Grant.

Competing Interests

Conflict of interest: The authors declare they have no competing interests.

The authors would like to thank research assistants Zahra Nur Khaled, MD, and Sobiya Sawani, MD, for collection and analysis of data.

Supplementary data