Influence of Examiner Experience on Clinical Performance of Visual Inspection in Detecting and Assessing the Activity Status of Caries Lesions

Visual examination is the most commonly used method for caries lesion detection because it is an easy technique that is routinely performed in clinical practice.¹  The method has demonstrated high specificity, and it is the only validated method used to assess the activity status of caries lesions.^1-3  However, it has presented low sensitivity and reproducibility,^1,4  the latter because of the subjectivity of the procedure.^5,6 

In this context, attempts to improve the accuracy of caries detection methods have been proposed,⁷  such as the development of scoring systems. The International Caries Detection and Assessment System (ICDAS), based on visual inspection, allows for the accurate recording of the severity of carious lesions, from noncavitated stages to frank cavitation.⁷  The system was developed for use in clinical research, in clinical practice, and for epidemiological purposes.⁸  However, for caries activity assessment, ICDAS must be used in combination with other procedures.⁷ 

One of the adjunct methods for caries activity evaluation is based on the examiner's interpretation of the clinical features of the lesions.⁹  For this approach, the examiner evaluates some clinical parameters that are more often associated with active or inactive caries lesions (clinical characteristics assessment [CCA]). Another option is the lesion activity assessment (LAA), which is used in association with ICDAS and is performed by assigning numerical values (points) to three clinical parameters: visual appearance, whether the lesion is in a plaque stagnation area, and surface texture. The sum of these three independent scores is then used to determine whether the lesion is active or arrested.¹⁰ 

These two methods are established on the basis of different aspects of cognitive processes involved in diagnostic reasoning in clinical settings. The first method, based on the evaluation of the clinical features of the lesions (CCA), represents a descriptive approach to the decision-making process. The examiner evaluates the clinical features of the lesions and mentally weights each characteristic to reach a decision about the lesion activity. On the other hand, the assignment of numerical values for different clinical parameters to reach a decision concerning caries lesion activity is part of the prescriptive theory. Predetermined values are attributed to each clinical parameter, and the sum of the values indicates whether the lesion is active or inactive.

The descriptive theories about decision making are closer to clinical reality because clinicians are more accustomed to drawing on past experience to reach a diagnosis.¹¹  Moreover, this process is less time consuming and more practical to perform as part of a daily clinical routine.^11,12  Nevertheless, this process is greatly influenced by the professional's experience. Novice clinicians use a hypothetic-deductive reasoning strategy to reach a decision. This is a stepwise process and therefore consumes more time. On the other hand, experts frequently use cognitive shortcuts to make diagnoses, a process called heuristics.^11,12  Another limitation is that unchecked heuristic reasoning can lead to cognitive bias and diagnostic errors.^11,13 

Our working hypothesis was that the method based on CCA of lesions would be less time consuming but more greatly influenced by the examiner's experience regarding its reliability and accuracy. Conversely, the LAA method would present similar accuracy and reproducibility independent of the examiner's expertise. To test this hypothesis, we aimed to evaluate the performance of three groups of examiners with different levels of experience in assessing caries lesion activity using two different approaches associated with ICDAS. We also investigated the influence of the examiner's experience on ICDAS scoring.

This study was approved by the Committee for Ethics in Research, Dental School, University of São Paulo. Each participant's guardian signed a positive consent form.

The inclusion criteria were children with primary or mixed dentition and who had sought dental treatment at our school. Thus, eighteen children, aged three to eight years old, who presented at the Dental Clinic of the Dental School, University of São Paulo, were selected.

The assessments were performed by six examiners with different levels of experience: two undergraduate dental students, two specialists in pediatric dentistry, and two graduate students. The two undergraduate students were in their last year of study at the School of Dentistry of the University of São Paulo. The specialists had completed the specialization course at the same school, and they had at least four years of experience. The graduate students were enrolled in the PhD course in Pediatric Dentistry at the School of Dentistry of University of São Paulo, and they had at least six years of experience. Furthermore, only the graduate students had experience in using ICDAS in previous research.

The examiners were randomly divided in two groups (one dental student, one specialist, and one graduate student per group) using a coin toss. One group was first trained to use the method based on CCA of the lesions, and the other group used the LAA system. After the first series of examinations, the criteria were reversed.

Previous training sessions were conducted by two benchmark examiners (MMB and FMM). The training consisted of a lecture about ICDAS and about the adjunct activity criteria. Different sessions for each group were established. Then, hands-on training was conducted using 20 extracted primary teeth. Immediately after examining each tooth, the participants reviewed their scores by comparing them with the scores of the experts. There was a discussion about doubts and difficulties in carrying out the examinations. To avoid bias, each group of examiners underwent independent training using only one system of activity assessment. On a second occasion, 15 days later, the group that had used one index was trained to use of the other index, using the same methodology. The system used by each group for the first examination was randomly selected.

After the first training session, the examinations were performed. Before the examination, the teeth were carefully cleaned with a rotating bristle brush and a pumice/water slurry. Visual inspection was performed with the subjects positioned in a dental unit with operating light illumination, a 3-in-1 syringe, plane dental mirror (Duflex, Rio de Janeiro, Brazil) and WHO periodontal probe (Hu-Friedy, Rio de Janeiro, Brazil).

The surfaces of all of the teeth in the oral cavity, both primary and permanent teeth, were the experimental units. The criteria used were ICDAS and adjunct methods of activity assessment: CCA and LAA, depending on each group.

The approach based on the CCA of the lesions considered several clinical aspects, such as opacity, loss of luster, roughness, and location of the lesion (plaque stagnation area or not). The clinical characteristics associated with active or inactive caries lesions, according to different ICDAS scores are described in Table 1.

The other system was based on assigning different values for the lesion according to three characteristics: ICDAS score, whether it occurred in plaque stagnation area, and surface texture. After scoring the lesion, the examiners totaled the points and classified the lesion as active or inactive. A detailed description of the method is presented in Table 1.

First, one system was used for each subject according to the group of examiners. Fifteen days after the end of these examinations, other training sessions were held, and the group that performed the evaluation of activity using CCA of the lesions was trained to use the other system (LAA), and the other group was trained to perform the evaluation based on CCA. Then, at another appointment, the examiners blindly re-examined all of the children using the system that they had not used in the first examination. The evaluation was performed independently by each group of examiners during different weeks, applying the previous training to the criteria of the activity. The examiners were unaware of their own scores and of each other's scores. The duration of each examination was measured using a digital stopwatch for all of the examinations.

After all of the examinations, two senior lecturers who had experience in caries activity assessment and in using ICDAS also examined the children in a joint session, and the consensus of these examiners was considered the reference standard for all of the surfaces. These benchmark examiners (MMB and FMM) reached a decision about ICDAS score for each surface, and they used both methods together for caries activity assessment (LAA and CCA). Then, they decided only whether the lesions were active or inactive. These results were considered the reference standards.

Concerning the activity assessment, both systems, LAA and CCA, were compared by calculating intraclass correlation coefficients (ICC), sensitivity, and specificity using the results of benchmark examiners as references. Sensitivity and specificity values were compared using multilevel Poisson regression analyses. For these analyses, we considered five different levels in the following hierarchical order: examiner (level 5), patient (level 4), teeth (level 3), tooth surface (level 2), and evaluation (level 1). The outcomes for specificities and sensitivities were, respectively, false-positive (FP) results (true negative as a reference) and false-negative (FN) results (true positive as a reference). The independent variables were the method of caries activity assessment, the level of the examiner's experience, and the order in which the method was performed. When the analyses found a statistically significant association, we calculated prevalence ratios (PR) and respective 95% confidence intervals (CIs) for false FP or FN results. Appropriate statistical software was used for the multilevel analyses (MLWin 2.10, Centre for Multilevel Modeling, Bristol, UK).

The durations of the examinations were compared using two-way analysis of variance for repeated measures. With regard to the ICDAS scores, the ICCs were calculated by comparing the results of the benchmark examiners and the results of examiners with different levels of experience. Sensitivity and specificity were calculated at two thresholds: all lesions (ICDAS ≥ 1) or cavitated lesions (ICDAS ≥ 3). Multilevel Poisson regression analyses were also performed to compare the results regarding the levels of the examiners' experience. A receiver operating characteristic (ROC) analysis was also performed at both thresholds, and the areas under the curves were calculated. The level of significance was set at p<0.05.

Intra- and interexaminer agreement were calculated using weighted kappa statistics, considering the ICDAS scores for the examiners with similar levels of experience.

Overall, 11 (61.1%) boys and 7 (38.9%) girls, with a mean age of 5.3 years (standard deviation [SD] = 1.4), participated in the study. A total of 1734 surfaces of primary and permanent teeth in 18 children were examined. The children presented with dmf-t plus DMF-T ranging from 2 to 13 (mean ± SD = 8.2 ± 3.7). Considering the reference standard examination, 1264 surfaces (72.9%) were classified as sound, 32 surfaces (1.8%) were given an ICDAS score of 1, 117 (6.7%) were given a score of 2, 38 surfaces (2.2%) were given a score of 3 of ICDAS, 7 (0.4%) were given a score of 4, and 80 (4.6%) and 136 surfaces (7.8%) were given scores of 5 or 6, respectively. Moreover, 39 surfaces (2.2%) were restored, and five teeth (21 surfaces, 1.2%) were extracted because of extensive caries lesions. Considering the carious lesions, 326 were classified as active lesions, and 84 were inactive.

Therefore, we considered the sound surfaces plus inactive caries lesions (n=1348) versus active caries lesions (n=84) to obtain the data about caries activity assessment methods (Table 2). For the analysis related to the ICDAS scores (Table 3), lesions classified as score 1 to 6 of ICDAS were considered decayed (n=410) and sound surfaces (n=1264) as nondecayed at the all-lesions threshold. At the cavitated lesions threshold, surfaces scored 3 to 6 of ICDAS were considered decayed (n=261) and surfaces classified as 0, 1, or 2 of ICDAS nondecayed (n=1413).

Regarding the assessment of activity (Table 2), LAA showed similar or slightly better ICC than CCA compared to the classifications of the benchmark examiners. However, independent of the method of activity assessment and of examiner experience, the criteria used in the first examinations had significantly higher sensitivity values (PR, FN; 95% CI = 1.43; 1.10-1.87; p=0.008), but significantly lower specificity values (PR, FP; 95% CI = 0.54; 0.31-0.93; p=0.011).

Furthermore, the examinations performed by dental undergraduate students were of significantly longer duration than those performed by the more experienced examiners. For all of the examiners, assessments using LAA criteria were significantly longer, varying from 12% (specialist) to 20% longer for the graduate students (Table 2).

With regard to the comparison with the reference standards, similar ICC values were observed among the groups of examiners (>0.8). At the cavitated threshold, the undergraduate students and specialists, in general, showed higher sensitivity and lower specificity than the graduate students. In detecting all caries lesions, however, graduate students showed a slight increase in sensitivity, but the areas under the ROC curves were similar at the two thresholds in the different groups of examiners (Table 3).

Regarding the ICDAS scores, we found that the inter- and intraexaminer kappa values were high, independent of the examiners' clinical experience (Table 4).

Because visual inspection is a subjective method for detecting caries lesions and for assessing their activity, the examiner's experience can influence the accuracy of the method. Two methods of assessing the activity of caries lesions have been published for use associated with the ICDAS.^9,10  One method (CCA) is based on a descriptive theory of cognitive processes, while the other is based on prescriptive theories.^11,12  Therefore, we aimed to investigate the influence of examiners' experience on the assessment of the activity caries lesions using these two different approaches and on the detection of caries lesions using ICDAS. Visual scoring systems for caries detection, such as ICDAS, are also based on a prescriptive theory of clinical decision making.¹¹ 

With regard to caries activity assessment, we observed that examinations based on the numerical scoring of lesions took longer than examinations using the other method. Furthermore, for both methods, the more experienced examiners performed the examinations more quickly than the undergraduate students. These findings partially proved our hypothesis because methods based on prescriptive theories are usually more time consuming. Moreover, it would be expected that examinations performed by inexperienced clinicians would be of longer duration because they are used to employ hypothetical-deductive processes in comparison with examinations performed by more experienced dentists.^11,12 

Concerning performance, the LAA method showed a slightly higher ICC than the method using the clinical features of lesions. This finding could be due to the more objective nature of the LAA, while the method of individual interpretation is more susceptible to subjectivity. However, this superiority was not significant. This finding is corroborated by the sensitivities and specificities obtained by both methods.

With regard to the examiner's experience, there was no significant influence of either method. One interesting finding concerned the order of the methods for assessing caries activity (at the first or second session). This order was actually more important than the method used. The criteria used for the first examination presented higher sensitivity, while the second session of examinations had higher specificities. This finding could be explained by the extensive training of the examiners in assessing the caries activity statuses of the lesions. In our study, when the examiners performed activity assessments at a second session, they had already carried out the examinations previously. Therefore, they were more highly trained than the first time. They could have maintained a residual effect from the first method, particularly because both methods for caries assessment consider similar clinical characteristics.

The similarity between the methods could explain the absence of an influence of the examiner's experience on performance. The small number of evaluators can be considered a limitation of our study, as only two examiners at each level of experience were used. This choice was made because a larger number of examiners would imply a more lengthy assessment for each child, making it more tiring and stressful for the young children who participated in this study. Furthermore, it would have increased the number of sessions, which could have led to dropouts.

Considering caries detection, previous studies—conducted before ICDAS had been proposed—found significant differences related to the dentist's experience. In a study in which caries lesions were evaluated according to the examiners' interpretations, with no use of scoring systems, less experienced dentists more accurately performed both the visual and the exploration methods.¹⁴  This is an example of a descriptive approach to reaching a decision. It is likely that the less experienced examiners used hypothetic-deductive processes and that the experts used heuristic processes, which are more susceptible to bias. Conversely, other studies have shown that undergraduate students, performing visual inspection with no scoring system, yield lower specificity and reliability.^15,16 

We observed in the present study that the level of experience of the examiners had little influence on the performance and agreement of caries detection processes using ICDAS. These different results obtained with ICDAS and in the previous studies^14-16  were probably the results of the different approaches used to perform caries detection. It would be expected that the results with ICDAS would be less influenced by the examiners because a detailed description of each condition was provided for the examiners, as observed in our study. Another study using ICDAS corroborated our findings.¹⁷  Similar results were obtained when the examiners used objective methods, such as the laser fluorescence method.¹⁵ 

In fact, one goal of using visual scoring systems is the minimization of the inherent subjectivity of clinical examinations.¹  Nevertheless, although the use of ICDAS and other visual scoring systems seems to be logical, clinicians (mainly more experienced ones) think differently from each other in clinical settings. They are accustomed to drawing on past clinical experience to make a diagnosis, and therefore they can be more resistant to learning about prescriptive methods of clinical examination. Therefore, several attempts should be made to teach dentists how to perform ICDAS to improve their diagnostic accuracy and to standardize the caries diagnostic process throughout the world.⁷  The ICDAS e-learning program could be an alternative because it has improved the performance of dental students in detecting occlusal caries lesions.¹⁸ 

This was the first study to investigate methods of caries detection and activity assessment considering the different approaches of the theories behind clinical decision making. More studies should be performed to evaluate how clinicians are accustomed to making their diagnoses with regard to caries lesions and to propose more realistic methods with sufficient accuracy and reliability.

In conclusion, the examiner's experience does not significantly influence the performance of visual inspection for the detection and assessment of caries lesion activity after extensive training. Furthermore, both methods of caries activity assessment demonstrate similar validity, but the LAA method is more time consuming.

The authors certify that they have no financial or other personal interest in any product, service or company mentioned in this article.