College of American Pathologists Cancer Protocols—Getting It Just Right

To the Editor.—We would like to thank Nakhleh et al¹  for their recent editorial acknowledging concerns with the College of American Pathologists (CAP) cancer protocols and describing plans to address those concerns more fully through the 3 CAP resource committees that have been tasked with creating and maintaining the protocols thus far and the newly formed Cancer Protocol Oversight Project Team. The CAP and members who have worked to develop and promote the use of the cancer protocols and the associated checklists can be justifiably proud of the improvements they have brought to cancer reporting by pathologists. The use of checklists and structured data in medicine can have great potential benefit in the areas of patient safety and quality improvement.

However, we agree it is now appropriate to take stock of and address the unintended consequences or potential “collateral damage” that can also result from “too much of a good thing.” We have become more acutely aware of some of those issues because electronic reporting of cancer diagnoses will become a state-mandated requirement for all pathologists in California in 2019 (assembly bill 2325; California Legislature²), and the California Department of Public Health (the agency responsible for implementing this requirement for the state) is currently engaged with us in rulemaking about how these diagnoses are to be reported, what checklist or data elements are required, etc.

We agree with the concerns cited by Nakhleh et al,¹  including growth in the number of data elements, added time needed to sign out a case, lack of consistency among protocols, lack of clarity regarding accreditation requirements, and vendor variability in implementing electronic formats of the checklists. In the remainder of this communication, we would like to expand on some of these issues, share additional concerns, and offer suggestions.

Our principle concerns include the following issues:

• Paradoxical Increase in Error Rates and Negative Impact on Patient Safety.—This can result from the use of checklists that are too lengthy because they include elements for which the evidence base or clinical utility is poorly established or because too many “optional” elements are included to satisfy diverse constituencies. Inconsistent design and placement of elements or poor implementation by vendors (from a usability standpoint) can also contribute to error. If checklists increase the time and complexity of signing out a case, they also, arguably, increase the “cognitive load” on pathologists and, therefore, the likelihood of making an error, either on that case or on another they are signing out on days made more fatiguing by the cumulative increase in workload per case. One small study posits a correlation of checklist length with amended reports as a proxy for error.³  Although it has not otherwise been studied in pathology, to our knowledge, the effect of cognitive load on increasing medical errors has become a concern in other specialties and in nursing, as has the role of information system design factors in producing cognitive load.⁴  “Psychologic and sociologic studies have shown that in a shared context, concise, unconstrained, free-text communication is most effective for coordinating work around a complex task. Attempting to require professionals to encode data, or enter data in more structured formats, can be fruitful and is necessary for research or managerial purposes but does not come without a cost.”^5(p106) Even the decision not to use an “optional” item is still a cognitive task.

• Adoption of CAP Protocols and Checklists as the “De Facto” Standard by Other Organizations.—This was noted by Nakhleh et al¹  and is a testament to the value placed on this CAP product by other organizations. However, this places an additional burden on the CAP to ensure that the protocols and checklists are not misused by maintaining much closer liaisons with those organizations and greater control over their use of the CAP-produced protocols and checklists. Synoptic reports started out as valuable aides to oncologists, improving the readability of the pathology report by providing, in one location, a limited list of key tumor characteristics necessary for staging the patient, assessing prognosis, and guiding therapy. The CAP should be vigilant that this remains the goal—it is a slippery slope when government entities, accrediting bodies, hospitals, health plans, and others decide to use CAP protocols and checklists for research, “quality” assessment, payment policy, and other purposes for which they were not intended.

• Poor Implementation of the CAP eFRM (Electronic Forms and Reporting Module) Code by Laboratory Information System and Electronic Medical Record Vendors.—This is an area in which the CAP must find more effective alternatives than to “encourage” pathologists to work with their vendors and to “hope … that LIS vendors will be better able to meet the needs of their customers.” ^1(p2) All vendor implementations originate with the CAP checklists—the CAP should not leave this critical downstream piece of the solution up to vendors (who have no incentive to address safety or usability in their implementation of checklists) or to pathologists (who often have no leverage with vendors or may practice in settings in which multiple vendors' products are in use). We recommend that the CAP investigate creating a vendor certification regime that assesses synoptic or checklist interface features, such as patient safety (readability, likelihood of accidentally checking the wrong box, etc), usability by pathologists (cognitive load), and the ease and cost of implementation, upgrading, updating, and maintenance, etc. Only when vendor product deficiencies can be made public or graded in some way is there “hope” that the pathologist will have the leverage to insist on improved implementation.

• “Structured Data” Should Not Be an End in Itself.—Although many will agree there is potential value to the mining of structured data, the pathology report should not be allowed to devolve from a “medical consultation” into a “laboratory test” or the role of the pathologist to be converted from “the doctor's doctor” (a valued medical consultant) into a data-entry or data-aggregator technician. The current CAP checklists have unwittingly fostered that transformation by attempting to include or repeat too many data elements that properly belong elsewhere in the report or patient record; by “overquantifying” what are subjective or, at best, semiquantitative observations (giving the false impression of precision to an inherently imprecise observation); and by scattering free-text fields among other binary or mutually exclusive data element fields. This latter chops up the narrative “story” embodied in the pathology report, diminishing its value as a medical consultation. These factors further reduce readers' appreciation of the subjective and interpretive nature of much of the practice of pathology, leading to the impression that what may sometimes be “grey” should always be “black and white.” Perhaps most important, and little discussed, is that with rapid advances in artificial intelligence and natural language processing, these very expensive and nonstandardized data sets we are now constructing may become an interim step of ultimately limited value.

• The CAP Needs More Robust Methods of Not Only Creating the Protocols and Checklists but Also Testing and Validating Them Before Release.—Recognizing that these protocols and checklists have indeed become the de facto standard in our field and are no longer optional, but required of all pathologists, it is imperative that the CAP committees tasked with preparing them have appropriate support in their creation and design from researchers, statisticians, and human factors engineers—a small group of volunteer pathologists with limited staff support will produce the less-consistent products we have seen—but pathologists no longer have the option of not using a poorly designed product. The CAP currently obtains feedback on the protocols and checklists by posting them on a Web site, with an open comment period in advance of release, or waiting for complaints to be directed to them once released. The CAP should explore ways to alpha and beta test iterations of new or revised checklists and protocols that replicate real-world implementations, rather than depending primarily on voluntary responses from those with a heightened interest in a specific protocol or checklist.

These comments in no way diminish our appreciation for the work that has been done and the true value derived from the protocols and checklists developed during the past few years by the CAP and committee members. Those products represent tremendous work and a tremendous advance. We can now all practice with greater scientific rigor and currency because of the availability of those CAP products.

However, a threshold has been crossed because what started out as a valuable but optional “assist” is now a requirement for all pathologists, significantly affecting our cost and efficiency of practice. Judgments about the “quality” of our practice are going to be made and, in some cases, pathologists' livelihoods may be determined based on how and whether those protocols and checklists are used. We are very heartened to read that this is recognized by the CAP and support you in taking the pause for assessment and redirection that was outlined by Nakhleh et al.¹