Teaching and Assessing Systems-Based Practice: A Pilot Course in Health Care Policy, Finance, and Law for Radiation Oncology Residents

Through its Outcome Project, the Accreditation Council for Graduate Medical Education (ACGME) expects programs to shift the emphasis of resident education from structural aims to outcome goals. As part of the accreditation process, residency programs are now required to provide data on specific educational outcome measures and provide evidence for how this information is used to improve graduate medical education. The Outcome Project assesses 6 core competencies, which the ACGME introduced in 1999 and added to its common program requirements for core programs in 2002.1 Residents are expected to demonstrate competence in systems-based practice by understanding different types of medical practice and delivery systems including cost containment, practicing cost-effective medicine and resource allocation, advocating for and assisting patients in navigating the complexities of the health care system, and partnering with managers and other providers to coordinate care.2 The objectives for systems-based practice state that, “Residents must demonstrate an awareness of and responsiveness to the larger context and system of health care, as well as the ability to call effectively on other resources in the system to provide optimal health care.”1 Residents also are expected to function in different health care settings, practice cost-effective medicine, advocate for their patients and their profession, identify system errors, and take an active role in quality improvement.2 

Formal training in systems-based practice has traditionally been absent from medical education.3 Several novel programs designed to fulfill the objectives outlined by the ACGME have been described, including a university-wide graduate medical education core curriculum, multidisciplinary team meetings focusing on improving specific outcomes measures, and outcome cards detailing clinical cases involving each resident.4–6 Barriers to integrating health policy into graduate medical education include limited time, large volumes of medical knowledge to be covered, a lack of expertise, and residents' perceptions that learning health policy is less important than their other requirements.7 Harris et al8 reported the results of a survey of radiation oncology program directors regarding challenges faced in implementing the ACGME Outcome Project requirements. Of those surveyed, 56% reported having too little time and 45% reported having inadequate knowledge of the core competencies.

To function effectively within the broader health care system, physicians need a working knowledge of how United States health care is organized, funded, and guided by ethical and legal principles. Through our own research and discussions with health policy experts, we identified a set of topics necessary for a basic understanding of how health care is organized in the United States and how each topic applies to the field of radiation oncology. We then developed a pilot course in health care policy, finance, and law designed to fulfill the Outcome Project objectives for systems-based practice competency for radiation oncology residents. Our aims were to assess radiation oncology residents' baseline knowledge of and attitudes toward health policy; educate residents on basic topics of health policy, finance, and law; test the efficacy of the course; and involve residents in advocacy by having them attend the American Society for Radiation Oncology (ASTRO) Advocacy Day. This report describes our course and findings.

We created an online, case-based pilot course designed to teach radiation oncology residents the basics of health care policy, finance, and law. The course was created in an interactive format within the New York University School of Medicine Medical Informatics website. Lessons and course material were designed based on Outcome Project specifications related to teaching systems-based practice, recommendations from health policy experts, and our own research of current policy issues.

The pilot course consisted of 7 distinct learning modules. Each included at least 1 case study, a lesson covering a specific topic, and a quiz after the lesson. Embedded within each module were links to references and websites with additional information (figure). The modules covered:

1. Medicare and Medicaid,

2. employer-sponsored health insurance,

3. a case study on the impact of health care benefits on the US auto industry,

4. health care systems in Europe,

5. medical malpractice,

6. medical ethics law, and

7. managed care finance.

Prior to the course, participants completed a survey designed to assess their attitudes toward learning about health care policy, finance, and law, and a quiz to test their baseline knowledge. The quiz consisted of 34 multiple choice questions covering basic and advanced topics in health care policy, finance, and law, and was designed to stress important concepts within the course syllabus and modules.

The residents were given 2 months to complete the online module portion of the course (table 1). Following completion of the material, participants repeated the 34-question quiz. For the pre- and postcourse quiz, mean percentages of correct answers were calculated and 95% confidence intervals were generated. Statistical comparison of the 2 means, using a 2-sided, paired Student t test, was used to assess the efficacy of the course as an educational tool. The course and the efficacy analysis were reviewed and approved by the New York University Institutional Review Board.

In addition to the written and online portions of the course, residents also attended quarterly lectures on health policy issues. Invited experts discussed health policy issues important to radiation oncology, with an emphasis on topics relevant to residents and junior faculty. The residents also attended presentations given by local Medicare and Medicaid administrators.

An added objective entailed involving residents in the political process as advocates. During the 2 years that the course was offered, 3 residents attended ASTRO Advocacy Day in Washington, DC, where they received further education in health policy and advocacy and met with prominent radiation oncologists active in health policy and members of Congress and the congressional staff to discuss major health care issues facing the field.

Six radiation oncology residents completed the pilot course in health care policy. The group comprised 2 postgraduate-year 2 (PGY-2) residents, 1 PGY-3 resident, 1 PGY-4 resident, and 2 PGY-5 residents. All 6 residents attended the quarterly lecture series. Three of the 6 residents attended ASTRO Advocacy Day. Residents were given the opportunity to attend Advocacy Day during 2 consecutive years, and the same 3 residents chose to attend during both years.

All 6 residents completed the precourse quiz and health policy attitudes survey, and 5 completed the postcourse quiz (data for the 1 resident who did not complete the postcourse quiz were eliminated from the pre- and postcourse calculations). The mean score (percentage of questions answered correctly) on the precourse quiz was 64%. Following completion of the pilot course, the mean score increased to 84% (P  =  .05, 2-sided paired t test).

When the scores were analyzed based on subject matter, residents scored highest on health policy questions on both the precourse and postcourse exams. Prior to completing the course, residents scored a mean of 73% correct on health policy questions, and the scores improved to 93% correct following completion of the course material (P  =  .003, 2-sided paired t test). The mean precourse score for health care finance was 58%, and this improved to 77% on the postcourse quiz (P  =  .004). The mean precourse score for health care law and ethics questions was 50%, and the postcourse mean was 73% (P  =  .13).

table 2 shows residents' perspective on health policy prior to their participation in the course.

Radiation oncology program directors have a limited ability to comply with the Outcome Project due to lack of time and an inadequate knowledge of the ACGME core competencies.8 Standardized educational tools that include performance assessments such as ours could facilitate this knowledge. Through graduate medical education and professional societies, online educational tools such as this one could easily be shared and updated among all radiation oncology residency programs. For education outcomes that are not typically covered in clinical rotations but are required for accreditation, standardized courses with performance evaluations can help to educate residents and ease the burden of compliance placed on program directors. Although our course was specifically designed for radiation oncology residents, simple modifications would make it applicable to trainees in all fields of medicine.

One limitation of our study is that the sample size was very small. Radiation oncology programs are small, and this pilot course included all members of the New York University residency program. To further assess the efficacy of this course, it should be evaluated with a larger set of residents by expanding to either other departments within our institution or radiation oncology departments in other academic centers. Despite the small sample size, we were still able to document a significant improvement in knowledge of health policy. A second limitation is the inclusion of a quarterly lecture series and allowing half the residents to attend ASTRO Advocacy Day. These added educational offerings make it impossible to assess whether an online course alone is sufficient for educating residents in health policy. We caution that this report is not intended to prove the efficacy of an online health policy course alone, but rather to describe the feasibility of including a multifaceted policy course in resident education. Online courses can be updated and shared easily and residents can complete the course objectives at their leisure. However, it is our opinion that health policy education would ideally include experiences like expert lectures and activist participation, as in the course presented here. Finally, use of the same questions for the precourse and postcourse surveys raises the possibility that the residents recalled the answers more easily because they had previously seen and discussed the questions. Future versions of the course should include an externally validated assessment tool.

A course in health care policy, finance, and law can be integrated into a radiation oncology residency program. Through a case-based curriculum, residents can improve their knowledge of health care policy and apply that knowledge to their practice. In general, radiation oncology residents are motivated to learn about health care policy. Additional studies should be conducted to better describe the efficacy of different educational interventions to teach health policy, including online modules alone as well as approaches combining several educational formats. Finally, courses like the one we described here may assist programs in complying with the ACGME Outcome Project.