Essential Foundational Education Research Methods Articles for Graduate Medical Educators: A Delphi Study

While most graduate medical educators have some research training, the nuances of education research can differ from research in epidemiology and from clinical research commonly taught in medical school and residency.^1-3  Additionally, as educators transition from consumers of the literature to researchers conducting their own scholarship, questions arise about education research methods. Thus, educators may seek resources to guide them in developing, implementing, and disseminating education research projects.

The literature contains resources on a wide range of research topics, but graduate medical educators may struggle to find foundational methods articles that are practical and applicable to medical education problems. Specifically, there is a need for articles that provide a general overview of education research methods, or “primers,” that are useful to researchers with a diverse range of experience. Mentors often have their own files of helpful resources but may be missing articles that could be useful to mentees and collaborators.

This study aimed to comprehensively review the medical education research methods literature and perform a modified Delphi study to identify the most useful foundational articles for both novice and experienced education researchers. The resultant list of articles may help guide graduate medical educators in conducting studies and may also serve as a resource for faculty development and graduate medical education training in education research methodology.

We applied a conceptual framework based on the theory of spiral curricular learning, which posits that trainees should revisit basic concepts repeatedly, building upon existing knowledge with iterative cycles of more detailed learning.⁴  This lens influenced: (1) our aim of identifying articles that would both reinforce general concepts and provide more detailed content for deeper learning; and (2) our modified definition of “expertise” for our Delphi panel to include a range of education research experience.⁵  We assembled an author team of 10 panelists representing 3 institutions, 4 specialties (surgery, emergency medicine, anesthesiology, and infectious disease), and varied academic ranks (1 education fellow, 3 assistant professors, 5 associate professors, 1 professor) with education research experience ranging from novice to expert to participate in the consensus-building process, as well as one nonvoting medical student who administered the surveys, tracked participation, and prepared anonymous summaries of the results for each round.

An Ovid MEDLINE database search was conducted by 2 authors and a research librarian in October 2020 on foundational medical education research methods articles from 2000 to 2020. Our search strategy included both text and index search concepts related to faculty, medical education, physicians/medical staff, and methods. The search yielded 120 articles, which were independently reviewed by 2 authors for inclusion/exclusion criteria, with discrepancies resolved by discussion. Articles primarily about general education research principles (ie, overviews, or primers) were included; articles were excluded if they were not focused on medical education research methods, were published prior to the year 2000, were written in a non-English language, or had no available abstracts. Additional articles were identified from the authors’ files and by manually checking the references of all identified articles, and these also underwent 2 author inclusion/exclusion criteria screening.

Each round of voting for the Delphi process was completed online using Google Surveys (Google). Participants provided their names at the beginning of each round to track participation, but this information was reviewed only by the nonvoting member of the team who prepared the anonymous results for group review and discussion after each round. One panelist was also an author on an included article and did not screen nor vote on that article in any round. The first round prompted participants to read each article and rate it on a Likert-type scale from 1 to 7 (1=unlikely to be useful; 7=illuminating, highly useful). Each article needed to meet an a priori threshold of 3 on the 7-point scale to advance to the next round. The second round asked panelists to review results from round 1 and indicate whether each article belonged on a “must-read” list of foundational education research articles (response options were “no,” “maybe,” and “yes”). Articles with 80% of panelists selecting no were excluded; those with 80% of panelists selecting “yes” were included on a final list of articles, and the remaining articles were carried through to the third round. The final round presented both the round 2 results and links to the included articles thus far, and asked participants to vote “yes” or “no” to include each remaining article on the must-read list. Articles voted by 80% of the panel as “yes” for inclusion were added to final list (Figure). Descriptive statistics were calculated at each stage using Google Sheets. This study did not qualify as human subjects research and therefore did not require institutional review board review at the primary institution.

Fourteen of 120 articles initially identified from the literature search, and 11 of 33 additional articles identified from the authors’ files and bibliographic review of included articles met inclusion criteria, yielding a total of 25 articles.^3-28  Nine of 10 panelists participated in the first round proceedings, and all 10 panelists participated in rounds 2 and 3.

In the first round of our modified Delphi proceedings, one article was eliminated. In the second round, 8 articles were eliminated, and 7 were included in the list of essential articles. In the final round, 2 more articles met the threshold for inclusion for a final list of 9 essential articles (Figure, Table).

Our modified Delphi study identified 9 foundational education research methods articles that met consensus as essential resources for education researchers. The selected articles were published between 2003 and 2019 and included topics spanning research question development to study design and analysis to result presentation and dissemination.

Medical education research requires rigorous application of research principles with attention to education-specific considerations. This includes a thoughtful approach toward identifying a research problem and delineating a knowledge gap, appropriate research methods and outcomes, and transparent and rigorous data analysis and presentation of results. The identified articles highlight key concepts and may serve to deepen understanding of methodology as each article is likely to reinforce some key principles of the others as well as provide some new information that can build upon the researcher’s evolving foundational knowledge.

Graduate medical educators may find these articles to be a helpful starting point for their own faculty development and may also use this list for teaching and mentoring education research. Because there is no Medical Subject Heading (MeSH) term for “education research,” and because excellent foundational articles can be found in a wide variety of health professions and specialty-specific journals, this curated list incorporates a broad literature search which was augmented by hand-checking personal files as well as reference lists of all identified articles. Similar methodology may be applied to future efforts to identify key medical education research articles in specific niches.

This study is limited by omitting articles published since 2020 as well as those without an abstract. Also, while the Delphi panel was intentionally created to include a range in experience levels and specialties, it is unlikely to be representative of all education researchers, and readers may find the articles included to have variable utility. As we used the Delphi method to build consensus among our author team in this study, the panel and author team were identical (except for the one nonvoting author). This method has been used in similar studies but may lead to conflicts of interest between the roles of author and panelist.²⁹  Additionally, Delphi studies require the team to make decisions about the thresholds for consensus and formats of iterative rounds of voting. Although we relied on both established methods for modified Delphi studies as well as thoughtful consideration of the aim of each round of voting when developing our surveys, the study may have fallen short of our goal of identifying the true consensus regarding the most useful resources. To further foundational learning in medical education research, next steps may include developing an online compendium of annotated resources that could be revised periodically, as well as including additional experts for the rating process.

Our modified Delphi study of foundational education research methods articles identified 9 articles that stood out to our panel as the most useful resources for medical education researchers. This list may be used by graduate medical educators who are seeking methods resources, as well as program directors and mentors seeking to share helpful resources with trainees and colleagues.

The authors would like to thank Andrew Hamilton, MS/MLS, for his contributions as the study’s Health Science Education and Research Librarian, and Jacob Yarris, BA, for his graphic design assistance for the article’s figure.

Funding: The authors report no external funding source for this study.

Conflict of interest: The authors declare they have no competing interests.