Common Pitfalls to Avoid in Qualitative Submissions to JGME

A conceptual framework is a way of thinking about a problem or study and connecting the work to prior relevant literature. It may draw from prior theories or evidence-based best practices to highlight different aspects of a problem or research question.²  Lack of a guiding conceptual framework is a common problem in medical education scholarship in general.^2,3  The conceptual framework must be described in qualitative research submissions and should be made explicit in the introduction.

For example, let’s imagine conducting a study of professional identity formation of wizards at Hogwarts School of Witchcraft and Wizardry.⁴  If we use the phrase “professional identity information” in our introduction, we may believe we’ve done enough to connect the reader to our understanding of the framework. However, we must do background work to develop the framework, and explicitly describe how our framework scaffolds the study. For instance, we may consider professional identity formation through a social-contextual approach, believing identity is formed and shaped through social interactions and relationships.⁵  The social-contextual approach may lead us to consider relationships, such as interactions with fellow wizards and professors, at the forefront of our study and research question. We may use social identity theory to explore how wizards experience their status in an “in-group” versus an “out-group.”⁵  Our entire framework would be markedly different if we interpret professional identity formation through another approach, such as an individualist approach. Here we might focus instead on individual reflection, to better understand a wizard’s state of mind, as opposed to the social context.

Some authors do not explicitly state the aim of the qualitative study: what is the research trying to achieve? The aim of the study should also be highlighted in the introduction, typically after establishing the conceptual framework and the current gap in understanding this area.

In our above example, we need to describe what we are trying to achieve by “studying” professional identity formation. How are we trying to better understand the phenomenon? Our aim could be to explore how professional identity formation is shaped through relationships with professors in longitudinal courses, or it might be to explore how identity formation is shaped through relationships with fellow students, in the same Hogwarts “House.” The aim sets up the choice of methods, and the introduction should also clearly illustrate why this aim is important, relevant, and necessary to further our understanding of the phenomenon of study.

A paradigm, also called a worldview or philosophy, is a researcher’s assumptions about the nature of reality.⁶  The methodology is the approach to acquiring knowledge or new understanding about that reality and should be guided by the chosen paradigm.^7,8  The paradigm and the methodology must align. An example of a malalignment between the paradigm and methodology in qualitative research is designing the project within a constructivist paradigm, with the assumption that multiple socially constructed realities exist, but then analyzing data in ways that align best with a post-positivist paradigm, with the assumption that there is only one true reality.^6,7  In the Hogwarts professional identity formation study, if we claim a constructivist worldview, it would be incongruent to align our data coding process with a post-positivist paradigm, ie, the assumption that only one correct codebook and application of the codes exists. Here’s an example of this faux pas in the methods section:

Under a constructivist paradigm, codes may be co-created, shared, and discussed, but coders achieving “agreement” often implies a single, objective truth, which would be inconsistent with a constructivist worldview.⁶ 

We often see methodological jargon sprinkled throughout qualitative submissions without a clear demonstration of how these methodologies were employed. A common culprit is “modified grounded theory.” Here, researchers cite their use of “modified grounded theory” without identifying which version of grounded theory they modified, or how it was modified.

For example, “We used a modified grounded theory approach. We conducted 3 focus groups with groups of student wizards (4 to 6 participants per group) at the end of each semester. No a priori codes were applied to the data; rather, all codes were generated from the data using a constant-comparison method.”

A more rigorous practice would be to identify which version of grounded theory was used (eg, traditional grounded theory, symbolic interactionalism, or constructivist grounded theory) and the modifications made.^9,10  In this example, if traditional grounded theory was used, we should be more explicit about its key tenets of iterative data collection and analysis, and theoretical sampling.

Qualitative analysis should not be a black box. Researchers must describe their analysis in adequate detail. Here is a vague description for example: “Two authors manually analyzed the Hogwarts students’ focus group transcripts that served as the qualitative data source. They used an inductive coding approach to identify emerging themes for preliminary analysis. Six themes emerged from the data.”

In addition to describing that the coding process was inductive, the researchers should describe how codes were generated and categorized, whether the process was linear or iterative, and how the research team members were engaged in coding. The paper should include a narrative of how the researchers scrutinized the data and constructed themes.¹¹  Note that in a constructivist paradigm, neither codes nor themes just emerge, like toast from the toaster! Graphic representation of the analytic process can also be used to enhance readers’ understanding. All parts of the methods—from sampling to analysis—must be coherently described, for clarity and credibility. If authors are concerned about the manuscript’s flow and word count, consider use of graphics and online supplemental information, which are not included in the total word count.

An improved methods section, for our Hogwarts example, might be, “Two authors collected and iteratively analyzed Hogwarts students’ focus group transcripts over the 2-month study duration. A priori codes were generated from social identity theory, and inductive codes were created in the process of transcript analysis. Codes were continually added and refined as additional transcripts were reviewed. Two authors met weekly over 2 months to review and modify the codebook, and to apply codes to the data. After completion of coding, all 4 authors met to cluster coded data into categories. Visual displays were used to illustrate connections between codes, and to inform construction of themes.”

Saturation is generally defined as when the researcher no longer needs to collect data because no new information is accruing from incoming data.¹²  The problem here is that many researchers state that saturation was reached without explaining how they knew this: “Interviews were conducted with student wizards until saturation was reached. We reached saturation after 12 interviews.”

If researchers use the concept of saturation, they should describe which model of saturation they are using. For example, theoretical saturation is rooted in grounded theory and occurs when the thematic categories are considered adequately developed.¹²  In contrast, inductive thematic saturation specifically refers to the lack of emergence of new codes or themes.¹² 

Because of limitations to the concept of saturation—including challenges in knowing when saturation is achieved (or if it can be achieved)—researchers may consider using alternative language of “data sufficiency” or “information power.”^13,14  Determining if a study has sufficient information power, based on the current sample size, depends on the study aim, sample specificity, use of established theory, quality of dialogue, and analysis strategy.¹³ 

For example, “Interviews were conducted with student wizards until data sufficiency was reached after 12 interviews. Data sufficiency was determined when the data were felt to be adequate to answer the research question, in the context of our previously described conceptual framework.”

The authors need to convince the reader that their results are rooted in the data and reflect research participants’ beliefs and viewpoints. For example, for studies using interviews or narrative reflections, the participants voices should be included, such as through direct quotes.

In our Hogwarts hypothetical research study, if we propose a theme that relationships with professors in longitudinal courses affect the experience of “in-group” versus “out-group” membership, but that relationships with peers can mitigate this effect, we should include quotes to demonstrate that this theme reflects the students’ views. For example, we may include a quote from Harry to explain how his experience in Professor Snape’s class, which pushed him to identify as part of the “out-group,” was mitigated by engagement in a community of practice with Ron and Hermione.

While there are many ways to evaluate quality in qualitative research, 2 important principles of rigor that relate to qualitative research, conducted under a constructivist paradigm, are authenticity, or quality of the data, and trustworthiness, or quality of the data analysis.¹⁵  As above, authors should be transparent and sufficiently detailed in their description of how the data were collected and analyzed. They should also include a rationale for why they chose their data collection and analysis methods. While what constitutes rigor will vary according to paradigm and methodology, a few basic practices to enhance rigor include:

1. providing rich description and authentic depiction of the phenomenon being studied;

2. including reflexivity statements that acknowledge how researchers’ context, relationships, experiences, and assumptions shaped the study;

3. explaining how triangulation, or use of multiple data sources, enhanced comprehensiveness of the study; and

4. asking participants if findings generally reflect their experience or asking peers to offer contextual considerations.

In our hypothetical professional identity study, to develop authenticity, we should engage in and document reflexivity throughout the study. We should provide a thick description of the context, including the tense environment at Hogwarts, as Voldemort was gaining power, and the types of interactions between professors and students. To help establish trustworthiness, we could ask student wizards if the proposed themes resonate with their experience.

Transferability—the degree to which the results of qualitative research can be transferred to other contexts or settings—is another way to demonstrate rigor in qualitative research. Transferability stands in contrast to generalizability, which is best applied to quantitative research. Researchers can enhance transferability through thick description, the use of theory, and resonance with prior work and the conceptual framework.¹⁶  Quantitative research, under a post-positivist paradigm, often has the goal to generalize from a sample to an entire population. In our Hogwarts qualitative study, generalizability descriptions, such as in the methods or discussion section, would not fit a constructivist paradigm and would raise questions in reviewers’ minds:

Methods: “To enhance generalizability, we included student wizards from all 4 houses.”

Discussion: “Although we chose a purposeful sample of student wizards, it is possible that we may not have had a representative sample which would limit generalizability of the findings.”

Instead, transferability may be a more appropriate term. To enhance transferability, in addition to describing the context of our Hogwarts study findings to help the reader determine the applicability to their own setting, we can also demonstrate resonance by relating the findings to existing research, and theoretical engagement by relating our findings back to the theories used in our original conceptual framework.

No study can avoid all problems, but authors who successfully circumvent the above quality concerns are far more likely to see their manuscripts move further in the review process. As Professor Albus Dumbledore says, “Words are, in my not-so-humble opinion, our most inexhaustible source of magic. Capable of both inflicting injury, and remedying it.”¹⁷  Choose your words to be transparent, clear, and consistent with rigorous qualitative research approaches. We look forward to reading your articles.