## SUMMARY

Internal control over financial reporting (ICFR) audits have been the subject of intensive examination by the Public Company Accounting Oversight Board (PCAOB) and researchers but the process through which auditors make ICFR judgments is largely a “black box.” To understand ICFR judgments, we conducted semi-structured interviews with 20 audit partners. Common themes in our interviews suggest that the subjectivity inherent in the ICFR evaluation task contributes to resistance against ICFR audit findings and cougnterarguments from management. Moreover, auditors perceive that their judgments are being second-guessed by PCAOB inspectors. Auditors believe that managers have difficulty accepting that material weaknesses can exist without a detected error, that management's reflexive reaction is to deny/avoid a material weakness finding, and managers routinely claim that management review controls (MRCs) would have caught the detected control deficiency. Auditors cope with management's defenses by consulting with their national office and leveraging support from strong audit committees.

Data Availability: Requests for the data should be accompanied by a description of intended uses.

## I. INTRODUCTION

Section 404(b) of the Sarbanes-Oxley Act of 2002 (hereafter SOX) requires an external auditor to conduct an annual audit of internal control over financial reporting (ICFR) for all public companies whose public float exceeds 75 million (U.S. House of Representatives 2002; 2010). Although the profession has worked to develop and refine ICFR audits, regulators worldwide have expressed concerns over audit quality around ICFR (e.g., FRC 2011; PCAOB 2011, 2012a, 2017; CPAB 2012). For example, in the U.S., the PCAOB's concerns are so significant that in 2013 the Office of the Chief Auditor issued PCAOB Staff Practice Alert No. 11 (hereafter PCAOB Staff Alert) to specifically address “practice issues observed by the PCAOB staff … relating to audits of internal control over financial reporting” (PCAOB 2013a, 1). The PCAOB Staff Alert cautions auditors about “significant auditing practice issues” and “significant auditing deficiencies [that] have been frequently cited in PCAOB inspection reports” (PCAOB 2013a, 1). Importantly, problems with ICFR audits remain a primary concern for the PCAOB, as it was listed as one of the “three key areas with the most frequent audit deficiencies” in a staff brief on 2016 audits (PCAOB 2017, 1) and remained an area of common audit deficiencies observed in 2018 audits (PCAOB 2019a). The SEC has also indicated that monitoring ICFR is an ongoing priority for the Office of the Chief Accountant (Croteau 2013, 2014; Bricker 2017). The SEC Chief Accountant recently reiterated that, “adequate internal controls are the first line of defense in detecting and preventing material errors or fraud in financial reporting … when internal control deficiencies are left unaddressed, financial reporting quality can suffer” (SEC 2019a). As Kral (2017) attests, “SEC enforcers have been investigating and prosecuting a broader range of ICFR violations than ever before, thus raising the stakes for certifying officers and others involved in the financial reporting process.” Further, an analysis of accounting-related federal class-action lawsuits filed in 2014 found that issues around ICFR were the most frequently mentioned problem, occurring in 31 of the 53 legal cases identified (LaCroix 2015). While there has been a substantial amount of research related to Section 404(b) of SOX, much of this work has focused on how the audit of ICFR has affected quantitative measures of audit quality (such as discretionary accruals) or market reactions to ICFR opinions that report material weaknesses (Knechel, Krishnan, Pevzner, Shefchik, and Velury 2013). There has been, however, limited research on the difficult judgments and challenges auditors encounter on actual ICFR engagements (Earley, Hoffman, and Joe [2008]; Gramling, O'Donnell, and Vandervelde [2013]; also see Gramling, Maletta, Schneider, and Church [2004] and Asare, Fitzgerald, Graham, Joe, Negangard, and Wolfe [2013] for reviews). Specifically, although the ICFR audit, and its subsequent impact on reporting, have become increasingly important for regulators (SEC 2007; PCAOB 2013a, 2019a, 2019b), the literature to date (primarily archival and experimental) has not examined the difficult aspects of, and challenges associated with, the ICFR evaluation. To address this gap in the literature, we explore the “black box” of a significant judgment made within an ICFR audit, the evaluation of an observed internal control deficiency. We conduct an interview-based field study to better understand the challenges, complexities, and pressures auditors encounter in making the ICFR evaluation and the nature of auditor interactions with client management-related ICFR findings (including management's reactions and defenses and auditors' strategies to address those reactions). An advantage of the interview approach is that we are able to explore some of the challenges auditors encounter in practice when making ICFR evaluations, but which cannot be observed through traditional archival and experimental research. We conduct our study, which incorporates two cases to serve as reference points for our discussions, using a semi-structured interview approach (Cohen, Krishnamoorthy, and Wright 2002, 2010, 2017; Trompeter and Wright 2010; Westermann, Cohen, and Trompeter 2019). The cases were intended to prompt participants to think about the difficult ICFR judgments and decisions they confront on audit engagements. We interviewed 20 very experienced audit partners to understand the factors that they consider important when evaluating observed internal control deficiencies and the factors that can promote and/or inhibit the effective conduct of the ICFR audit. Participants read the cases in advance of the interviews and were asked to evaluate the cases by assessing the severity of the internal control deficiencies described and whether the facts of the client situation represented a deficiency, a significant deficiency, or a material weakness. In the interviews, participants provided details of how they made their evaluations, including the similarities with typical issues encountered with their own clients, and the factors that influence their ICFR judgments and decisions. An advantage of incorporating judgment-based cases in the interviews is that it allowed for a common starting point for discussions with the participants about client and ICFR conditions. We use a qualitative research approach because our study is aimed at gaining a richer, more comprehensive, in-depth understanding of each auditor's judgment and decision-making process (Gendron and Power 2015) on an ICFR audit and how the related regulations and standards are interpreted and applied in practice. In particular, we focus on the process of evaluating identified control deficiencies. A qualitative approach allows for consideration of a broader set of variables than is feasible in experimental or archival work, and it allows us to gain constructive insights into the thought processes that are “inside the black box” of important judgments that are completed during the ICFR audit. Consistent with Malsch and Salterio (2016), a qualitative research approach is well-suited for research that is aimed at understanding judgment and decision processes that cannot be observed from merely reviewing, for example, firm audit manuals or other proprietary “how to” guidance from audit firms. Such materials do not provide the necessary information about how firm guidance is actually perceived and then applied by audit professionals. Importantly, we focus our research on the judgment and decision-making processes of audit partners evaluating identified internal control deficiencies, because partners bear ultimate responsibility for determining whether a material weakness exists (and the ensuing adverse ICFR report). Audit partners are also tasked with managing the client relationship and face scrutiny from PCAOB inspectors. Partners are responsible for conducting high quality audits that comply with professional standards in addition to adequately addressing concerns related to the ICFR audit raised in the PCAOB Staff Alert (PCAOB 2013a, 2). The PCAOB Staff Alert identifies specific areas of concern about the conduct of the ICFR audit that should be of particular importance to engagement partners and senior engagement team members (PCAOB 2013a, 2). Collectively, the alignment between our participants and research goals merits the use of a qualitative research approach. Overall, we find that the decision to identify a detected control problem as a control deficiency, a significant deficiency, or a material weakness is quite complex. For example, the decision involves a wide variety of variables ranging from technical items like materiality, scope, aggregation, and extrapolation, to more qualitative factors such as the client's tone at the top. An important factor contributing to the complexity of the ICFR judgment is that auditors must balance the demands of maintaining positive client relations while remaining firm against managements' resistance to ICFR findings and the tendency to discount the auditors' findings. Managers often argue that their management review and other compensating controls could mitigate the control problems identified by auditors. We also find that the PCAOB significantly influences how participants approach the ICFR audit task and ultimately make their judgments. That is, our participants report that practicing auditors are likely to consult with other partners and technical experts on ICFR issues because they are worried about the negative personal consequences that would ensue if an engagement they led was to receive a PCAOB inspection deficiency. Further, participants indicated that auditors tend to document more than they believe is sufficient to support the audit opinion under the existing audit standards. This strategy, where auditors provide what they perceive to be excessive documentation, appears to be a commonly employed practice that is used to accommodate the PCAOB's professional guidance and to bolster their positions in the event of a PCAOB inspection. Finally, consistent with a regulatory focus on C-suite controls, we find that the assessment of the tone at the top and audit committee involvement are among the most important evaluations in the auditors' overall ICFR judgment process. Our research extends the literature by providing information about how ICFR judgments are made in practice (i.e., the key factors that influence the decision process) and how regulatory guidance “comes to life” as it is interpreted and applied by audit practitioners. We also provide a comprehensive portrayal of the ICFR evaluation process that has the potential to enhance knowledge of the ICFR evaluation task. Our description of the decision-making process around ICFR has the potential to stimulate follow-up research examining the issues identified and explored in this study. Finally, our research has the potential to influence standard setters and regulators, because we provide evidence that how auditors perceive regulation can influence their ICFR judgments. Interestingly, our findings suggest that some aspects of the ICFR evaluation task have been changed, not because of a motivation to enhance audit quality, but because of the auditors' perceptions of regulatory pressures, which is potentially an area of concern for clients, regulators, and the audit profession. ## II. EVALUATING THE SEVERITY OF AN INTERNAL CONTROL DEFICIENCY In this paper, we examine the judgment and decision-making process audit partners employ when evaluating identified internal control deficiencies in an ICFR audit. We rely on PCAOB Auditing Standard No. 2201 (AS 2201), An Audit of Internal Control over Financial Reporting That Is Integrated with an Audit of Financial Statements (PCAOB 2007) and PCAOB Staff Alert No. 11, Considerations for Audits of Internal Controls over Financial Reporting (PCAOB 2013a) as the authoritative framework to guide the process used to evaluate the severity of observed internal control deficiencies and the factors prescribed for auditors' consideration when evaluating observed ICFR deficiencies. Depending on the severity of the weakness, an identified deficiency can be evaluated as a deficiency, a significant deficiency, or a material weakness.1 The differentiating factors in determining a significant deficiency versus a material weakness are the (1) likelihood and (2) materiality that a potential (or actual) misstatement would not be detected on a timely basis (PCAOB 2007, A-1-26). The following are circumstances that the PCAOB identifies as strong indicators that a material weakness exists: (1) restatement of previously issued financial statements to reflect the correction of a misstatement; (2) evidence of material misstatements (identified by the audit team) that were not prevented or detected by the client's internal controls; (3) ineffective oversight of the financial reporting process by the entity's audit committee; and (4) indication of fraud (either material or immaterial) by senior management (PCAOB 2007, A1-28). Recently, the SEC charged four public companies for their failure to maintain effective internal control over financial reporting, stressing that “companies cannot hide behind disclosures as a way to meet their ICFR obligations. Disclosure of material weaknesses is not enough without meaningful remediation. We are committed to holding corporations accountable for failing to timely remediate material weaknesses” (SEC 2019a). The standard setters' guidance on evaluating the severity of internal control deficiencies is limited in AS 2201. The PCAOB Staff Alert, however, highlights a number of relevant issues related to the ICFR evaluation, two of which are particularly salient in the present context: the importance of management review control(s) (MRCs) and the role of the audit committee (PCAOB 2013a). The PCAOB Staff Alert, discussed below, highlights many of the issues cited in reports of the PCAOB's audit inspections of the six largest firms in the period surrounding its release. For example, MRC-related deficiencies identified in the firm inspection reports include insufficient testing of the design and operating effectiveness of MRCs and inadequate testing to determine that the MRCs operate at a level of precision to prevent or detect material misstatements, or serve as appropriate compensating controls (e.g., PCAOB 2012c, 2013b, 2014a, 2014b, 2014c, 2015). In the PCAOB's Staff Alert (PCAOB 2013a, 34), the role of MRCs is emphasized in evaluating the severity of observed deficiencies. Specifically, the PCAOB Staff Alert states that in “forming a conclusion about whether a control deficiency or combination of deficiencies is a material weakness, the auditor should evaluate the effect of compensating controls, if any. This includes testing the compensating controls to determine whether they operate at a level of precision that would prevent or detect a misstatement that could be material. This includes evaluating whether the control addresses the risk of material misstatement to the relevant assertion intended to be addressed by the deficient control.” In many situations, the compensating control is a MRC, and as a result the PCAOB (2013a, 19) includes a detailed discussion of MRCs, acknowledging that, “auditors often select and test management review controls in audits of internal control. Such management reviews might be performed to monitor the results of operations, such as (1) monthly comparisons of actual results to forecasted revenues or budgeted expenses; (2) comparisons of other metrics, such as gross profit margins and expenses as a percentage of sales; and (3) quarterly balance sheet reviews. These reviews typically involve comparing recorded financial statement amounts to expected amounts and investigating significant differences from expectations.” Importantly, sufficient testing of MRCs remains a common audit deficiency (PCAOB 2017). The PCAOB Staff Alert (PCAOB 2013a, 20) then states that “many management review controls are entity-level controls, so testing those review controls can be an appropriate part of a top-down approach.” The PCAOB Staff Alert (PCAOB 2013a, 20) also acknowledges “that entity-level controls vary in nature and precision and that some entity-level controls might operate at a level of precision that would adequately prevent or detect misstatements on a timely basis. Other entity-level controls, by themselves, might not operate with the necessary level of precision, but might be effective in combination with other controls in addressing the assessed risk of material misstatement.” The PCAOB actively monitors auditors' evaluation of the effectiveness of MRCs, noting in their 2018 inspection of the audits of brokers and dealers that MRCs were not sufficiently tested as part of the auditors' testing of compliance internal controls (PCAOB 2019b). Based on the above discussion, we develop two broad overarching research questions (RQs) aimed at shedding light on the ICFR evaluation process as follows: RQ1: What are the challenges, complexities, and pressures auditors encounter in evaluating identified control deficiencies? RQ2: What are typical management reactions, defenses, and issues auditors face when communicating ICFR deficiencies to clients, and how do auditors address these reactions? ## III. METHOD ### Participants Participants were recruited from seven large international accounting firms as part of a grant received from the Center for Audit Quality (CAQ). Firm representatives were asked to identify partners who had significant experience with ICFR decision-making and negotiations with clients about ICFR reporting. To maintain participant anonymity, participants' names and contact information were provided to an independent coordinator from the CAQ who established an interview schedule and conference call line for researchers and participants. The blind conference call approach was adopted to maintain participants' anonymity and to encourage them to provide candid responses to our questions. Three of our participants had 12–15 years of auditing experience, eight had 16–20 years of auditing experience, and nine had over 20 years of auditing experience. Table 1 provides descriptive information about our sample participants including their gender, educational background, and industry experience.2 TABLE 1 Participants (n = 20) Each of the participants had experience auditing ICFR for public companies and almost all had significant experience evaluating both significant deficiencies and material weaknesses. Thus, they are appropriate participants to discuss and evaluate the auditors' judgment and decision-making processes for the ICFR audit. All interviews were conducted after the end of the 2014 busy season. Thus, the partners had experience with at least one audit cycle following the issuance of the PCAOB Staff Alert on the audits of ICFR (PCAOB 2013a). In addition, as discussed earlier, the issues raised in the PCAOB Staff Alert were already identified in the 2010 to 2013 firm inspection reports; therefore, auditors had familiarity with the ICFR audit areas that were of concern to the PCAOB. ### Procedures Following several qualitative studies in the auditing and governance domains (e.g., Beasley, Carcello, Hermanson, and Neal 2009; Cohen et al. 2002, 2010, 2017; Trompeter and Wright 2010; Hermanson, Tompkins, Veliyath, and Ye 2012; Westermann, Bedard, and Earley 2015; Westermann et al. 2019), we take an interview approach to address our research questions. We use the patterned behavior descriptive interview approach (Latham and Sue-Chan 1999) where participants are posed questions based on past behaviors or intended responses to situations that they encounter on the job. Adopting this approach allows us to gain insights into the factors that influence partners' decision-making on ICFR evaluations and to capture additional information that cannot be obtained in the confines of a survey or experiment (Hirst and Koonce 1996; Gendron, Bédard, and Gosselin 2004; Beasley et al. 2009). All interviews were conducted by at least two of the researchers. To ensure consistency across interviews, one researcher, who was very experienced in conducting interviews, served as the lead interviewer for all 20 interviews. Each of the remaining authors served as the secondary interviewer on one or more of the interviews. To help the research team establish a consistent tone and approach across all interviews, when authors were not serving as the secondary interviewer, they were silent observers of the interviews. Two doctoral student research assistants, who were blind to the research questions, listened and took notes during the interview sessions.3 Immediately following the interview, the two research assistants met and developed a single interview transcript for each interview. Two different research assistants, who were blind to the purpose of the study and the research questions, independently coded the transcripts according to themes established by the researchers. Each of the two research assistants was trained how to code the responses into themes separately by one of two researchers (i.e., each of the two researchers only trained one research assistant). Following coding recommendations from Miles and Huberman (1994), the researchers developed themes from the PCAOB Staff Alert (PCAOB 2013a), participants' responses, and the Asare et al. (2013) model of ICFR judgment.4 The PCAOB Staff Alert indicated factors that auditors should consider in their ICFR judgment (e.g., the level of precision of the MRC), and Asare et al. (2013) identified factors influential in auditors' ICFR judgments based on their review of the literature (e.g., the pressure to retain clients on auditor judgment). The final coding structure includes 17 unique themes used by the research assistants to code the interview responses (see Table 2). Initial inter-rater agreement on the coding of interview responses was 94.07 percent. Differences were then resolved by the independent coders, and the data presented is the reconciled data. Cohen's Kappa, a measure of inter-rater agreement beyond that predicted by random chance, was 0.939 (p < 0.01). TABLE 2 Interview Themesa To set the context for the patterned behavior interviews on partners' judgment about the ICFR deficiencies, we developed two cases with the assistance of a high-level, technical review, national partner of a Big Four firm. The expert described complex ICFR issues encountered on audits that had been referred to the national office for technical review and typical ICFR situations that lead to back and forth exchanges between auditors and their clients. Based on these descriptions, two cases were designed to prompt auditors to discuss the issues and challenges they face when determining whether to classify a control deficiency as a significant deficiency or a material weakness (see Appendix A for key case facts). We incorporated two control deficiencies in the fact pattern of each case to stimulate discussions about how auditors aggregate detected control deficiencies into an appropriate classification level (i.e., deficiency, significant deficiency, or material weakness) when determining the ICFR opinion. The sole purpose of the cases was to stimulate discussions with the participants about the real world issues they encounter on ICFR audits. Accordingly, we were not interested in “solutions” or interpretations of the cases. The first case involved a large financial services company with multiple offices to encourage discussion of issues that arise with more complex clients. The audit found some omissions in applying a new SEC rule regulating lost-holder accounts (dormant and unclaimed accounts in the financial system). This deficiency resulted in financial statement exposure from fines and penalties that exceeded the tolerable misstatement, but were less than the overall materiality. Management argued that the internal audit department would have likely detected this weakness, and consequently, an appropriate remedy to ensure compliance by the local offices would have been implemented. The control deficiency involved misclassified assets in the financial statements, which affected earnings, for the local offices concerned. Management indicated that these deficiencies should not be classified as material weaknesses, because the MRC, which depends on system-generated data, would have detected any errors that approached materiality. The second case involved a publicly traded commercial real estate firm. Two control deficiencies were detected during the ICFR audit. First, an analyst applied the incorrect interest rate when determining the fair value of the commercial buildings for two cities, which resulted in an overstatement of the fair value of assets in the financial statements. The error approached, but did not exceed overall materiality. Second, there was a computational error in the valuation model used to determine the fair value of investment securities that led to an overstatement of unaudited pre-tax income and total assets. The client argued that the two errors should not be classified as material weaknesses (either individually or when aggregated) because, had any error approached materiality, a MRC would have detected it. Participants read the cases prior to the one-hour interview. Ten questions were included to prepare participants for a discussion of ICFR issues, but our interviews did not necessarily follow the order, or address all, of the questions (see Appendix B). The order of the discussions referencing the cases was counter-balanced in order to ensure that the ICFR issues identified by the national audit partner expert and incorporated in the two cases were covered across our interviews. Importantly, the cases were only used as a starting point in each of the interviews and were general enough to stimulate discussions of typical client scenarios. The follow-up questions from the researchers were focused on the participants' clients rather than the case specifics. Further, as evident in the participants' responses, they used the case as a base to explain situations that they encountered in practice. For example, participants' comments included: “My favorite thing was that [your cases said] each error represents a one-time incident. Clients love to say that” (Participant 4). The cases “are hitting it right on the head. ‘One-time incident, isolated, internal audit would have caught,' are all part of a tried and true list of client responses” (Participant 7). Consistent with a semi structured interview approach (Cohen et al. 2002, 2010, 2017; Beasley et al. 2009; Trompeter and Wright 2010; Westermann et al. 2019), while the questions in the research instrument served as a guide for the interviews, researchers posed follow-up questions based on the participants' responses. Accordingly, the interviews were very flexible to allow space for respondents to elaborate on themes raised, and some topics were not necessarily addressed in all interviews. ## IV. RESULTS We organize the participants' responses related to the overarching RQ of the challenges auditors encounter in classifying detected ICFR problems, into two broad categories (see Table 3 for a summary of key findings and related opportunities for future research). The first, addressed by RQ1, addresses the challenges, complexities, and pressures associated with identifying and addressing potential significant deficiencies and material weaknesses. The second, addressed by RQ2, considers the nature of the management-auditor interaction in the ICFR decision-making process. TABLE 3 Key Takeaways RQ1: What are the challenges, complexities, and pressures auditors encounter in evaluating identified control deficiencies? To gain insight into the factors auditors consider when determining whether a control deficiency reaches the level of a significant deficiency or material weakness, we asked participants to discuss the factors that influenced their decision-making when classifying an identified control deficiency. We found that factors related to materiality, extrapolation, expansion of the extent of testing, aggregation of deficiencies, the source of the deficiency, compensating controls, PCAOB oversight, and views about audit quality affected auditors' decision processes. ### Materiality AS 2201 requires auditors to consider both the likelihood and the magnitude of the potential misstatement that could have occurred. In a synthesis of the literature, Asare et al. (2013, 141) observe that little is known about auditors' materiality judgments in the ICFR context. Our participants were quite direct in acknowledging that the ICFR task was a challenging one and that there were complexities in assessing the magnitude of the potential misstatement. It is hard; you have to judge the magnitude. If you can box it in and say the whole amount … is less than materiality, then they do not have a [material] weakness. [However,] in your case the potential misstatement could be material, so this is the definition of a material weakness. (Participant 13; Comments on Case 2) In their conclusions on inspections of ICFR audits, the PCAOB (2012b) inspectors expressed concern that auditors might place too much emphasis on the magnitude factor (relative to the likelihood factor) when concluding about the severity of a detected control problem associated with identified errors in the financial statements. When describing the ICFR evaluation process, participants typically began by discussing materiality. Notably, some auditors indicated they were unable to derive an appropriate classification assessment for the case facts that they had received because the case did not include complete materiality data that would normally be available on an audit. Related to the PCAOB's observation, we noted that materiality considerations were at the forefront of auditors' judgments about whether to assign an evaluation of a significant deficiency versus a material weakness. One participant noted: I had that one as a significant deficiency, because the total potential financial statement exposure is less than materiality, but exceeded tolerable misstatement. If the total amount was greater, it would have been a material weakness. (Participant 4; Comments on Case 1) In contrast, and seemingly more in line with the PCAOB's perspective on an appropriate ICFR evaluation, some participants focused on the specific case facts and circumstances to discuss how the magnitude of potential misstatement(s), rather than the mere magnitude of the actual detected misstatement related to a control deficiency, would influence their judgment. For example, one participant noted that, while the uncovered error might be small, the key judgment is to determine potential exposure in assessing the severity of the error and ultimately categorizing the deficiency as either a significant deficiency or material weakness. Related to materiality, as the error is subtle (less than 5 percent pretax income), the total error itself would not necessarily be material, but the potential error is material, so we had that as a material weakness. (Participant 4; Comments on Case 2) Thus, auditors often face the challenge of evaluating the potential effect of a control deficiency rather than the actual effect. Specifically, when judging the severity of a deficiency, the auditor must consider the potential magnitude—rather than the actual magnitude—of any resulting misstatement. Stated differently, at one extreme, if a deficiency resulted in a material error in the financial statements, then the evaluation of the severity of the error is relatively simple. However, at the other extreme, when the deficiency led to a small error—or even no error—in the financial statements, it becomes much more challenging to judge the potential likelihood and magnitude of an error that may result from the deficiency. ### Making Decisions Regarding the Nature and/or Extent of Testing Based on the Discovery of Deficiencies In grappling with their evaluation of the potential for material errors related to detected control deficiencies, auditors reported that they consider the need to expand the extent of their testing. The case facts that served as a reference point for our interviews featured a client with “Seventy-three subsidiary offices and affiliated companies operating in more than 20 different states.” The locations “were required to adopt common financial accounting and internal controls, as well as common operations and procedures, which are established at the national headquarters.” When deciding about the magnitude of the control deficiencies detected in the initial sample, auditors reconsidered whether they had sampled a sufficient proportion of the client's operations. If the exposure exceeds tolerable misstatement but does not exceed materiality, it is a judgment on how much further I'd go based on the exposure left. When we look at locations, we usually pick the most material locations to examine; the smaller locations are usually left off that testing list, depending on if the sample was selected judgmentally. Have we captured the biggest 24 offices [in our sample], do they make up 70% of the transactions? Do the other locations have a much lower level that would factor into my decision making to see how much further I'd go? (Participant 10; Comments on Case 1) Some auditors indicated that when assessing the magnitude of the problem they would consider expanding the nature and/or extent of the audit procedures to all of the offices/subsidiaries of the engagement. There is probably a new process put in place, based on local offices. We would look at a couple of processes and controls to verify decisions at local offices … All 73 subsidiaries should be examined, as they likely roll up to a significant amount between level one, two and three assets, and that is where the fix would occur. The decision/action taken at the beginning of the process is fundamental, as it leads firms down the wrong path. (Participant 3; Comments on Case 1) ### Extrapolating Findings to Full Set of Financials When determining whether any detected control deficiencies rise to the level of a material weakness versus a significant deficiency, auditors described how they extrapolate from the specific deficiency to the control conditions at the client as a whole. The auditors' considerations require an understanding of the nature and cause of the deficiency itself, assessing the frequency and pervasiveness of the control, and considering the deficiency in the context of the auditors' understanding of the client's overall business structure. For clients with subsidiaries and segments, auditors' considerations included how centralized versus decentralized the client's operations were and how common and consistent the client's internal controls were across the individual locations to help them determine the magnitude of the potential misstatement. It depends, looking at the locations, 3 out of 24, that is only about a 10 percent error rate, so when you look at it and extrapolate it to the remaining population, I'd need to get an understanding of the remaining exposure. (Participant 10; Comments on Case 1) This one has the potential to rise [to a material weakness]. Extrapolate… if you had all these subsidiaries and they all had different structures, then you would just try and narrow those entities that have the control deficiencies so you can squeeze down the possible misstatements. If it were a common control situation that cuts across everything then you would have something that you could extrapolate off of. (Participant 20; Comments on Case 2) In summary, it is challenging to extrapolate from a deficiency and project its potential effect to the financial statements as a whole. Specifically, the nature of the control (e.g., some controls are performed on every transaction daily or even less frequently; some controls are to be applied across the organization, while others may be limited to a specific unit or region) and the nature of the pattern(s) in the deficiency affect the ability to extrapolate. For example, it may be relatively simply to project an error when patterns suggest that the deficiency is limited to August transactions that originated in the St. Louis office. It may be much more complicated to extrapolate and project when there is no apparent pattern in the deficiencies, for example, across locations and throughout time. ### Aggregating Multiple Deficiencies AS 2201 guidance asks auditors to consider whether, or not, the combination of two or more identified deficiencies could give rise to a material weakness. Auditors reported that determining whether or not the aggregation of detected control deficiencies is appropriate under the particular circumstances requires assessing the connections in the deficiencies and evaluating whether there are links in the underlying control deficiencies. I'd gravitate toward a material weakness, because the overriding control problem under both of these [control deficiencies] is that the branches aren't complying with the standards that are provided. The corporate policies are being communicated by headquarters (the decentralized entity), but … the branches aren't complying with corporate guidance/policies. (Participant 10; Comments on Case 1) Let's say there are three different things that popped up in isolation that tie back into this local office information system and the updating of that system, and that that is causing all three individual items that occur. We would argue it's not the three individual items causing the issues; it's really the updating of the local system. Then, when we consider all things that could go wrong if the system is not up to date … could be three items aggregated, or 100 … if not kept up to date or accurate. (Participant 15; Comments on Case 1) To summarize, auditors sometimes experience difficulty in assessing whether two seemingly unique control deficiencies might be interrelated such that misstatements not detected by each individual control might occur simultaneously, and when aggregated, might lead to a material misstatement. For example, in Case 1 (as referred to in the quotes above) participants noted the difficulty in determining whether the two errors were independent or if there was a common root cause—i.e., lack of compliance with national office directives by some geographic units—that would necessitate aggregation of the errors. Making judgments about interrelated deficiencies and the likelihood the misstatements would occur in similar timeframes can be complex and highly subjective. ### Identifying the Source of Control Deficiencies The considerations surrounding the aggregation of one or more control deficiencies described by the participants suggest that auditors are focused on determining whether an issue (or multiple issues) could be linked back to a common underlying factor(s). Indeed, in almost all of the interviews conducted, there was some discussion of the importance of identifying the source of the control deficiency in the ICFR audit.5 Our discussions also revealed that auditors struggle with this aspect of the ICFR task. The auditors' own acknowledgment of difficulties in identifying the source of control deficiencies is consistent with observations in PCAOB inspection reports identifying shortcomings and the need for auditors to improve their efforts in identifying the “root cause” of ICFR audit deficiencies. Evidence of the strong emphasis coming from the PCAOB staff is that the summary of observations from 2010 ICFR audit inspections discusses the importance of identifying the “root cause” of deficiencies 15 times in the 22-page report (PCAOB 2012b). Auditors appear to have internalized this critique and applied it to inform their evaluation of the client's ICFR because their descriptions often discussed challenges in identifying the source (root cause) of the detected deficiencies. It is often also difficult to get to the source of the control problem. What didn't work? Was the control missing? Was there a control that was not designed well? Was there a control that was not operating well? Why? … I make my team and the client go backward for the source of the error, to go through a root cause analysis independently. (Participant 2; Comments on Case 1) What was the root cause of the spreadsheet error [in the case]? Was it an input error or was the formula not updated correctly in the worksheet? I would have to understand more about the error. This is an example of the “working backwards” root cause analysis. (Participant 2; Comments on Case 2) ### Assessing Compensating Controls After evaluating the source of the identified control deficiencies, auditors next grapple with determining whether the compensating controls identified by the client have the potential to mitigate the likelihood of a material error. As we discuss in detail later on, auditors face considerable resistance from clients when they question the effectiveness of the controls that clients regard to be serving a mitigating function. Consistent with regulatory guidance, auditors' evaluations of the compensating controls hone in on assessing the level of precision at which the compensating control is operating to determine whether it was sufficient to prevent or detect a material misstatement (PCAOB 2007, 68; SEC 2007, footnote 49). Some participants noted that clients tend to identify compensating controls that lack precision—often budget-to-actual analyses—which auditors deem to be insufficient to mitigate the risk of material misstatement. Auditors are then faced with the challenge of convincing management that their MRC is not precise enough to detect a misstatement that was not prevented by the control. That is a difficult argument to make when the auditor does not have data to prove that a misstatement would not be detected by the MRC. It is similar to a “he said, she said” kind of argument. Participants also indicated that they would gather additional evidence to confirm whether or not the compensating control was designed and operating properly to enable the auditor to classify what would have otherwise been considered a material weakness into the less severe classification level of a significant deficiency. The key is to really focus on our understanding of the precision of that mitigating control, to support the belief that it is precise enough to capture items, differences, or errors that may be more material. So, we would expand our testing of that control to look for specific instances of what types of errors it did capture, and really tried to capture. We look to see if there was enough data that would point to the examples of the control operating to give us a lot of confidence of what level of precision it operates in. (Participant 11) In looking for a mitigating control, we would evaluate internal audit's plan. Was the internal audit plan looking at these things? And, did the internal auditor not find them because internal audit had not gotten there yet? Is it in their plan? Was it something they check for? (Participant 3; Comments on Case 1) ### Anticipating and Adjusting to PCAOB Oversight It is not surprising that, given the nature of regulatory oversight (Johnson, Keune, and Winchel 2019), the auditors' observations about the ICFR process inevitably led to comments on the challenges and pressures they experience due to heightened awareness that their judgments are subject to PCAOB scrutiny. PCAOB scrutiny (“second-guessing”) is especially sensitive in the ICFR setting because the highly subjective nature of the task makes it possible that a given set of client conditions can be susceptible to alternative interpretations by different professionals. Overall, the findings presented in this section, and in the following section (i.e., Considering Audit Quality) confirm the findings reported by Johnson et al. (2019). All 20 of the participants indicated that concerns about being second-guessed by PCAOB inspectors influenced their ICFR judgments (see Table 2). The participants' responses suggest that there is a “culture of fear” surrounding PCAOB inspections that influences how the ICFR audit is conducted and what is documented in the workpapers. As Participant 4 observed: “We are auditing to not have a comment in a lot of ways to the PCAOB.” Participants were unambiguous in acknowledging that during the conduct of the ICFR audit, they are always cognizant of the consequences of a negative PCAOB inspection finding. Further, some participants argued that this specter of a negative PCAOB inspection finding leads teams to opt for the more conservative classification level when the classification is not clear cut. It [i.e., the impact of a PCAOB inspection] depends on the firm's approach towards the ramifications of a negative PCAOB inspection. If a partner has suffered those [negative] consequences, they tend to be risk averse and more willing to interpret things in a stricter way than they otherwise would. It does not mean that it is right. It is just a perspective. The message is getting through—that accommodating for client relationships is not going to be a winning decision—we must evaluate [ICFR] today on the greatest level of severity that could occur. (Participant 3) In some ways it appears that the culture of fear surrounding PCAOB inspections is driven both by the inspections process and how the audit firm chooses to react to inspection findings because audit firms often have tied audit partner compensation to PCAOB inspection results. At the firms, because of the clout that the PCAOB has, they are very sensitive to comments, and if you receive a comment at a firm, if you're a partner or director it directly influences your compensation. End of story! (Participant 20) This participant notes that the anxiety about the inspection process is not restricted to the partnership or leadership level. Rather, partners' concerns are transmitted and influence the lower level staff at audit firms. It [the inspection process] impacts everybody, everything they ask around documentation relies on granularity … The realization started to creep across partners and managers. There's no way a partner or manager could execute on that without pushing it down … the burden became so heavy they had to push it down to the staff people so they could do it. Then the staff became aware of that … the staff people are not immune to the attitudes of the partner and manager, and the pressure that they feel at the manager and partner level does seep down to the staff level and it impacts how people go about things. (Participant 20) A common topic that also arose from discussions of PCAOB inspections was that the inspection process prompts more detailed documentation on ICFR audits. On an overall basis, all participants discussed PCAOB oversight, and the topic was mentioned 128 times (more than six times per interview on average) over the course of our interviews. We have all become much more cognizant of [audit] documentation because of the PCAOB. Our firm has issued Practice Guidance because of PCAOB issues. (Participant 2) Some participants noted that changes in the documentation approach implemented by their firms were done to make it easier for reviewers and inspectors to understand the audit workpapers. Indeed, it appears that audit firms have worked to make a smoother path for inspectors in order to limit negative PCAOB inspection outcomes by more standardization of the audit process. [We changed towards] making things more transparent to someone not involved in the engagement. Standardized across the firm—forms to fill out through the process through firm-provided documents. Fill out auditor judgment forms and link risks—[because] linkage not clear and evident to someone who is not familiar with the audit/engagement, hinders the ability to follow a roadmap. (Participant 4) Many participants perceived these changes in audit approach and documentation to be more burdensome adjustments to their work that did not enhance audit quality. I think the first couple of inspection periods with the PCAOB did enhance the overall audit quality. [But] in the last couple of examinations I've been through, it's been a stretch to find something in the audit process that [provides a lot of added value]. At the end of the day, the risk reduction is not happening. We're simply adding more documentation to the file, and more costs. I could not care less about the cost, if it's a better product, but now I'm doing busy work and I have to subject the client to additional procedures simply because I have a regulatory body saying if you don't, I'm going to create a public report of you. (Participant 6) ### Considering Audit Quality Interestingly, while auditors acknowledged that the inspections had led to some improvements in audit quality, their discussions revealed some frustration over what they perceived to be an intrusion into auditors' exercise of professional judgment. Even the partners we interviewed, who generally had a positive outlook about the PCAOB, complained that more recent gains from the inspection process have become, in their opinion, more marginal. The auditors' perceptions were that things are now at the point where they are realizing very little benefit in terms of audit quality improvements as a result of the inspection process. The following comment from a participant, who reported serving as an engagement quality control leader at their firm, captured participants' general sentiments about the impact of current regulatory inspections on audit quality. Now we're going from seven feet to seven feet one inch, seven feet two inches, and now the things we are getting challenged on are only things we would appreciate. We would argue there are 51 angels as opposed to 52 angels dancing on the head of the pin. I'd say there is great rigor, but the rigor is to some degree a bit overboard sometimes. (Participant 20) In particular, auditors were concerned that the PCAOB inspections process, rather than the Board's standard setting process, was establishing (new) requirements for audit quality. One partner suggested that the PCAOB is the proverbial elephant in the room that auditors must be cognizant of at all times, as it represents a combination of the relevant standard setting body and the profession's regulator. When I … call my EQR [Engagement Quality Review professional] we are always referencing: “If the PCAOB was to look at this, what would their questions or concerns be?” In recent PCAOB examinations, I believe that we have been asked questions, or asked to do certain things, beyond the audit guidance and requirements that are out there. It is almost as if the PCAOB [the inspections teams] has been making some audit standards instead of reviewing the audit standards. Because their [inspection] reports do not have a gradation of language, it is an audit failure—that is the black and white language that shows up in the report. The general public does not have the ability to determine if it is a significant matter or not. (Participant 6) Auditors believed that on engagements subject to PCAOB inspections, the PCAOB inspectors often focused their review on areas that were under the PCAOB's inspection spotlight rather than directing their attention to client-specific areas of concern. The PCAOB tries to instill their will of what clients need [to do] through us. We spend a lot of time walking through the results of our inspections [with the PCAOB inspectors], not client by client, but “these are the themes that come out of the most recent inspections, so that's what we'll be inspecting in the upcoming audit.” (Participant 15) Auditors also report that they struggle to manage what they perceive to be the indirect but intrusive effects that the PCAOB inspection process potentially has on the auditor-client fee relationship, testing scope, and judgment. Participants indicated that as the inspections' focus changes and auditors seek to respond to areas that have high inspection risk, auditors could end up changing their testing approach and/or assessments, which could impact audit fees and client resources. Note that the findings in this study reveal that our participants perceive that their clients are not sensitive to their position and tend to be resistant to increased fees and other audit changes that arise from increased auditor effort to address areas of concern to the PCAOB. Sometimes they [clients] get frustrated. We're really transparent with our clients and audit committees about what the PCAOB is finding, and how it's changing our methodology. In a lot of ways, we are increasing our fees because it is taking more time for us to do all the documentation the PCAOB is requiring. Clients have heard a lot about it, and they're probably getting tired of it. Every year we need to do a bunch of additional work. We take more of the client's time, more support, more corroboration. We send clients feedback forms at end [of the engagement]. They [clients] don't understand why so much additional work and why the necessity of documentation. You keep saying it's PCAOB, PCAOB, PCAOB. Not a whole lot we can do other than reminding them we're a regulated industry. (Participant 4) PCAOB inspection causes greater expectations of defining the importance of controls, and this should cause a more objective evaluation of what the issue is. When opinions change, it would cause questions to arise why the opinion changed currently when the company has been audited for several years. This can be a source of contention with clients. We then explain that we reserve the right to do the same thing and re-evaluate. (Participant 3) The comments above express both auditor frustration, and their sense that clients are also frustrated, which auditors believe places them in the difficult position of trying to placate both their clients and the PCAOB inspectors. Moreover, the auditors themselves complained that changes implemented in their audit approach to pass PCAOB inspections are not cost effective. For example, some auditors complained that the PCAOB might be unnecessarily driving up audit costs. I've been in business for 20 years, largely involved in real estate and financial institutions. For the past 15–18 years (as CPAs) we've always used appraisals [that clients] obtained from third parties. We've looked at them (appraisals) for generalized reasonableness—meaning if the appraisers look at commercial property we want to be sure they don't consider residential property in their write up. The PCAOB is [now] saying we should not, and cannot, rely on those appraisals. We actually need to reorder an appraisal on that property in order to reach a conclusion on that property for that opinion. (Participant 6) However, we need to emphasize that these auditor observations about their clients' attitudes and resistance to increased fees could be merely a mirroring of the auditors' own attitudes and not necessarily whether or not their clients actually hold these views in practice. RQ2: What are typical management reactions, defenses, and issues auditors face when communicating ICFR deficiencies to clients, and how do auditors address these reactions? After managing the complexities associated with evaluating detected control deficiencies, the next major challenge audit partners encounter is communicating their conclusions to the client and persuading the client to accept the auditor's evaluation of their ICFR, particularly when material weaknesses and significant deficiencies are detected. Auditors reported that their clients' management often push back on ICFR issues, arguing that the detected deficiencies were “isolated occurrences,” that there were compensating controls to address the concern (but auditors typically deemed such controls to be inadequate), and several other defenses that would have prevented the material weakness classification. These denials from management and tactics to avoid a material misstatement evaluation require that auditors enlist assistance from the audit firm's in-house technical support and the client's audit committee. ### Difficult Conversations on Tone at the Top For auditors, the tone at the top is a necessary starting point in their evaluation of the client's ICFR. One participant observed that: “an appropriate tone at the top is more like table-space, you need to be there as a starting point” (Participant 19). However, because the tone at the top directly involves management, auditors report that it is difficult to have these conversations. Another factor that contributes to problems in discussions between management and auditors is the inherent subjectivity and sensitivity of the nature of the auditor's assessments. As one participant noted, “It's like the Supreme Court quote on pornography: It's hard to articulate it, but you definitely know it when you see it” (Participant 8). Since the assessment of tone at the top is often so subjective, many auditors refer to the assessment as something of a “gut” feel when making the determination. The conversation with management over its tone at the top can be awkward and, thus, sensitivity is required. It is not easy as an auditor, but sometimes you have to roll up your sleeves and have the hard discussion. If you (i.e., the audit firm) were more accommodating than you should have been in the past, it makes the conversation more difficult. (Participant 3) Auditors judge client managements' negative reaction to the auditors' ICFR findings as an important indicator of a weak tone at the top, particularly when management does not even believe the issue is worth discussing. Every engagement team investigates, to some degree, tone at the top. If you have instances where someone (management) dismisses something out of hand, you get to a point where you question things. If there are tone at the top issues, you take them pretty seriously; people react to that quickly. (Participant 20) Auditors believe that another indicator of a weak tone at the top is when the management team values the accounting function solely in terms of its cost and ability (or inability) to contribute to profit generation. Auditors believe that these types of managers tend to view the accounting function and internal controls as something that they have to do—a necessity of running a business—rather than recognizing these functions as effective investments in their business and effective risk monitoring or risk mitigation tools. Consistent with partners' observations of some managers placing inadequate value on ICFR, the SEC Division of Enforcement recently announced that as part of their ICFR initiative they had filed actions against companies for failure to maintain adequate ICFRs for seven or more consecutive annual reporting periods (SEC 2019b). Participants noted that when management does not value internal controls, it often results in their companies having less qualified personnel and understaffing of key internal control functions. In some cases, yeah, our clients are focused on the bottom line. There are certain boards, CFOs, CEOs that don't see accounting as value added. They see it as an overhead cost. They don't want to invest in it. When they look at investments, they don't put their dollars there. They put their dollars to areas that expand operations, boost revenue and cost reduction. Accounting, marketing, HR is what gets hit first when costs are cut. Not everybody does that, but there are a lot of companies out there with that mindset. (Participant 4) The participants also noted that these narrow views about the value of the internal control sometimes extended to managers' lack of appreciation of the role and value of the internal audit department. We have certainly come across the budget/“not income producing” [attitude that] extends to the internal audit department. The company has a weak internal audit or internal control department, and lower level individuals not as experienced as they should be. Management does not want to pay the salary for a [professional at the] manager level that can challenge people and really understand the control environment, or they just want bodies in internal audit. (Participant 5) Interestingly, some of these views that the internal control represents more of a cost than a benefit to the organization were not restricted to management only. Auditors reported that they also encounter audit committees that are not fully appreciative of the importance of strong controls. This lack of appreciation can extend to the point of board members undervaluing the strong internal controls that are in place and exploring the feasibility of downgrading the existing internal controls. Participants commented that, in addressing such instances, it often requires an education process to get their clients' audit committees and boards to understand why a strong investment in accounting and control resources translates to real benefits to shareholders. Some companies have very robust accounting functions, right segregation of duties. Then the audit committee says, “You got an ‘A' team, but do we need it all or is it just increased costs and can we go with a ‘B' team? Can it be done to save costs?” Some companies accept it, [but] some view that this [accounting] is a cost that must be incurred to access capital. (Participant 4) ### Management's Unwillingness to Accept ICFR Findings As discussed above, auditors usually linked discussions of tone at the top to the conversations they have with management about the control problems they detect. While there was general agreement that the discussion was not too difficult if it centered around the discovery of an issue classified as a control deficiency, there was general consensus that the conversation became more difficult if the auditor assessed the control problem to be a significant deficiency, and even more contentious when a material weakness was assessed. Auditors indicate that they are aware that management might truly believe that the explanation they offer is valid. Other auditors believe that clients are “unable to listen to or hear the relevant facts if [they] go to them and say, ‘Hey, you've got a material weakness'” (Participant 6). To persuade management, some auditors report that they make presentations to management that incorporate flow charts to explain how the key concern is the potential for error, not necessarily a found error. In these circumstances, the client has to be walked through the auditor's understanding that if the potential error is greater than what the auditor deems to be material (material enough to record an audit adjustment), then the error meets the threshold for being a material weakness. Participants indicated that as partners, they must manage a delicate balance of being objective and firm, while at the same time demonstrating an understanding and appreciation for the client's perspective. As one respondent indicated: It is a difficult conversation to have, but it does happen. I have been involved in three [material weakness classifications]. We go through that process, and management does not believe that the issue needs to be addressed. For example, [they believe] the auditor is looking at something the wrong way; then the auditor can say the opportunity exists for this not to be right. The audit process is a confirmation process, validating controls. It's not an easy decision where you have to look at it from 50,000 feet; [you] must look at the big picture, the forest, not the trees. You [the auditor] get a feeling for what the environment is like based on the issues and other controls. It's not a yes/no on the design and effectiveness [evaluation]. This makes [ICFR] different from substantive procedures. (Participant 3) #### The Isolated/One-Time Incident Defense A common strategy that clients employ to deny the existence of a material weakness is to claim that the control issue detected by the auditor was an isolated event, a one-time item, or due to an individual who is new on the job (and is learning while on the job). My favorite thing was that [the management in your cases said] each error represents a one-time incident. Clients love to say that, “it's an isolated incident” and “it's the specific facts and circumstances,” but there is no such thing. (Participant 4, Comments on Case 1 and 2) You don't get a free pass for on the job training, so they should have certain shadowing or on-the-job training, in effect a control procedure to get people up to speed … I don't think you get a free pass because of that. For me it's more you may choose to do some additional testing of that person's work product to be sure that was the only product or item tainted. It's tough to identify an isolated error without doing more testing. (Participant 19) A clients' tendency to seek to contain the detected control problem to a specific circumstance, which would not require further action, is analogous to prior findings on audit sampling documenting that auditors often decide against projecting detected sample errors to the population (e.g., Burgstahler and Jiambalvo 1986; Dusenbury, Reimers, and Wheeler 1994; Hermanson 1997; Wheeler, Dusenbury, and Reimers 1997; Elder and Allen 1998). For example, auditors tend to project to the population less than 50 percent of the time (Hermanson 1997), focus on information that contains errors to subpopulations, and weigh the containment information more than error frequency (Dusenbury et al. 1994). Burgstahler and Jiambalvo (1986) and Hermanson (1997) note that choosing to isolate errors can be risky because isolated errors might be representative of other unique errors in the population. Consistent with Hermanson's (1997) observation that auditors with higher task-related expertise were more likely to appropriately project errors to the population, audit partners interviewed for this study reported that they scrutinize and refute management's attempt to contain the issue to an isolated incident. Auditors' responses to the isolated event response are typically first to explain to the client “the standards don't allow us to say there is a one-time thing; we can't say it was an isolated issue” (Participant 5). The participants reported that they have to communicate to management that they cannot accept the “isolated event” explanation. Auditors have to educate clients that according to standards a control should work with new or seasoned individuals and to discuss whether the issue is in fact only a one-time event. Second, these conversations typically require auditors to expand their testing to persuade the client to accept the auditor's classification. Several auditors also observed that whenever management puts forth the one-time or isolated incident excuse, they see it as a red flag, which can make the auditors more skeptical. Other auditors report that the isolated incident excuse leads to diminished management credibility and more audit testing because “if [you see] one ant in the room there are probably more that you don't see” (Participant 13). #### Management Defense—A Material Error Would Have Been Caught Another common response managers employ to try to refute a material weakness finding is the two-pronged argument that (1) if the error associated with the control deficiency was material it would have been detected and (2) since the detected misstatement was not material then there is no material weakness. Seventy-five percent of management start in a position that something in their system would catch it, or it would be caught if material. Only 25 percent of the companies recognize “you're right, we did not catch it.” (Participant 3) Management thinks they would catch anything material; this is where the discussion usually takes place … It is definitely more challenging when there is no error. Making sure everyone is on the same page when every transaction that is subjected to the control operated effectively, if the answer is right [i.e., there are no material problems with the financial statements] it doesn't mean that the control works. This is a hard concept for people to wrap their head around. It is difficult to try and make the argument that the process worked, but the control did not. (Participant 5) The above comment highlights our observation that getting management to accept negative judgments made by auditors is most difficult when the ICFR audit identifies significant internal control deficiencies, but there were no required audit adjustments to the financial statements that were associated with the control deficiencies. Consistent with the auditors' concerns, the former SEC Deputy Chief Accountant observed that it was “surprisingly rare to see management identify a material weakness in the absence of a material misstatement” and reported that the Office of the Chief Accountant, the Division of Enforcement, and the PCAOB were working together to address the issue (Croteau 2013). Auditors report that part of the difficulty in getting managers to appreciate the existence of a material weakness in the absence of a material error is that the auditor is arguing, in essence, that the client was “lucky” that its financial statements were fairly presented even though the internal controls were not adequate. Not surprisingly, in such situations, management often argues that the internal controls must be fine because there were no errors found in the financial statements. This is illustrated by the following: When you get into “What could happen?” [as a result of the internal control deficiency], that's where the relationships and discussions get more difficult. (Participant 3) There are many clients that roll their eyes and think we're being self-serving and kind of ridiculous. If there was no error [in the financial statements], there could still be a hole. If it is a big hole, but no misstatement, yeah, they may have gotten lucky. (Participant 4) Management's difficulty in accepting the potential for material misstatement without the existence of an actual misstatement can be understood through the cognitive concept of the outcome effect (Fischhoff 1975; Brown and Solomon 1987; Baron and Hershey 1988; Lipe 1993; Peecher and Piercey 2008). If the auditor discovered a financial statement error during the audit, it is much easier to claim the existence of a control deficiency—or else, how did the error occur? Thus, the outcome (i.e., a discovered error) makes the weakness appear obvious to management. However, in the absence of such an outcome, it is difficult for management to acknowledge (or at least easier for management to argue against the existence of) a control deficiency. The difficulty of these conversations suggests that clients do not have a clear and complete understanding of the concept of an audit of internal control that is integrated with an audit of financial statements (as outlined in AS 2201). In many ways, the burden of proof of concluding that a material weakness exists, when there is no corresponding financial statement error, ultimately depends on the ability of the auditor to clearly demonstrate that a material error could occur, given the nature of an observed internal control deficiency. Successfully navigating this situation (i.e., persuading management to recognize and acknowledge a control problem in the absence of a financial statement error) is potentially made more difficult by the outcome effect (Fischhoff 1975) that leads to a cognitive bias against recognizing and/or acknowledging that a material weakness actually exists when no corresponding financial statement error has been detected. #### Management Defense—MRCs are Effective Compensating Controls Auditors often interact with managers who tend to have “a knee-jerk reaction” arguing against the auditor's conclusion that there is a control problem by moving to a discussion of mitigating controls for the deficiency identified by the auditor. Specifically, one common theme that emerged is that management frequently contends that MRCs operate as compensating controls for many observed deficiencies, and that those controls will prevent material misstatements from occurring. That's exactly right, even in the general IT control issues, management will say, “We do this detailed financial statement review, and we would have caught it.” (Participant 9) Thus, an important part of the ICFR audit is evaluating the validity of management's claims that errors will be caught by higher-level entity-level controls (ELCs). Accordingly, the evaluation of mitigating controls, such as MRCs, is a core part of the auditor's evaluation of the client's ICFR. The two cases used to facilitate discussions with the participants featured management who claimed that the entity-level MRCs would have detected the errors identified by the auditors. Many of the participants chose to respond directly to such management claims. For both cases, the management claims: “We would have caught it.” What we need to understand is what would have caught it, and why [their mitigating controls] didn't. The auditor has to say, “You have this process. Why didn't it catch it?” If we didn't test it, we need to test it to put some credibility to that statement. (Participant 8) The ELC identified in the case—the detailed review as mitigating control—is an annual control. This is the only management review control, and it is at a high level, so [it is] not a good control to mitigate down to a significant deficiency. [Typically], what I find is that ELCs and management controls usually do not operate well enough to identify material [misstatements] for a company that is as decentralized as this one. (Participant 2; Comments on Case 2) #### Management Defense—Internal Audit Serves as an Effective Mitigating Control Related to the ELC response they typically encounter, audit partners also indicated that clients often argue that the internal audit function (IA) could be relied upon to catch control problems. Indeed, one participant indicated that the discussion cases for this study “are hitting it right on the head. ‘One-time incident, isolated, internal audit would have caught,' are all part of a tried and true list of client responses” (Participant 7). However, respondents questioned the idea that the client's IA could serve as a component of the MRCs. Many of the participants emphasized that the client's IA cannot serve as an effective review control if they are active participants in the performance of the control function. In the example (i.e., the study's case) it is hard to argue that there is a compensating control because there is nothing else apart from IA looking at that control, and really, IA shouldn't be part of your control structure. IA should look at making sure the controls exist, they should be validating the controls. [Doing more] would basically compromise their independence, and then they are not independent. (Participant 20; Comments on Case 1) Others shared this observation that IA cannot serve the functions both of monitor and control agent, which is evident from the following comment. The other side is, once you start down the path saying IA will catch things, then IA switches from a monitoring to a control [function], so it changes the IA's character. When IA becomes the control, you end up having to consider the effectiveness of the monitoring function. They [IA] lose objectivity relative to the control they are performing. (Participant 3; Comments on Case 1) Overall, management's automatic reaction to potential internal control weaknesses is often to argue that MRCs would have caught any material errors. The problem is that management believes that they can simultaneously perform and review the control activity. It appears that management does not appreciate that these roles are incompatible. This issue is further exacerbated by management's reliance on IA to perform the two incompatible roles of an agent in the control procedure as well as a reviewer of the control procedure. Thus, auditors report that they must be vigilant and remain skeptical when management offers such excuses. Although management is likely to raise the defense of a MRC or IA as a compensating control for an observed deficiency, auditors are cognizant that these management defenses and explanations are quite often merely attempts to avoid the stigma associated with a material weakness finding. #### Management Views Material Weakness as a Stigma to be Avoided Auditors believe that another factor that contributes to a combative material weakness conversation with clients is that some managers treat ICFR exceptions as a personal issue. In the auditor's judgment, managers perceive that they are being accused of negligence, incompetence, or are in some way at fault for the control problem identified by the auditor. Client management has a reputation to uphold and does not want to be denigrated in front of the audit committee and the Board. Thus, impression management, on the part of client personnel, is one factor that can explain the disagreement. Some companies' management, controller or CFO, I don't know if they are compensated or evaluated on having no significant deficiencies or material weaknesses, but in some client situations, I get the sense they take it very personally, and they think it reflects badly on them to have anything brought to the audit committee. In some cases, they will fight tooth and nail not to bring something to the audit committee. (Participant 5) ### Additional Resources to Address Client Defense Strategies—An Effective Audit Committee Our observations above reveal that how auditors manage their discussions with the client can be crucial to the successful completion of the ICFR engagement. While each partner believes that coming to the right conclusion and, ultimately, the correct ICFR opinion, is paramount, the client relationship is also important, and maintaining the relationship is central to a successful ICFR engagement. If the assessment is subjective, it makes the judgment harder and more important for auditors to do their jobs. And [that] increases the difficulty to meet the expectations of regulators and standard setters. We should not be advocates for the client, [but] we need to be open minded at the same time. The balance between trying to keep your client satisfied and trying to meet your professional responsibility is important. It is an art, not a science. (Participant 4) However, achieving this balancing act often demands that auditors consider the multiple parties within the client organization, as demonstrated by the following participant comment. The auditor is trying to balance relationships, so the auditor does not want to throw the client under the bus and run to the audit committee. You work with management [i.e., the client] to reach a conclusion about the matter, and then you share with the audit committee when appropriate. (Participant 3) With some clients, auditors encounter audit committees committed to fulfilling their fiduciary responsibility to ensure high quality financial reporting. These audit committees show strong support for the auditors in the face of management opposition. In such circumstances, auditors can leverage the power of the audit committee to shape the appropriate management response. The importance of having both audit committee members who have financial reporting expertise and an audit committee that supports the auditor is illustrated by the following. We took it [a dispute over a material weakness] all the way. It was between me and the client [management]; we took it to the audit committee. The audit committee happened to have a chair that was a CPA. [The financials] required a restatement, [and] the client [management] said we do not need to provide notice of material weakness until the end of year, although the issue came up on a September quarterly interim filing. We said that, in conjunction with the restatement, management needs to identify that there is a material weakness in the controls. And they were hard pressed to agree to that, so we go in front of the audit committee. The audit committee chair went to bat and said [to management] “If you don't [issue a material weakness] I expect your resignation on my desk shortly,” and I appreciated that. (Participant 6) Overall, the subjectivity in the evaluation of internal controls creates a natural source of tension and disagreement with management. However, a sound governance culture, as evidenced by a strong audit committee, has the potential to help the audit of ICFR run more smoothly and be more effective. In addition, to be truly skillful at the ICFR audit task, auditors have to be comfortable with a high level of ambiguity. Prior research in other audit settings, however, has documented that auditors seek to avoid ambiguity (e.g., Bamber, Snowball, and Tubbs 1989; Nelson and Kinney 1997; Luippold and Kida 2012; Joe, Vandervelde, and Wu 2017). ### Additional Resources to Address Client Defense Strategies—Increased Internal Consultations Another prominent factor that received significant attention from participants is the extent to which engagement partners consult with their firm's national office to seek advice or validation of their professional judgments. It appears that one strategy auditors employ to cope with client pushback and regulatory pressure is to garner support within the firm to strengthen their position. As one partner noted, regarding their relationship with the national office: As an engagement partner, we cannot put ourselves in an “us vs. them” [with the national office]. That already exists between us and the client. We cannot “go to bat” for the client, as that puts us in a bad place. Consultation is never a bad thing. Either, we find out that we were right and validate the decision, or [the consultation] brings up new ideas to show we were wrong and how to improve. (Participant 3) The “routinization” of seeking national office reassurance could lead to a loss in individual professional judgment expertise, as partners might over-consult (frequent use of the national office as a knee-jerk response) and use the national office as a “crutch” to substitute for their own professional judgments. One partner's comments on why he/she would “go to national” directly describes this issue. [For] me as [the one] signing the opinion line, I don't want to be in that pool by myself. I want the national office involved that sees three times as many companies as me. This gives me additional support for when I have those discussions with my clients. Helps me think through the conclusions and be able to articulate them carefully. So, I go to our national and regional offices frequently. (Participant 4) Further, to achieve increased consistency throughout the firm, audit firms work hard to promote as uniform an approach as possible. In fact, based on our observations, the more prevalent the perception of a need to consult and obtain reassurance from the national office, the more likely it is that the audit can become homogeneous. Consider the following: Our firm has specialized ICFR experts that review all advanced filers as we go through our initial stages of documenting. An ICFR expert reviews with the existing engagement team and examines documents, and documentation of walkthroughs, and selecting key controls. At each milestone, he'll come in and see what we have documented to ensure we meet standards. (Participant 4) Similarly, another partner points out that there is a strong infrastructure to support and encourage internal consultations as a place to turn to for advice. [There are] two layers we can bounce conclusions off: other partners or a research center. [We] have an ICFR steering committee, have resource members, all assigned to ICFR auditing. I sit on that committee, so I am someone that helps [other] people work through problems like this. In terms of firm methodology and policy, where it's a close call for material weakness, we are required to consult with our regional technical partner and the national group. (Participant 5) The need for balancing judgment at the engagement team level and trying to achieve consistency across the audit practice of the entire firm is echoed in the following comment: I think the engagement team goes along with the initial assessment … goes to the conclusion … but if it's on the borderline between significant deficiency or material weakness, they'll bring it up. They'll have a position on it, and we'll discuss it. It's a joint decision. You don't have someone sitting up there … it's just a dialogue. The only thing a national office perspective brings is consistency across the firm. (Participant 20) Collectively, the evidence suggests that firms are trying to achieve consistency as much as possible in their audit approach, but they run the risk that the loss of individual professional judgment can be a tradeoff in the pursuit of consensus and standardization across the firm. ## V. CONCLUSION The ICFR audit has been the subject of intensive examination by regulators and researchers in recent years. Regulators of audited financial statements (i.e., the SEC and the PCAOB) have recently issued several written and oral communications expressing concern about public companies' ICFR and the ICFR audits (Panucci 2016; Quaadman 2015). For example, in an address to audit committees, the Chief Accountant of the SEC emphasized internal control as a “key issue” that requires attention from audit committees in order to achieve more reliable reporting to investors (Schnurr 2015). Several additional SEC staff speeches (e.g., Croteau 2014) have also questioned whether companies are properly identifying, evaluating, and disclosing material weaknesses to investors. The PCAOB has also issued inspection reports and staff practice alerts (e.g., PCAOB 2013a) aimed at improving the audits of ICFR. The SEC has even acknowledged that the focus on ICFR is a coordinated effort with the PCAOB, signaling the importance of the ICFR evaluation to stakeholders. Motivated by regulators' concerns over the audits of ICFR and ICFR evaluations by preparers, we conducted an interview-based examination of the challenges and issues auditors face during the ICFR audit. Our participants were 20 highly experienced partners from seven of the largest auditing firms. Using cases, developed with the aid of a technical review senior national partner expert, to serve as a reference point for a discussion of common issues and typical challenges encountered in the ICFR audit, we conducted semi-structured interviews with audit partners. Our analysis of the issues discussed during the interviews reveals that the regulatory oversight of the ICFR audit has had a deep and profound impact on both the way ICFR audits are conducted and auditor incentives around the ICFR task. This is not surprising because the passage of SOX led to two new simultaneous requirements: the ICFR audits for publicly held companies and the imposition of an external regulator to govern how audits are conducted. Partners' responses provide important insights about the process by which auditors actually make ICFR judgments and decisions, which has been a relatively unexplored area. We find that the evaluation of whether an observed internal control deficiency would rise to the level of significant deficiency or material weakness is a complex decision requiring the consideration of a number of variables. A common starting point for the ICFR evaluation process is a focus on the materiality of a detected or potential misstatement. That approach is consistent with the professional standards that require auditors to determine the magnitude of the misstatement that could potentially occur because of the identified control deficiency (PCAOB 2007). However, the partners noted that in order to reach a conclusion, they had to consider variables such as the extent of additional work to be completed, extrapolation, and aggregation. Our interviews suggest that the PCAOB's notion of RCA when discussing ICFR audit deficiencies has become successfully ingrained in auditors' thinking when evaluating their clients' ICFR deficiencies (PCAOB 2017). Auditors' approach focused on identifying the source of the client's deficiency, including determining whether or not there was a single cause for multiple errors, and whether there was actually greater exposure than the amount associated with the source of the specific uncovered error. Moreover, we found that the discussion of materiality and the analysis of the source of the deficiency are often linked to evaluations of the client's MRCs. Auditors reported that management generally perceives that their MRCs serve as a compensating control for any observed deficiency and that the MRCs would catch the (detected) error before it results in a material misstatement. As a result, the evaluation of MRCs is a key component of an auditor's evaluation of the client's ICFR. The subjectivity in the evaluation of MRCs and ICFR deficiencies in general contributed to auditors feeling vulnerable to second-guessing and criticisms from PCAOB inspectors. Not surprisingly, given the change from self-regulation to PCAOB oversight, auditors expressed concern that the inspections process had the potential to detract from substantive audit quality because a focus on “passing” a PCAOB inspection tended to govern their audit approach. One of the more prominent findings in our interviews was the significant amount of effort dedicated to getting client management to accept the auditor's evaluation/classification of identified ICFR problems. Participants indicated that it is often challenging to get the client's management team on board with classifications of significant deficiencies or material weaknesses, as managers frequently believe that any observed internal control deficiency is merely an isolated incident. Managers also had difficulty accepting that the potential for a material misstatement was the key factor in evaluating ICFR deficiencies. Auditors reported that their clients tended to assume that unless a material error was detected, there was no reason to raise the severity level of the control deficiency. In order to cope with pressures from client management, auditors relied on either a strong audit committee or internal consultations with experts in their audit firm to push back against management resistance. Figure 1 presents an additional summary of our results based on the common themes that emerged from both the evaluation of ICFR deficiencies and how this relates to communicating and resolving ICFR deficiencies with management. For example, one important theme, RCA, involves an evaluation process that could extrapolate from a single incident to the overall client level. The challenge in communicating and resolving ICFR deficiencies in RCA is getting management to understand a potential misstatement needs to be recognized. Summary of Findings Summary of Findings As engagement partners increasingly turn to their firm's national office to get advice or validation of their most difficult professional judgments around ICFR, audit firms achieve more consistency across their audit practice, and they perceive this to be an effective response to help mitigate second-guessing by the PCAOB. Importantly, we find that some audit firms have placed monetary penalties on partners for audit deficiencies, which appears to weigh on the minds of the partners. However, a downside of this approach is that it could lead to a check-the-box mentality that could be detrimental to an auditor's professional judgment. An important research question to consider in the future is to evaluate to what extent, if at all, audit firms are converging in the types of documentation provided. Next, future research can examine to what extent, if any, firms are adopting a homogeneous, plain vanilla ICFR audit process across different industries. Finally, future research can examine if a PCAOB-driven ICFR audit process improves or does not improve audit quality (also see a variety of specific research questions in Table 3). With respect to our study's implications for regulators and standard setters, we find that there is a great deal of ambiguity in what constitutes a deficiency, a significant deficiency, and a material weakness, and what distinguishes each of these three categories from each other. This ambiguity appears to contribute to our observation that there is significant disagreement between auditors and client managers on the classification of ICFR deficiencies. It could be beneficial for users if the SEC were to provide more explicit guidance to preparers with respect to what constitutes, for example, a material weakness, and how that differs from a deficiency or a significant deficiency. This would allow for more consistency and comparability across companies, rather than relying on auditors to hold steadfast against client pressures to minimize the classification of detected ICFR deficiencies. The PCAOB and its inspection teams might also find it beneficial to further consider how auditors interpret standards and inspection reports, and how these interpretations affect the conduct of the audit engagement. For example, one can assume that unnecessary documentation is not a goal of PCAOB inspectors; therefore, PCAOB inspectors could consider providing additional guidance regarding more appropriate levels of audit documentation when they detect excessive and unnecessary documentation that does not contribute to improved audit quality (or an improved ability to assess audit quality). As in all studies, there are limitations that represent opportunities for future research. First, we employed a qualitative approach to the most consequential professional judgment in the ICFR audit, assessing the severity of an observed internal control deficiency. Although that did lead to a probe of at least part of the “black box,” we did not control for certain variables or theoretically examine causal links between various factors and the final ICFR assessment decision. This leads to the important research question of how certain variables, such as complexity of the issue, the industry expertise of the auditors, and other factors, will influence the effectiveness, as well as, the efficiency of the ICFR decision process. Relatedly, the cases used as a reference point for our interviews discussed control problems in the financial services and real estate industry sector, but several of our participants were not financial services or real estate industry experts. While our cases were simple, and likely did not require industry expertise, we acknowledge a potential limitation that having financial services or real estate industry experts as participants might yield different responses. Future research could explore this concern. In addition, we did not have access to the actual PCAOB inspection reports or the audit workpapers related to our participants. Thus, we cannot directly link the reports to the actual work that is done and make any inferences on the direct or indirect link to measurable dimensions of audit quality. A future study could use archival data to make the link between factors such as audit fees, client importance, and industry effects on the ICFR audit process. Despite these limitations, our research provides the initial evidence on the “black box” of the ICFR process, and we are hopeful that this could spur future research in the area. For example, future qualitative research in the area can turn to institutional theory which examines how social norms or rules within an organization, or an entire profession, become entrenched and emerge as authoritative guidance for behavior within an organization (Scott 1995). The ever-increasing role of regulation in accounting and auditing has led to increased application of elements of institutional theory in a number of related areas such as corporate governance (Beasley et al. 2009; Cohen et al. 2010; Westermann et al. 2019) and public accounting firms' expansion into consulting (Malsch and Gendron 2013). In essence, institutional theory argues that audit firms achieve legitimacy by using a set of accepted practices and adhering to the expectations of the oversight body (i.e., the PCAOB) even if those practices are not a perfect fit for the firm. Quite interestingly, consistent with elements of institutional theory, our findings suggest that auditors are doing at least some additional work because of perceived pressures from the regulators, and not because the work improves audit quality, which is potentially a concern to the auditing profession. Moreover, our results reveal that engagement partners increasingly turn to their firm's national office to get advice or validation of their most difficult professional judgments around ICFR. This is done to help audit firms achieve as much consistency as possible across their audit practice and to help mitigate second-guessing by the PCAOB, which is entirely consistent with an institutional theory perspective. Increasingly, auditors perceive that the work conducted for the ICFR audit is done merely to forestall a negative reaction from a PCAOB inspection, rather than to enhance quality. This has led to excessive documentation and conformity that is done to reduce second-guessing by regulators, which is also consistent with Westermann et al. (2019). Importantly, we find that some audit firms have placed monetary penalties on partners for audit deficiencies, and this appears to weigh on the minds of the partners. This potentially could lead to a mind-numbing, check-the-box mentality that could be detrimental to an auditor's professional judgment. Future research might consider whether we are also seeing evidence of institutional theory in that firms are converging in the types of documentation provided and adopting a homogeneous, plain vanilla ICFR audit process that is driven by fear, as opposed to a desire to better serve the public interest. ## REFERENCES Asare, S., Fitzgerald B., Graham L., Joe J., Negangard E., and Wolfe C. 2013 . Auditors' internal control over financial reporting decisions: Analysis, synthesis and research directions . Auditing: A Journal of Practice & Theory 32 (Supplement): 131 166 . Bamber, E. M., Snowball D., and Tubbs R. M. 1989 . Audit structure and its relation to role conflict and role . The Accounting Review 64 ( 2 ): 285 299 . Baron, J., and Hershey J. 1988 . Outcome bias in decision evaluation . Journal of Personality and Social Psychology 54 ( 4 ): 569 579 . Beasley, M., Carcello J., Hermanson D., and Neal T. 2009 . The audit committee oversight process . Contemporary Accounting Research 26 ( 1 ): 65 122 . Bricker, W. 2017 . Statement in connection with the 2017 AICPA conference on current SEC and PCAOB developments . Brown, C., and Solomon I. 1987 . Effects of outcome information on evaluations of managerial decisions . The Accounting Review 62 ( 3 ): 564 577 . Burgstahler, D., and Jiambalvo J. 1986 . Error projection and uncertainty in the evaluation of aggregate error . Auditing: A Journal of Practice & Theory 19 ( 1 ): 257 264 . Canadian Public Accountability Board (CPAB). 2012 . Report on the 2012 inspections of the quality of audits conducted by public accounting firms . Cohen, J., Krishnamoorthy G., and Wright A. 2002 . Corporate governance and the audit process . Contemporary Accounting Research 19 ( 4 ): 573 594 . Cohen, J., Krishnamoorthy G., and Wright A. 2010 . Corporate governance in the post-Sarbanes-Oxley era: Auditors' experiences . Contemporary Accounting Research 27 ( 3 ): 751 786 . Cohen, J., Krishnamoorthy G., and Wright A. 2017 . Enterprise risk management and the financial reporting process: The experiences of audit committee members, CFOs, and external auditors . Contemporary Accounting Research 34 ( 2 ): 1178 1209 . Croteau, B. T. 2013 . Audit Policy and Current Auditing and Internal Control Matters . Remarks Before the 2013 AICPA National Conference on Current SEC and PCAOB Developments, Washington, DC, December 9. Available at: https://www.sec.gov/news/speech/2013-spch120913btc Croteau, B. T. 2014 . Audit Policy and Current Auditing, Independence, and Internal Control Matters . Remarks Before the 2014 AICPA National Conference on Current SEC and PCAOB Developments, Washington, DC, December 8. Available at https://www.sec.gov/news/speech/2014-spch120814btc Dusenbury, R., Reimers J., and Wheeler S. 1994 . The effect of containment information and error frequency on projection of sample errors to audit populations . The Accounting Review 69 ( 1 ): 257 264 . Earley, C., Hoffman V., and Joe J. 2008 . Reducing management's influence on auditors' judgments: An experimental investigation of SOX 404 assessments . The Accounting Review 83 ( 6 ): 1461 1485 . Elder, R. J., and Allen R. D. 1998 . An empirical investigation of the auditor's decision to project errors . Auditing: A Journal of Practice & Theory 17 ( 2 ): 71 87 . Financial Reporting Council (FRC). 2011 . Audit inspection unit annual report 2010/11 . Fischhoff, B. 1975 . Hindsight does not equal foresight. The effect of outcome knowledge on judgment under uncertainty . Journal of Experimental Psychology. Human Perception and Performance 1 ( 3 ): 288 299 . Gendron, Y., and Power M. 2015 . Research forum on qualitative research in auditing . Auditing: A Journal of Practice & Theory 34 ( 2 ): 1 2 . Gendron, Y., Bédard J., and Gosselin M. 2004 . Getting inside the blackbox: A field study of practices in “effective” audit committees . Auditing: A Journal of Practice & Theory 23 ( 1 ): 153 171 . Gramling, A., O'Donnell E., and Vandervelde S. 2013 . An experimental examination of factors that influence auditor assessments of a deficiency in internal control over financial reporting . Accounting Horizons 27 ( 2 ): 249 269 . Gramling, A., Maletta M., Schneider A., and Church B. 2004 . The role of the internal audit function in corporate governance: A synthesis of the extant internal auditing literature and directions for future research . Journal of Accounting Literature 23 ( 1 ): 194 244 . Hermanson, H. M. 1997 . The effects of audit structure and experience on auditors' decisions to isolate errors . Behavioral Research in Accounting 9 (Supplement): 76 93 . Hermanson, D., Tompkins J., Veliyath R., and Ye Z. 2012 . The compensation committee process . Contemporary Accounting Research 29 ( 3 ): 666 709 . Hirst, D., and Koonce L. 1996 . Audit analytical procedures: A field investigation . Contemporary Accounting Research 13 ( 2 ): 457 486 . Joe, J. R., Vandervelde S. D., and Wu Y. 2017 . Use of high quantification evidence in fair value audits: Do auditors stay in their comfort zone? The Accounting Review 92 ( 5 ): 89 116 . Johnson, L. M., Keune M. B., and Winchel J. 2019 . U.S. auditors' perceptions of the PCAOB inspection process: A behavioral examination . Contemporary Accounting Research 36 ( 3 ): 1540 1574 . Knechel, W. R., Krishnan G., Pevzner M., Shefchik L., and Velury U. 2013 . Audit quality: Insights from the academic literature . Auditing: A Journal of Practice & Theory 32 (Supplement): 385–421 . Kral, R. 2017 . Management's annual report on ICFR . LaCroix, K. M. 2015 . PwC report reviews 2014 securities suit filings and settlements, analyzes trends likely to drive future suits. Available at: https://www.lexisnexis.com/LegalNewsRoom/securities/b/securities/posts/pwc-report-reviews-2014-securities-suit-filings-and-settlements-analyzes-trends-likely-to-drive-future-suits Latham, G., and Sue-Chan C. 1999 . A meta-analysis of the situational interview: An enumerative review of reasons for its validity . Canadian Psychology 40 ( 1 ): 56 67 . Lipe, M. 1993 . Analyzing the variance investigation decision: The effects of outcomes, mental accounting and framing . The Accounting Review 68 ( 4 ): 748 764 . Luippold, B. L., and Kida T. E. 2012 . The impact of initial information ambiguity on the accuracy of analytical review judgments . Auditing: A Journal of Practice & Theory 31 ( 2 ): 113 129 . Malsch, B., and Gendron Y. 2013 . Re-theorizing change: Institutional experimentation and the struggle for domination in the field of public accounting . Journal of Management Studies 50 ( 5 ): 870 899 . Malsch, B., and Salterio S. 2016 . Doing good field research: Assessing the quality of audit field research . Auditing: A Journal of Practice & Theory 35 ( 1 ): 1 22 . Miles, M., and Huberman A. 1994 . Qualitative Data Analysis: An Expanded Sourcebook . Thousand Oaks, CA : SAGE Publications, Inc . Nelson, M. W., and Kinney W. R. 1997 . The effect of ambiguity on loss contingency reporting judgments . The Accounting Review 72 ( 2 ): 257 274 . Panucci, M. 2016 . Remarks before the 2016 AICPA conference on current SEC and PCAOB developments . Peecher, M., and Piercey D. 2008 . Judging audit quality in light of adverse outcomes: Evidence of outcome bias and reverse outcome bias . Contemporary Accounting Research 25 ( 1 ): 243 274 . Public Company Accounting Oversight Board (PCAOB). 2007 . An Audit Of Internal Control Over Financial Reporting That Is Integrated With An Audit Of Financial Statements And Related Independence Rule and Conforming Amendments. Auditing Standard (AS) No. 2201 . Washington, DC : PCAOB . Public Company Accounting Oversight Board (PCAOB). 2011 . Assessing and Responding to Risk in the Current Economic Environment. Staff Audit Practice Alert No. 9 . Washington, DC : PCAOB . Public Company Accounting Oversight Board (PCAOB). 2012 a. Maintaining and Applying Professional Skepticism in Audits. Staff Audit Practice Alert No.10 . Washington, DC : PCAOB . Public Company Accounting Oversight Board (PCAOB). 2012 b. Observations from 2010 Inspections of Domestic Annually Inspected Firms Regarding Deficiencies in Audits of Internal Control Over Financial Reporting. Release No. 2012-006 . Washington, DC : PCAOB . Public Company Accounting Oversight Board (PCAOB). 2012 c. Report on 2011 Inspection of Ernst & Young LLP. Release No. 104-2012-272 . Public Company Accounting Oversight Board (PCAOB). 2013 a. Considerations for Audits of Internal Control Over Financial Reporting. Staff Audit Practice Alert No.11 . Washington, DC : PCAOB . Public Company Accounting Oversight Board (PCAOB). 2013 b. Report on 2012 Inspection of Ernst & Young LLP. Release No. 104-2013-146 . Public Company Accounting Oversight Board (PCAOB). 2014 a. In the Matter of KPMG LLP's Quality Control Remediation Submissions. Release No. 104-2014-166 . Public Company Accounting Oversight Board (PCAOB). 2014 b. Report on 2013 Inspection of BDO USA, LLP. Release No. 104-2014-232 . Public Company Accounting Oversight Board (PCAOB). 2014 c. Report on 2013 Inspection of KPMG LLP. Release No. 104-2014-167 . Public Company Accounting Oversight Board (PCAOB). 2015 . In the Matter of BDO USA, LLP's Quality Control Remediation Submissions. Release No. 104-2015-189 . Public Company Accounting Oversight Board (PCAOB). 2017 . PCAOB Publishes Staff Inspection Brief Previewing 2016 Inspection Findings. Available at: https://pcaobus.org/News/Releases/Pages/staff-inspection-brief-2016-preview-11-9-17.aspx Public Company Accounting Oversight Board (PCAOB). 2019 a. Staff preview of 2018 inspection observations . Public Company Accounting Oversight Board (PCAOB). 2019 b. Annual Report on the Interim Inspection Program Related to Audits of Brokers and Dealers. Release No. 2019-002 . Quaadman, T. 2015 . Re: PCAOB draft strategic plan 2018-2022 (PCAOB Release No. 2018-001, August 10, 2018) . Schnurr, J. 2015 . Remarks before the UCI audit committee summit . Scott, W. 1995 . Institutions and Organizations: Foundations for Organizational Science . London, U.K .: Sage Publication Series . Securities and Exchange Commission (SEC). 2007 . Commission Guidance Regarding Management's Report on Internal Control over Financial Reporting under Section 13(a) or 15(d) of the Securities and Exchange Act of 1934. Release Nos. 33-8810, 34-55929, FR-77; File No. S7-24-06 . Washington, DC: SEC. Available at: https://www.sec.gov/rules/interp/2007/33-8810.pdf Securities and Exchange Commission (SEC). 2019 a. SEC charges four public companies with longstanding ICFR failures . Securities and Exchange Commission (SEC). 2019 b. Spotlight on Financial Reporting and Audit Fraud Group . Trompeter, G., and Wright A. 2010 . The world has changed: Have analytical procedure practices? Contemporary Accounting Research 27 ( 2 ): 669 700 . U.S. House of Representatives. 2002 . The Sarbanes-Oxley Act of 2002. Public Law No. 107–204 . [ H.R. 3763 ]. Washington, DC : Government Printing Office . U.S. House of Representatives. 2010 . The Dodd-Frank Wall Street Reform and Consumer Protection Act. Public Law No. 111-203 . [ H.R. 4173 ]. Washington, DC : Government Printing Office . Westermann, K., Bedard J., and Earley C. 2015 . Learning the “craft” of auditing: A dynamic view of auditors' on-the-job learning . Contemporary Accounting Research 32 ( 3 ): 864 896 . Westermann, K., Cohen J., and Trompeter G. 2019 . PCAOB inspections: Public accounting firms on “trial.” . Contemporary Accounting Research 36 ( 2 ): 694 731 . Wheeler, S., Dusenbury R., and Reimers J. 1997 . Projecting sample misstatements to audit populations: Theoretical, professional and empirical considerations . Decision Sciences 28 ( 2 ): 261 278 . ### APPENDIX A #### Key Facts Case Contexts ##### Case 1 • Client is a large publicly traded national financial services company with subsidiary offices across the U.S. The company has a decentralized accounting system but all subsidiaries follow a common set of financial accounting and internal control, and operational policy and procedures, which are established at the national headquarters. Subsidiaries submit a monthly report for national office. • The SEC issued a new rule for financial companies servicing retail customers which requires them to locate customers who might not be aware that they have assets greater than25 owed to them.

• During ICFR testing the auditor discovers that a small percentage of the locations selected for testing had not implemented the “search and notification” changes in operating procedures imposed by national headquarters. The lack of implementation was attributed to incomplete/ineffective updating of the local office information system. The financial statement exposure (limited to fines and penalties) from this deficiency exceeds the tolerable misstatement but is less than materiality.

• Although the client did not identify this control deficiency, management believes that their internal audit department would have likely detected this weakness during a routine audit and therefore presents a strong mitigating entity-level control.

• The audit team also found that some of the company's securities were inappropriately classified as “available for sale” securities instead of as “trading securities” in the client's 4th Quarter financial statements due to a misapplication of the securities by the subsidiary offices to corporate. Misclassifying these securities resulted in the unrealized loss being reflected as a reduction in other comprehensive income. If the securities had been properly classified as trading securities the unrealized loss would have been reflected in the income statement.

• Management believes that because there is a detailed management review control of the annual financial statements before they are filed, a material error would have been prevented from occurring in the annual financial statements. The control was designed to operate at the corporate level and is the only management review control performed.

##### Case 2
• The client is a large regional publicly traded real estate investment firm based in the Northwest. In addition to real estate holdings, the company invests in real estate related securities such as Mortgage Backed Securities (MBS) and Collateralized Debt Obligations (CDOs). The company is eligible to and has selected the “fair value” option for each of its real estate holdings.

• During the interim ICFR testing, the audit team identified a control deficiency in the valuation of the company's income producing properties. An analyst had applied the incorrect interest rate in the discounted cash flow analyses used to determine fair value. The error was not detected by the client and resulted in an overstatement of the property asset account in the 1st Quarter financial statement filing. The error approached, but did not exceed, materiality for the quarter. The controller's review control over this process is based on system-generated data and reports produced by the entity's information system.

• The audit team is in the midst of the year-end substantive testing phase of the audit. While testing the accuracy of the fair value of the client's financial instruments, the team detected a computational error in the valuation model used for the MBS investment, which led to an overstatement of “trading securities” portfolio. As a result of the error, unaudited pre-tax income had been overstated by approximately 3.27% (on a year-to-date basis) and total assets were overstated by less than 1%. The amount is below materiality. Management agreed to record the proposed audit adjustment to correct the error.

• Management believes that the two errors described above are immaterial and that any error that approaches a material level would have been detected by the controller who completes a robust management review control (part of which is described above). Management also believes that their review control operates at a precise enough level to prevent these types of overstatements to the property and fair-value securities accounts.

### APPENDIX B

#### Pre-Interview Questions Within Case Contexts

##### Case 1
1. How would you classify each of the items detected by your team? (Deficiency; Significant deficiency; Material weakness; None of the above):

• The “Lost Holders” issue:

• What factors most influenced your decision (i.e., what caused you to choose your preferred option (above) instead of one of the other possibilities)?

• The “misclassification” issue:

• What factors most influenced your decision (i.e., what caused you to choose your preferred option (above) instead of one of the other possibilities)?

2. If you had a client, and these were the only two deficiencies (lost holder and classification), how would you view these two deficiencies, in combination: (Deficiency; Significant deficiency; Material weakness)?

• What factors most influenced your decision (i.e., what caused you to choose your preferred option (above) instead of one of the other two possibilities)?

3. What opinion do you believe is warranted for [this client's] ICFR audit?

• Unqualified

• Disclaimer of Opinion

• Adverse (due to detected Material weakness)

• a.

What factors most influenced your decision (i.e., what caused you to choose your preferred option (above) instead of one of the other two possibilities)?

4. Would past experience with this client potentially have affected your responses above? If yes, please explain how it would?

5. What evidential matter would you typically seek to obtain to support your responses in number 4 above? Please elaborate on how this evidential matter might specifically help in developing your response above.

6. If there were no material misstatements found on the audit but you still believed there are weaknesses in the controls for the above processes, what evidence would you need to obtain in order to finalize your control opinion?

7. Assuming that your audit team concluded that one of the detected ICFR deficiencies noted above should be classified as a material weakness and management disagreed, how would you explain your position to management such that they could be in a position to agree with your conclusion, even if no material misstatement in the financial statements was found?

8. In making the decisions above, how much do you rely on each of the following and explain how you would rely upon them:

• The technical resources available in your firm's automated database

• Other members of the audit team

• Quality review partner (or other partners)

• Technical experts at your firm's national office

##### Case 2
1. How would you classify each of the items detected by your team? (Deficiency; Significant deficiency; Material weakness; None of the above):

• The “valuation of income producing properties” issue:

• What factors most influenced your decision (i.e., what caused you to choose your preferred option (above) instead of one of the other possibilities)?

• The “valuation of MBS investment” issue:

• What factors most influenced your decision (i.e., what caused you to choose your preferred option (above) instead of one of the other possibilities)?

2. Would you view these two deficiencies, in combination, to be a: (Deficiency; Significant deficiency; Material weakness)?

• What factors most influenced your decision (i.e., what caused you to choose your preferred option (above) instead of one of the other two possibilities)?

3. What opinion do you believe is warranted for [this client's] ICFR audit?

• Unqualified

• Disclaimer of Opinion

• a.

What factors most influenced your decision (i.e., what caused you to choose your preferred option (above) instead of one of the other two possibilities)?

4. Explain whether the substantive error in fair value measurement influenced your assessment of the internal controls over financial reporting at [this client]. Discuss the reasons why it did or did not influence your selection in the appropriate audit opinion.

5. Would past experience with this client potentially have affected your responses above? If yes, please explain how it would?

6. What evidential matter would you typically seek to obtain to support your responses to number 5 above? Please elaborate on how this evidential matter might specifically help in developing your response above.

7. If there were no material misstatements found during the audit but you still believed there are weaknesses in the controls for the above processes, what evidence would you need to obtain in order to finalize your control opinion?

8. Assuming that your audit team concluded that one of the detected ICFR deficiencies noted above should be classified as a material weakness and management totally disagreed, how would you explain your position to management such that they could be in a position to agree with your conclusion, even if no material misstatement in the financial statements was found?

9. In making the decisions above, how much do you rely on each of the following and explain how you would rely upon them:

• The technical knowledge available in your automated workpaper platform

• Other members of the audit team

• Quality review partner (or other partners)

• Technical experts at your firm's national office

1

An internal control deficiency exists when either the design or operation of the control under consideration does not allow the entity's management or employees to detect or prevent misstatements in a timely fashion. More serious internal control deficiencies can be categorized into one of two groups, significant deficiencies or material weaknesses, depending on their severity (PCAOB 2007). The standard describes a material weakness in internal control as a deficiency, or combination of deficiencies, that results in a reasonable possibility that a material misstatement would not be prevented or detected on a timely basis. Moreover, a significant deficiency is a deficiency or a combination of deficiencies in internal control that is less severe than a material weakness yet important enough to merit attention from those charged with governance.

2

Approval for this human subjects study was granted by the Institutional Review Board (IRB).

3

Doctoral students served as transcribers because the participating accounting firms did not permit digital recording of the interviews. The doctoral students were given only general information about the project, specifically, that it was designed to understand how auditors make judgments about internal control deficiencies on ICFR audits.

4

Two co-authors independently read and initially coded the materials into summary categories (that were based upon PCAOB [2013a], participants' responses, and Asare et al. [2013]). The co-authors then met to refine the final list of themes, which were used by the research assistants to code the transcripts. The final list of themes is presented in Table 2.

5

This includes responses made during participants' comments on the aggregation of control deficiencies.

## Author notes

We acknowledge research support from the Center for Audit Quality (CAQ). We thank members of the Research Advisory Board at the CAQ for their assistance in developing and refining the case materials used in this study. Additionally, we would like to thank the participants who took part in the study. We especially want to thank our editor Dana R. Hermanson for providing such timely and insightful guidance. We also want to acknowledge Kim Westermann, Lori Holder-Webb, Steve Salterio, Kelly McKenna, the Nyenrode Business University and North Carolina State University research workshop participants and participants at the 2017 Midyear Audit conference for comments on earlier drafts.

The views expressed in this article and its content are those of the authors alone and not those of the CAQ. Please note that this paper was the 2019–2020 recipient of the Glen McLaughlin Prize for Research in Accounting Ethics from the Steed School of Accounting (The University of Oklahoma).

Jeffrey R. Cohen, Boston College, Carroll School of Management, Department of Accounting, Boston, MA, USA; Jennifer R. Joe, University of Delaware, Lerner Business & Economics, Department of Accounting and Management Information Systems, Newark, DE, USA; Jay C. Thibodeau, Bentley University, Department of Accounting, Waltham, MA, USA; Gregory M. Trompeter, College of Business, Kenneth G. Dixon School of Accounting, University of Central Florida, Orlando, FL, USA.

Editor's note: Accepted by Dana R. Hermanson, under the Senior Editorship of Christopher P. Agoglia.