Building Higher-Order Thinking Skills Through Assessment of User Account Access

Innovative technologies and marketplace demands are constantly evolving, requiring those in the accounting profession to deepen their skillsets to work with emerging technologies and understand topics such as business process and system risk, controls, data management, and data analysis. The accounting profession and professional organizations support the continual enhancement of student’s information technology (IT) knowledge as well as higher-order thinking skills (HOTS) during educational studies to prepare students for their careers (AACSB, 2020; AICPA, 2022a; PwC, 2020). The American Institute of Certified Public Accountants (AICPA) Foundational Competencies Framework for Aspiring CPAs (2022a) states that students should be able to (1) describe necessary controls to mitigate risks, (2) evaluate the effectiveness of the controls, (3) compare technological tools to recommend the best option(s) given time and cost constraints and job specifications, (4) leverage relevant technologies to facilitate data analyses that support organizational goals, and (5) evaluate alternative answers/decisions to issues/problems/question (p. 2). Additionally, CPA Evolution, a joint project between the AICPA and the National Association of State Boards of Accountancy (NASBA), aims to transform the Certified Public Accountant (CPA) model to recognize these skillsets and competencies that CPAs will need as they move into their careers (AICPA, 2021b). The Uniform CPA Examination Blueprints require students to pass three core sections: Financial Accounting and Reporting, Taxation and Regulation, and Auditing and Attestation. Within the Auditing and Attestation section, students must apply knowledge of an entity’s control environment, including the design and implementation of IT general controls and entity-level controls (AICPA, 2022b). In the case study, students apply knowledge of procedures to understand how an entity has responded to risks arising from the use of IT, including identifying and testing the design and implementation of relevant IT general controls. Additionally, students evaluate the effective design and implementation of relevant automated and manual transaction-level internal controls.

In the higher education literature, researchers consider creative, analytical, and evaluative thinking skills as the most advanced of the HOTS (Bloom, 1956; FSSE, 2022). Prior research suggests that the best approach to developing HOTS is to embed them within the technical content of the curriculum instruction (Bunney et al., 2015; De Villiers, 2010; Dickins & Reid, 2022). As students develop the peak skills of HOTS (creating and evaluating), they use the data to create information and evaluate the results.

Accounting firms value creating and evaluating HOTS due to the growth and advancement of technology replacing the need for mere technical knowledge (De Villiers, 2010). Many technologies, such as robotic process automation and artificial intelligence, routinely apply and analyze data (Kokina & Davenport, 2017); accountants must be adept at creating information and evaluating the output (PwC, 2017). Therefore, faculty can use certain instructional materials and case studies to simultaneously teach students specific technology or software and how to build on their creation and evaluation HOTS. This case study allows students to think creatively and evaluate output to address data-driven questions. Students’ mastery of creative and evaluative skill sets is critical to competency development and career success.

Bloom’s Taxonomy (1956) contextualizes the levels of intellectual skills, beginning with a lower-order thinking mindset (remembering, understanding, applying) and moving to a higher-order thinking mindset (analyzing, evaluating, and creating). As one advances up the pyramid, human cognition progresses from a lower- to a higher-order thinking mindset. The peak of the pyramid includes analyzing, evaluating, and creating. Analyzing is defined as going “beyond knowledge and application, actually being able to see patterns that they can use to investigate a problem” (Bloom, 1956, p. 22). Evaluating requires “assessment of information and concluding such as its value or the bias behind it” (Bloom, 1956, p. 23). Lastly, creating combines elements to form an original product (Conklin, 2005). The 2017 PwC 20^th CEO survey results stated that “the hardest skills to find are those that machines cannot perform. Seventy-seven percent of CEOs agree that it is difficult or somewhat difficult to find/recruit people with creativity and innovation skills” (PwC, 2017).

Higher-education scholars contend that if faculty members view these higher-order skills as important, they will incorporate practices that promote them into their teaching curriculum (Lattuca & Stark, 2009). Pedagogical practices, such as case-based teaching, have proven to promote HOTS in students (Hall et al., 2004) and encourage students to engage in a deep level of learning (Biggs, 1987). When students engage in a deep level of learning, they participate in meaningful learning, where they study to understand the material thoroughly and, in turn, enhance their HOTS (FSSE, 2022).

The objectives of the case are twofold: students 1) build analytical and evaluative HOTS and 2) gain technical knowledge on IT general controls that companies use to support business processes. External and internal auditors increasingly prioritize topics like IT general controls due to the growing reliance on IT systems (Dzuranin & Mălăescu, 2015). Tapis et al. (2020) state that “the use of technology in accounting is accelerating, with the profession calling for more technology knowledge, skills, and abilities to be integrated into accounting” (p. 26). In their article, the authors explore how these topics impact students and educators in academia today, as the “hot position” of an information technology auditor is gaining recognition for the upcoming decade (Tapis et al., 2020).

Effective IT general controls support the information provided by a company’s IT systems: “General IT controls are policies and procedures that relate to many applications and support the effective functioning of application controls” (AICPA, 2021a, p. 326). IT general controls have three classifications: 1) access management, 2) change management, and 3) operations. This case focuses on a specific portion of access management. Controls surrounding access management reduce the risk of unauthorized access to data that could allow individuals to access, modify, or delete critical financial data. Access management controls ensure that authorized and appropriate users gain access to the IT environment, while restricting them to performing authorized and appropriate actions. Three categories of access controls include 1) user administration, 2) authentication, and 3) security. This case focuses on authentication access controls, specifically user access reviews (refer to Figure 1).

A user access review is “a control to periodically verify that only legitimate users have access to applications or infrastructure” (Ramaseshan, 2019, p.1). In the field, auditors can detect if only authorized users have access to the system and if transferred or terminated users have access revoked quickly, as unauthorized users increase the risk of theft, fraud, or exploitation of vulnerabilities. Testing of these controls can be completed through (1) provisioning testing—obtaining a system-generated list of new and modified users from each key application or system and reconciling the list to a new hire / transfer system-generated listing provided by human resources, and (2) deprovisioning testing—obtaining a system-generated list of terminations within the period from human resources and comparing the human resource system-generated company termination date to the application or system termination date to ensure access was removed quickly (i.e., per company policy).

Testing user access management controls reduces the risk of unauthorized access to data and requires using HOTS. First, users must understand how the control mitigates the risk of unauthorized access so they can appropriately test the control. Secondly, the user applies this control to the company’s circumstances to perform testing (i.e., what documents to collect to perform testing and the directionality of testing). After testing, users create and analyze a table of results and evaluate the results to formulate an appropriate conclusion.

Lee and Sawyer (2019) provide a literature review of supplemental instructional cases related to IT controls and create a robust case study focused on user access management within Excel. Students gain an understanding of IT general controls and advanced Excel functions, and they document the operating effectiveness of user access management controls. This instructional case differs in that students test IT controls using two different software packages (Excel and IDEA), in response to the new foundational competencies required by the AICPA Foundational Competencies Framework for Aspiring CPAs (AICPA, 2022a).

In its entirety, the case reveals how faculty can teach technical IT accounting skills while enhancing students’ analytical and evaluative HOTS. Students perform data analytics through IT audit testing in Excel (part A) and IDEA software (part B). Students test to ensure all current system users are authorized and that terminated or transferred users’ access is revoked in a timely manner. In part A of the case, students perform provisioning and deprovisioning testing in Excel. In part B of the case, students perform similar analytics in IDEA. Students build on the peak of HOTS, creativity, when designing appropriate test steps. In parts A and B, students evaluate the audit findings. Therefore, this case not only allows students to build on their creative, analytical, and evaluative HOTS but also provides insight for faculty on how to blend the teaching of technical content with soft skill development.

The case assumes the student accepted a new position as an internal audit associate. The newly engaged associate assists their leadership team in performing user access reviews, specifically testing provisioning and deprovisioning controls. The senior presents the new associate with a variety of documentation, requiring them to filter through the data and files. The new associate then performs appropriate test steps using the software requested by the company. Refer to the Appendix for specific case details. In part A of the case, the associate performs provisioning and de-provisioning tests using Excel. Upon completion of part A of the case study, the student should be able to:

1. Understand foundational knowledge of IT general controls and IT auditing.

2. Apply knowledge of IT testing to perform data analytics within Excel.

3. Create appropriate test steps and information to analyze.

4. Analyze and evaluate IT testing results to determine an appropriate conclusion.

   In part B of the case, the associate moves firms. At the new firm, the associate assists with de-provisioning tests using a different software, IDEA. Upon completing part B of the case study and discussion within the classroom, the student should be able to:

5. Apply knowledge of IT testing to perform data analytics within IDEA.

6. Analyze and evaluate IT testing results to determine an appropriate conclusion in the information created.

7. Analyze and evaluate the key differences between the two data analysis software tools.

During class discussions, faculty members emphasize the importance of creative, analytical, and evaluative HOTS throughout the case study. The presentation slides and teaching notes provide further details about each skill set.

Efficacy is measured in two ways: 1) non-author review and analysis and 2) student surveys of the case. Three non-authors reviewed and provided feedback on the case study. First, a former Assistant to the Vice President for Finance and Administration at a Tier 1 research institution and lecturer with 17 years of teaching experience reviewed the case study and implemented it in their graduate-level accounting systems course. The instructor stated,

• “The case was very valuable in the graduate-level systems course, which dedicates significant time to IT risk and how to audit and evaluate IT general controls. It is often difficult to find cases and activities using ITGC-related data, which provide students with hands-on practice and an introduction to ITGC concepts. This access testing allows students to take the concepts learned about access management and apply them in a realistic setting. Students must place themselves in the role of the ‘process owner’ and 1) identify the risks associated with system access and employee terminations and then also 2) design and create a tool (Excel or IDEA) and the process by which access can be monitored and the ‘process owner’ alerted when access changes are needed.

  Students find this case challenging because they are provided only foundational guidance and must design the process themselves. In addition to using this case with Excel and IDEA, I have also assigned it with Alteryx. This case is very flexible and allows the instructor to modify the data easily with other scenarios. For example, the instructor can add users with access who are not on the employee master file and introduce employees going on extended leave and returning.”

  Second, a director in the Cloud & Digital Consulting division of a Big 4 accounting firm shared the following.

• “This case provides a practical and well-defined structure by which the learner can experience the cohesive process that our audit teams follow during control testing. The key value, in my perspective, is twofold. Firstly, the learner is encouraged to consider the rationale for the methodology; participants are challenged to consider ‘what are you testing’ instead of simply following instructions ‘because that is how we do it.’ This highlights the need for our professionals to be well-versed in our audit methodology and process, and the rationale for our test conditions and underlying business value. Secondly, the blunt practicality of the description from the Senior Associate highlights the reality of continuous on-the-job training for an auditor of any level. It could be argued that the detailed instructions provided by the senior are significantly more valuable than what one would likely see in the field, but the pedagogical imperative remains, where the learner experiences being asked to perform a test for the first time with limited information.”

Third, a faculty member teaching an undergraduate information systems assurance course at a Tier 1 research institution implemented portions of the case study. A total of 76 students completed the variation. Students performed both the provisioning and de-provisioning tests as part of a larger case focused on testing IT control design. The assistant professor stated,

• “The data from this case was easily transferrable into their current case study. My 76 students gained practical experience with technology and were forced to think critically about how to interpret potential exceptions.”

This case provides instructors the flexible opportunity to use the case materials to supplement their curriculum materials.

Additionally, students completed a survey at the end of the case study to assess their perception of the case. A Tier 1 research institution implemented the full case study over four years in a graduate AIS course. A total of 13 students completed the case study. The survey included six Likert-type scale questions (where 1 = strongly disagr 2 = disagree, 3 = neutral, 4 = agree, and 5 = strongly agree) and an open-ended feedback question. Table 1 reflect that, on average, students agreed that the case study was interesting and developed their understanding of how to perform data analytics in Excel and IDEA. Students agreed that the case study increased their understanding of th efficiencies that a new emerging software provides and improved their evaluative and creative skill sets. Students believed that, overall, this case study was a beneficial learning experience. Student responses were recoded as 0 o with a response of 1, 2, or 3 = 0 (disagree) and a response of 4 or 5 = 1 (agree). The percentage of students that agreed with Q1–Q6 included 94%, 86%, 94%, 90%, 90%, and 94%, respectively. A chi-square test revealed significant agreement in all cases (p < 0.05).

Furthermore, mean differences were compared among the years using an independent-samples t-test and an analysis of variance (ANOVA). The independent-samples t-test revealed no significant differences in the mean values for Q1–Q6 between any of the years (2017 and 2018, 2018 and 2019, and 2017 and 2019). The ANOVA results did not yield any significant differences among the years, suggesting no statistically significant mean differences. After the non-significant results of the ANOVA tests, a follow-up Scheffe’s test was conducted to further investigate potential differences. The Scheffe’s test results indicated that there were no significant differences observed between any pair of years (p > .05). This suggests that there are no statistically significant mean differences among the years being compared.

Lastly, for the open-ended question, the following includes examples of student feedback.

• “I think the value of the audit trail that IDEA adds is a big deal. Excel is fantastic, but it is hard to implement strong controls in Excel, which can make for significant risks when in practice. I think with more practice in IDEA the time savings would be even greater, and the depth of analysis would likely grow as well.”

• “I think the case study is overall beneficial. Starting to introduce students to programs other than excel is going to help them succeed long-term as companies become more adamant on using the software they have invested so heavily in.”

• “I think it was beneficial as it gives you an idea of what types of things we will be doing when we start working and how to use different tools to perform the tasks. Also, I am glad I was exposed to IDEA and got an opportunity to practice in it.”

• “Overall, I thought it was beneficial because, for someone that struggled with the assignment in excel, it was much easier to navigate in IDEA, and from the assignment, I got to see the difference between using the two software packages and how to evaluate each.”

Other feedback included technology struggles for the students. Examples of this type of student feedback included:

• “I thought the length of the case was appropriate and analyzing our audit findings connected the technology with its usability. For improvements, working with our University Technology Support to make the IDEA software more reliable on the computers could create a more positive experience.”

Due to the challenges technology can present (i.e., Excel crashing, IDEA errors), this allows for a greater learning experience for the students and teaching opportunities for the instructor. When students encounter road bumps along the way and technology does not seamlessly work every time, faculty can reinforce this concept and encourage students to look for solutions.

During the full case study implementation, students recorded their start and completion times during the deprovisioning testing in Excel. By collecting the time measurements, students could easily understand the time efficiencies achieved by using emerging technology software after completing the case study. For example, within part A of the case study, students recorded the time when starting and completing de-provisioning testing in Excel. The completion time range was 20 minutes to 2.5 hours (n=139). Second, students recorded the time when starting and completing de-provisioning testing in IDEA. The range of completion time was 5 to 25 minutes (n=139). After the case, the faculty shared the time differences with students. During the discussion, students could easily identify how emerging data analytic software can provide great time savings and efficiencies for auditors.¹ It is important that, during the class discussion, faculty share the importance of not being technological tool dependent.² The instructor should also share with students that there is a learning curve for all emerging technology software that includes a time investment. After this, future projects benefit from further time savings and effective data analysis. While there are limitations in collecting and measuring this time-saving data, overall, this data collection intends to highlight an essential element of time efficiency and effectiveness.

Overall, this case study provides an example for accounting educators seeking to understand how to improve accounting students’ HOTS while improving students’ knowledge of business systems, risk, IT controls, auditing, and emerging technology. Students learn how to work with emerging technologies in the profession, how to evaluate a situation and formulate answers for data-driven questions. Higher education scholars contend that purposeful teaching to promote higher-order thinking contributes to developing critical thinking skills (Miri et al., 2007). The more time faculty can devote to understanding the integration of HOTS and technical content, the better prepared a student is for their future career.

I express my gratitude to the numerous students from the University of Georgia who engaged with and completed this case, offering valuable feedback that played a pivotal role in making substantial enhancements. Additionally, my sincere thanks extend to the anonymous reviewers and the editors, whose insightful feedback greatly contributed to the significant improvement of this manuscript.

Congratulations! You have recently started as a new internal audit associate with R&R, Incorporated. The internal audit department at R&R, Inc., includes multiple teams. Your first year involves rotations with each internal audit team; your first assigned team is the information technology team. The senior on your engagement has asked that you help him with user access testing, specifically provisioning and de-provisioning tests. You are familiar with this type of audit testing based on your previous class work; however, a quick refresher search reminds you that user access testing focuses on granting access to users, termination of access, and access reviews. A quick search of R&R’s handbook informs you that the company grants every new employee access to an application or system on their first day. If an existing employee is transferring departments, access is granted within 24 hours. You remember from your coursework that appropriate user access is essential to maintaining a good internal control structure, as inappropriate access could result in a major risk for the company. Additionally, the handbook states that the company revokes an employee’s access within 24 hours if they are terminated or move departments. You know that timely termination of access is also key to maintaining an appropriate internal control structure and avoiding major risks.

Your senior sends you an email with the following notated in the body of the email:

Good morning,

Attached is the spreadsheet you will need to use for testing. I do not have time to schedule a meeting with you right now, but I created a list to explain at a high level what each tab entails. Below the list includes a detailed explanation of each tab and directions for you. Please reach out with any questions!

• Master Worksheet: Documentation of test steps and results

• New.ModAppUsers-1.1-12.31: System-generated listing of all new and modified users to each key application from 1/1/2022 to 12/31/2022

• HR.NewHires.Transfers-1.1-12.31: Spreadsheet provided by HR and is a listing of all new hires and transfers within each department from 1/1/2022 to 12/31/2022

• HR.TermUsers-1.1.-12.31: Spreadsheet provided by HR of all users who were terminated during the audit period with their termination date

• All tabs after (i.e., Kanlam through Zoolab): System-generated lists from each key application that include the dates of when a specific user’s access was terminated from the application

The master worksheet tab includes the provisioning and de-provisioning tests you will complete. Access testing is at the top of the page, and terminations testing is at the bottom. This tab will include all your test steps and document your results. Further details for logical access and terminations testing are provided below.

For logical access testing, you will examine new and modified user access within the company. The control you are responsible for testing is “Access to all applications is only granted to R&R, Inc., new hires or employees.” The following logical access test step is written in your work papers: “Test Step: Obtain a system-generated list of new and modified users from each key application within the audit period. Reconcile the list to the new hire/transfer listing provided by HR.”

We include more details within the testing work paper so that a new associate will understand the exact process and what we are testing next year. Before you begin your testing, please fill in the answer to the question “What are you testing?” To help you answer this, first think about the files you received that will be helpful in this testing procedure. The tab “New.ModAppUsers-1.1-12.31” is a system-generated listing of all new and modified users to each key application from 1/1/2022 to 12/31/2022, and the tab “HR.NewHires.Transfers-1.1-12.31” is a listing provided by HR of all new hires and transfers within each department from 1/1/2022 to 12/31/2022. In addition, consider the risks presented to the company if inappropriate access is granted to an application (i.e., access to an application is granted to a non-R&R, Inc., employee). In your answer, please include the names of the two tabs that you are testing and the appropriate direction to perform the testing (i.e., are you testing to ensure that all users on the system-generated list of new/modified users are also on the HR new hire / transfer listing or testing to ensure that all users on the HR new hire / transfer listing are on the system-generated list of new or modified users? Think about which direction poses the most risk to the company). Once you document this, perform the testing. If exceptions exist, document them in the Master Worksheet and include a sentence or two on your next steps regarding these exceptions.

Next, you will test the control that states, “Access to all applications are terminated within one day (24 hours) of when employees leave R&R, Inc.” The following tabs are provided to help you perform the testing. The tab “HR.TermUsers-1.1.-12.31” is a spreadsheet provided by HR of all users who were terminated during the audit period with their termination date. All tabs at the end of the workbook (i.e., Kanlam through Zoolab) are systemgenerated lists from each key application that include the dates of when a specific user’s access was terminated from the application. Company policy is that all application access should also be terminated within 24 hours, regardless of weekdays or weekends. Therefore, you need to test that company policy is being followed and that all users are terminated within 24 hours. (It may be helpful to start by consolidating all the information listed on the application tabs into one excel tab. Then proceed with your testing. However, feel free to test as you see best.)

As I mentioned, we are trying to update our documentation, so before beginning testing, you need to create your test step. Use the access testing test step to create your test step. Be as specific as possible when creating your test step without including the names of tabs because those may change from year to year. Then, answer the next question “What are you testing?” Again, consider the risks presented to the company if access is not terminated in an appropriate timeframe.

After writing your test step and answering the questions, enter the start time of when you begin testing terminations (we will discuss why this is important later). Begin your terminations testing. If exceptions exist, include them in the Master Worksheet. Once you have completed your testing and documented any exceptions in the Master Worksheet, record your end time.

Lastly, think about what your next steps should be regarding these exceptions. Record your answer in the Master Worksheet. As I mentioned earlier, if you have questions, please reach out!

Thanks again and talk soon!

-Adam S.

Complete the Excel Testing (S) Master Worksheet by performing the following steps:

• Open the “Master Worksheet” tab in the “Excel Testing (S).xlsx” workbook.

• Document what you are testing with the logical access test step.

• Perform logical access testing using Excel.

• Document any exceptions, and if exceptions exist, document your next steps.

• Create the terminations test step.

• Document what you are testing with the terminations test step.

• Record your start time.

• Perform terminations testing using Excel.

• Document any exceptions.

• Record your end time.

• If exceptions exist, document your next steps.

After you completed your user access testing within Excel, you, unfortunately, had to move to a new state and company. You liked your past work, so you decided to stick with IT Audit. Now, you are employed with Riley Parks.

On your first day of work, your manager assigns you termination testing. (At least you have had some practice! You are testing a very similar control as your last engagement with R&R, Inc. The control in your new company specifies that the company terminates access to all applications within one day of employees leaving the company. However, instead of completing the testing in Excel, Riley Parks has a software called Caseware IDEA that they use to perform audit testing. You are provided with very similar documents to use in your testing. Your testing workbook is labeled “IDEA Testing (S).xlsx.” The following is a high-level overview of each tab provided within the workbook.

• Master Worksheet: Documentation of test steps and results

• HR.TermUsers-1.1.-12.31: Spreadsheet provided by HR of all users who were terminated during the audit period with their termination date

• All tabs after (i.e., Sheet 1 through Sheet 25): System-generated lists from each key application that include the dates of when a specific user’s access was terminated from the application.

Your test step states, “Obtain a system-generated list of terminations within the audit period from HR. Compare the system-generated company termination date to the application termination date to ensure access was removed in a timely manner, within company policy of 24 hours.” Based on your previous experience, you know you are testing to ensure that all users’ application termination dates are within 24 hours of the termination date provided by the HR system-generated termination listing. However, instead of completing the testing in Excel, you will complete the testing in Caseware IDEA. All tabs (except for the master worksheet) will need to be uploaded into IDEA to perform the testing.

Before uploading files into IDEA, enter your start time on the Master Worksheet tab. After import, check to ensure date columns are imported correctly into IDEA (as a date field). If the date column is not a date field and is uploaded as a character field instead of a date type, it needs to be changed. Therefore, change the column type from a character column to a date column with the mask “DD/MM/YYYY.” Now you can begin your testing. Once you have finished testing and identified any exceptions, screenshot your results to include in your testing workbook. Once complete, enter your finish time.

Review your findings and follow up. Do you see any trends? If so, document your results.

Complete the IDEA Testing (S) Master Worksheet by performing the following steps:

• Open the “Master Worksheet” tab in the “IDEA Testing (S).xlsx” workbook.

• Record your start time.

• Perform terminations testing using IDEA.

• Document any exceptions.

• Record your end time.

• If exceptions exist, document your next steps.