Designing and Developing Exercises to Enhance Capability - A Meta Analytic View

While prevention and the mitigation of risk are key guiding principles relating to potential safety, security, health and environmentally-related (SSHE) incidents, emergency preparedness and response (EP&R) efforts are equally important aspects of a holistic view of the risk-management world that considers both the antecedents and potential consequences of an incident. (Figure 1).

An important part of the EP&R focus is on regular and meaningful response exercises as a means to help improve and sustain organizational capabilities, especially as personnel move into different roles. With this in mind, and especially with respect to many large organizations, it is necessary to ensure that timely and effective response capabilities are maintained and that response personnel proficiency is supported through an ongoing, recognized training process, e.g., an approach that is consistent with an Incident Command/Management System (ICS/IMS) or International Maritime Organization (IMO) framework. (IPIECA-IOGP 2015; FEMA 2011) For example, the IMO has been working to revise training modules to “provide up to date guidance for preparedness and response to marine oil spills.” (IMO 2017) A number of key components associated with training can be effectively obtained during a variety of exercises, and depending on the scope of the exercises, they can be used to: (API, 2014)

• Clarify team roles and responsibilities

• Improve response coordination and communications internally and externally

• Assess and validate policies, plans, procedures, training, equipment, assumptions, and contract or mutual aid agreements

• Identify potential gaps in the resource supply chain

• Identify various opportunities to improve planning efforts and enhance future exercise programs

A variety of potential response exercise objectives for a broad suite of response personnel are provided in Table 1 below. (IOGP-IPIECA, 2014)

• Utilize National Oil Contingency Plan

• Utilize company Contingency Plan for Abatement of Marine Oil Spills

• Employ incident management team (IMS) & organization structure

• Develop & implement site safety & health plan

• Develop & implement incident action plan (IAP)

• Ensure Incident Commander clearly states incident response priorities

• Implement improved planning session & meeting effectiveness processes

• Enable effective response team intra-section communications

• Enable effective national inter-agency communications

• Optimize the use of oil industry & government agency personnel diverse skills

• Demonstrate effective integration of industry representatives and government agency personnel

• Activate & operate on-scene incident command center

• Identify & prioritize protection of environmentally sensitive areas (ESA’s)

• Identify & prioritize protection of economically important areas (EIA’s)

• Implement the national oil spill dispersant use approval process

• Exercise oil spill dispersant delivery logistics, deployment and monitoring capabilities

• Exercise ability of Oil Spill Response Organization (OSRO) to obtain response equipment customs clearances in a timely fashion

• Develop effective oil spill response containment, recovery, and cleanup work plans

• Develop effective oil spill response equipment logistical, deployment and demobilization plans

• Identify & address incident-related security concerns

• Implement claims management & settlement process and establish hot line

• Manage & document daily and cumulative oil spill cleanup costs

• Ensure Incident Commander clearly communicates key external messages to Public Affairs and the Public Information Officer and that press releases are consistent with those messages

• Conduct initial & periodic response team incident status update briefings

• Implement process for documentation of key response team actions & decisions

• Document key response team decisions using appropriate tools

• Communicate exercise status through use of Public Affairs internet site

• Manage media & community expectations and concerns as a joint local and regional effort

• Document exercise learnings, next steps, and gap closure plans

Having well defined objectives at the outset of an exercise is important, but it is equally important to compile lessons learned at its completion. In particular, the identification of possible areas for improvement while fresh in everyone’s minds is of value, whether developed from the written evaluator feedback or as part of section member debriefs (the “hot wash”) and After Action Reports (AAR), since the items are often the most relevant and most readily acted upon by the organizers and participants. The following section examines several representative topics that were identified during a number of post-exercise evaluations and offers some perspectives on the comments.

Since section debriefs are brainstorm-type indicators, they may not include the context applicable to specific items unless care is taken to document as much supporting information as possible during the ensuing discussion. Therefore caution should be exercised when reviewing these items since, without any specifically described context, they may lead to incorrect assumptions and conclusions. For those areas of improvement that have context and supporting documentation, they may be addressed, and possibly used, as a basis for refining objectives during future exercises.

With regard to lessons learned, feedback received immediately following an exercise during the well accepted “hot wash” process can be helpful, but are generally not able to go into depth because of potential time constraints and may not indicate any specific major gaps that exist with respect to exercise processes. Any significant items identified in the feedback sessions or by evaluators may be addressed or assigned by individual Section Chiefs as appropriate.

As a sampling of the kinds of comments that are received, the following examples represent some of the more common themes identified from debriefs and feedback that could warrant further action. They are not meant to be all inclusive by any stretch of the imagination, since the range of comments can go from how well the printers and internet connections work, e.g., “it was difficult to operate computers due to internet connectivity issues,” to the amount of room provided for each section, to how well Operations and Planning members were able to interact.

“Unified Command should make a concerted effort to provide frequent, timely and effective situation updates to the entire command center. A more formal and more comprehensive situation briefing on day 1 in the command post, during the early transition phase from local support to those coming in from elsewhere, should be conducted to make sure everyone is on the same page.”

Comment: It’s not unusual for responders and command center personnel who are on scene early in a response/exercise to get caught up in the emergency phase, i.e., the Initial Response of the “Planning P” (Figure2), and potentially fail to recognize that not all personnel in the room have the benefit of the incident history and current status. It’s important bring all personnel up to speed as effectively as possible, and inclusive briefing sessions have been found to be effective.

With respect to external stakeholders, Unified Command should consider making themselves available for longer media appearances to provide sufficient information and respond to media and other stakeholders (i.e., self-imposed 30 minute time limit during media events increased the level of frustration among those attending.

Comment: While this may be true, Unified Command has a number of responsibilities that demand their time. Other possible solutions that have been found to work include having representatives from the UC address general questions within a set timeframe and providing for others such as Technical Specialists to remain and address more specific, potentially operational questions.

“The Joint Information Center (JIC) should make all efforts to stay ahead of the curve (i.e., warnings for land impact on potentially sensitive shorelines). Earlier development of strategies to manage expectations and meet commitments (i.e. to give hourly updates by leadership) should be a focus of the JIC.”

Comment 1: The JIC is a critical component of an emergency response since they are often the filter through which stakeholders will likely receive information. It’s not unusual that there will be a number of comments relating to how well the JIC functioned. This is often a thankless task that can be fraught with all sorts of criticism. However when it is done well, and it often is, comments such as the following may result:

• The talking points developed for the press conference were very good and the press conference was well conducted.

Comment 2: The use of social media started well, and was expanded as the exercise continued – to the point where it was quite comprehensive. The JIC did a great job capitalizing on all available tools: tweets, PRs, talking points, website, press conferences, town hall meetings, blogged, reaching out to stakeholders by scheduling working lunches, etc. The goal was to keep stakeholders informed. This was a breakthrough effort that was extremely well planned and provided value.

“The exercise design should test aspects of the response beyond the first 2 days since there is an inherent limitation to only examining the first operational periods of a response. They are usually very focused on the emergency of the situation and don’t allow for a longer view of other focus areas.”

Comment: It is general the case that many exercises are designed to test response organizations during a simulation of the first two days of a response. Focus areas include Tier I response capabilities and the integration of initial responders, e.g., local Strike teams, with larger spill management teams, since these components can be viewed as mission critical during an emergency response. If this is not managed well, it may be expected that the ongoing, developing stages of a spill response could become derailed. Additionally, an important early-stage requirement of a spill response/exercise is the preparation of an “Incident Action Plan” since this defines the key plans for the upcoming response period. If this is not handled well and completed in a timely fashion, responsible party credibility and response sustainability may suffer a significant set-back. While it is probably the case that there is room for improvement in examining the later stages of a spill response, it is difficult to test these in a realistic sense when considering the constraints of time and personnel resource, i.e., exercises are of finite and relatively short duration.

A potential recommendation is to identify specific issues around the longer term aspects of a spill response and develop additional, focused exercise protocols that can be used to test or validate processes and procedures in a tailored exercise format for specific sections and section interfaces (i.e., Operations/Planning; Operations/Logistics, etc.) with specific timelines in mind.

Real world incidents can also certainly be used as a basis for providing inputs to a robust AAR process, especially if exercise personnel have participated in both. In reality, inputs related to or derived from an actual incident can serve as a reality check during the exercise-related debriefs since they can serve to highlight potential best practices, improvement opportunities, or actual experiences with other potential gaps and how they were managed. This also serves to point out the value of identifying other types of information that may come out of an emergency response or exercise. These may include lessons learned, good practices, or other categorizations. A key aspect is how to document what is identified and/or learned so that others may benefit from knowing where to find them, since all too often, lessons are not learned and sub-optimal performance may be repeated. (Chopra, 2015) For example, the US Department of Homeland Security (DHS), and presumably other international governmental organizations, maintains a “Lessons Learned Information Sharing” system in order to disseminate information across the emergency response community. Specific categories to be documented included are shown in Table 2. (DHS, 2007)

Again, drawing from the view of the US Federal Emergency Management Administration (FEMA), “Planning makes it possible to manage the entire life cycle of a potential crisis. Strategic and operational planning establishes priorities, identifies expected levels of performance and capability requirements, provides the standard for assessing capabilities and helps stakeholders learn their roles.”¹ The Evaluate/Improve component of the cycle (Figure 3) is a critical component of the continuous improvement process, a key feature of most safety and risk-related efforts.

As indicated by actual examples of AAR comments and suggestions for areas for improvement, a number of suggestions are actionable, but may not result in substantial enhancements of exercise outcomes, e.g., provide better printer capability. Other potential actions may provide significant improvements but may ultimately prove to be difficult to implement, e.g., expand the time period being tested from one operation period to multiple periods. The purpose of this discussion is to help focus efforts on those actionable suggestions that may result in the most significant enhancement during the preparedness cycle feedback process.

The response community is generally quite interested in providing as much value as possible so it can be reasonably considered that lessons learned and suggestions for improvements will be considered seriously. By continuing the process of AAR and looking for those highest value-added items from a broad suite of actual exercises, the evolution of exercises within the preparedness cycle will continue. Significant resources (time, people, expenses) are committed to large-scale exercises and there is a large effort to ensure that efforts are not sub-optimal. While the regulatory aspect of the evaluation process should not be minimized, since exercises are designed to test specific goals and objectives, additional value can certainly be gained if a robust process for exercise evaluation is established (Table 2). (IOGP-IPIECA, 2014)

Some major themes that were identified during the review of AARs from a variety of exercises included:

• Effective Integration of Response Teams During the Early Stages of a Response

• Effective Communications between a Variety of Stakeholders

• Proactive Engagement with External Stakeholders and Management of Expectations

• Expansion of the Timescale of an Exercise beyond the Emergency Phase

Because of the potential impact that most of these would have if not handled well, it is not surprising that they often rise to the top during hot wash and AAR discussions. In most cases, the prospect of a quick hit, one time fix is not very likely and it is important to involve engaged individuals in the process of working to enhancements and solutions. It is an ongoing process that requires personnel who have a vested and potentially long term interest. This can be a challenge since many exercise participants have other “day jobs” that may make it difficult to commit significant amounts of time to the endeavor, but that does not mean that the attempt should not be made.

This paper has attempted to present an analytical view of recent OSRO-industry exercises with the goal of considering the types of comments and recommendations that result from After Action Reviews (AAR). The purpose was to find some level of thematic commonality of those reported by participants and evaluators with the intention that future exercises can benefit from continuous improvement. It is clear that there is a relatively small number of major themes that should be considered with an eye toward focusing on actionable suggestions. By considering the more significant findings, and their impact on the perceived success of an emergency response exercise by potential stakeholders, the value of the exercise can be extended beyond that of meeting the particular objectives of a specific event. By keeping an open mind and potentially embracing new approaches or the use of emerging technologies, a positive impact on preparedness, response and recovery may result.

It is hoped that by considering actionable recommendations that come out of the AAR process that the spill response community will be able to capture the benefits that result from going through the efforts associated with designing and running spill response exercises. As with many processes, it is the feedback process, and its incorporation into future planning for exercises that helps to enhance the overall value to its users.