Lessons Learned and Corrective Actions from BSEE Initiated Unannounced Source Control Exercises In 2018–2019

[Code of Federal Regulations 30 (CFR) 254]

[Homeland Security Exercise and Evaluation Program (HSEEP)]

Exercises are the primary tool for assessing the preparedness of the BSEE regulated community and for acknowledging their achievements as well as identifying any areas for potential improvement. Source control GIUEs aim to help operators objectively assess their capabilities and identify any gaps, deficiencies, and vulnerabilities to be resolved prior to experiencing a real oil spill event. However, while an individual operator selected for a source control GIUE often benefits from the lessons learned during the exercise hot wash, the resulting strategies for improving preparedness are not always externally distributed for consideration and implementation by other operators. The intent of this report is to share lessons learned from BSEE's source control GIUEs for the purpose of helping to improve the existing capabilities of regulators, operators, incident management teams (IMTs), and other responders. All stakeholders are encouraged to review this document and incorporate the identified lessons learned into their future trainings and exercises.

BSEE - OSPD conducted 14 GIUEs in fiscal year 2018 (October 1, 2017 – September 30, 2018) and 15 GIUEs in fiscal year 2019 (October 1, 2018 – September 30, 2019) ranging from generic functional exercises to complex multi-day full-scale exercises with multiple equipment deployments. Of the 14 total GIUEs conducted by OSPD in 2018, 6 were functional exercises and 8 were full-scale exercises involving the deployment of response equipment. Of the 15 total GIUEs conducted by OSPD in 2019, 11 were functional exercises and 4 were full-scale exercises involving the deployment of response equipment.

Seven of the exercises conducted by OSPD during the 2018 and 2019 fiscal years involved complex subsea drilling and production source control scenarios. In previous years, BSEE focused on an operator's ability to successfully perform response actions necessary to cap an uncontrolled blowout of a subsea well. During the 2018 and 2019 fiscal years, OSPD coordinated with other offices in BSEE including District Operations Support, Technical Assessment, Well Analysis, Pipelines, District Offices, and the Houston Energy Technology Center to develop more complex but plausible source control scenarios. The BSEE exercise planning teams focused on other aspects or different types of source control operations including surface blowouts on floating production platforms, relief well plans, and subsea well intervention activities. Increasing the scope and scale of source control GIUEs in this way resulted in significant lessons learned as identified and documented by BSEE and the operators who participated in these exercises.

OSPD is maintaining this coordinated effort to design and develop new and challenging source control scenarios for the purpose of ensuring maximal preparedness. The GIUE planning teams design these exercises based on operators' previous exercise performance, incident history, the potential impact of a discharge posed by their operations, and other relevant factors.

Lessons learned were summarized and compiled for each of the 7 source control GIUEs OSPD conducted in the 2018 and 2019 fiscal years. OSPD used the following 6 categories adapted from the HSEEP guidance to classify lessons learned: Communications, Leadership and Management Processes, Organizational Structures, Plans or Procedures, Resources, and Training.

• ➢ Graphics, storyboards, equipment diagrams, and timelines should be printed and posted in a prominent location where IMT personnel and members of the Unified Command (UC) will be able to readily access and understand the incident's Common Operating Picture (COP). Additionally, COP software should be utilized if available and displayed on various monitors around the incident command post (ICP).

• ➢Operators that utilize an online Incident Action Plan (IAP) and COP need to ensure that all responders have full access to the electronic system in order to provide appropriate situational awareness for the entire response organization. Lack of access to electronic systems due to licensing issues can severely hamper the initial response. UC Members without access to the online IAP and COP must acquire information through direct engagement with other IMT members or will otherwise be unable to obtain critical information.

• ➢Identify a single liaison who will be responsible for contacting the facility person in charge (PIC) for information.

• ➢ Ensure that the Operations Section Chief provides updates from the Source Control Branch during all relevant ICS planning meetings or invite the Source Control Branch Director to periodically brief the UC.

• ➢ Across numerous source control exercises, IMTs, and GIUE Planning Teams, the term “Worst Case Discharge” (WCD) was utilized inconsistently. During several source control GIUEs, the term “WCD” was used within the Source Control Branch to communicate flow rates for the total well fluids discharging at the source. This interpretation of the “WCD” term caused confusion when it was referenced out of that context. This is because the term “WCD” also refers to the maximum daily discharge of oil identified in an OSRP according to 30 CFR 254.47. The WCD rate described in an OSRP is often significantly different than the discharge rate calculated during the flow assurance processes. The WCD rate calculated by flow assurance methodologies often yields a more challenging scenario for many source control activities because it accounts for the total well fluids discharging from the well and not just the oil. The WCD value describing the estimated volume of oil being discharged at the source may be appropriate for surface water response planning but should not be utilized for source control operations. It is recommended that a separate term be defined to represent the discharge rate calculated during the flow assurance process. Having two distinctive terms will reduce the possibility for an IMT to act on the incorrect WCD value which could result in an escalation of consequences.

• ➢ Fully staff the Source Control Branch early in the GIUE unless the Exercise Lead specifies that source control operations will not be included in the evaluation. If the operator is unclear of BSEE's expectations for source control objectives during a GIUE, they should immediately seek clarification from the Exercise Lead rather than make assumptions.

• ➢ Consider inviting a PIC for the type of facility identified in the scenario to participate in the source control GIUE. The PIC could indicate what decisions would be made by personnel on the offshore facility in response to the given scenario and any subsequent exercise injects.

• ➢ If BSEE provides a representative to serve in the Source Control Support Coordinator (SCSC) role, consider assigning a person trained in source control operations to serve as a dedicated assistant for the SCSC if one has not already been provided by the agency.

• ➢ Assigning the Source Control Branch mission group leaders (e.g., Flow Assurance Group Leader, Capping Group Leader, etc.) with clearly defined objectives in accordance with any existing response tactics manuals (RTMs) will better coordinate the work of the groups into a more cohesive unit. Traditionally, these group leaders are conduits for communication and for driving tasks.

• ➢ If Source control personnel are not experienced with the Incident Command System (ICS), operators should consider assigning an ICS coach to the Source Control Branch to help ensure necessary information is transmitted up the chain of command during an incident or exercise.

• ➢ Complete the necessary ICS paperwork in real-time during the exercise.

• ➢ Recommend IMTs develop initial critical information requirements (CIR) that can be used in the emergency phase of an incident to ensure appropriate information reaches the UC before the incident specific CIRs are developed during the initial ICS Objectives meeting.

• ➢ Develop a list of requirements for a subsea trajectory model to ensure all appropriate information is provided to the environmental unit when the model is initially requested.

• ➢ When information from flow assurance is used to calculate spill volumes, the spill volumes should also continue to be calculated from the slick descriptions.

• ➢ Depending on the facility involved in the source control GIUE scenario, operators should be familiar with any fully developed emergency disconnect plans that may be in place. For example, if a drillship or MODU moves offsite in the wrong direction during an emergency and drags a disconnected riser, such an action could significantly impact some or all of the subsea infrastructure which might exist below the facility.

• ➢ All simulations and modeling work should include assumptions and data required for replication by a third party. Providing detailed technical work not only allows for the replication of the simulation but also provides clarity and transparency on the simulation's limitations. This may reduce the likelihood of using the information inappropriately.

• ➢ Adding important information in the comments section of the 213s and including the price of resources could expedite the approval process. Relief well procedures should include a process for validating that proposed plans are executable. Considerations should include but are not limited to:

  • ○ Potential effects from the anticipated formation and intersect angle that many will have on detection distance for the proposed ranging tools (e.g., active ranging hindrances in salt formations).

  • ○ Uncertainties in the calculated wellbore position at the ranging point and if these uncertainties are within the acceptable detection distance limits of the proposed ranging tools.

  • ○ Analysis of offset wells to determine the impact (if any) to ranging activities and the potential collision threats

• ➢ Well kill simulations should be based on the most likely rates provided by the flow assurance calculations during the planning phase. Multiple relief wells may be required depending on the intersect point and flow assurance calculations. In the event that multiple relief wells would be required, additional risk analyses should be performed.

• ➢ Contracted personnel and equipment resources can be severely strained by multiple responses occurring simultaneously. Consider developing contingency plans that include the activation of reserve personnel and equipment for managing multiple responses simultaneously.

• ➢ When BSEE issues assignments associated with a source control GIUE for completion by a specified date, necessary exercise participants should be retained in their ICS roles until the assignments have been completed and submitted. If a source control GIUE assignment is not completely understood, seek clarification from the exercise lead regarding BSEE's expectations.

• ➢ Ensure all IMT personnel have access to ICS 214s so that all required notes and decisions can be captured according to ICS procedures.

• ➢ Increase the IMT's awareness of their internal subject matter experts who are available to help them answer a variety of technical questions.

• ➢ Where existing Response Tactics Manuals (RTMs) are available and describe many of the potential risks to be considered, these resources may be used and customized as necessary to improve efficiency rather than having to generate these risk analyses from scratch. The RTM's source control guidance should include a clear description of each of the required missions, a list of mission objectives, the required information and their sources necessary for achieving the objective, and a list of deliverables with a method for verifying the deliverables were received. Providing this interface information will improve communications and increase efficiency across the Source Control Branch.

• ➢ A source control GIUE scenario involving a simulated power outage resulted in a realization that the ROV onboard the identified facility would not be operational. Therefore, the subsurface BOP could not be manually manipulated via the onboard ROV. This demonstrated a need for independent back-up power for an onboard ROV to manipulate the BOP when power is not available.

• ➢ Provide training and regular refresher courses on the use of the IAP software that will be available at their ICP.

• ➢ Where response tactics manuals (RTMs) have been developed for the Source Control Branch, ensure they are regularly updated and that the players are trained in the use of those resources.

In addition to the lessons learned identified by OSPD, preparedness analysts also required corrective actions from various operators. Corrective actions are not designed to be punitive. Instead, they may be assigned to an operator independent of how BSEE rates the GIUE. The purpose for corrective actions is to provide operators an opportunity to resolve identified shortcomings and gaps identified during the GIUE to ensure they are adequately prepared for future responses. For the benefit of all the stakeholders, BSEE is sharing a summary of the corrective actions which were assigned during the 2018–2019 fiscal years.

• ➢ Update the OSRP to provide an accurate Qualified Individual (QI) list.

• ➢ Revise the OSRP to reflect the actual incident management team (IMT) structure, clearly define what roles will be filled by the operator's personnel and contracted IMT personnel, and state under what conditions the contracted IMT personnel will be used.

• ➢ Ensure all IMT personnel are appropriately trained.

• ➢ Develop and carry out an ICS and a Response Tactics Manual (RTM) training plan for all Source Control Branch personnel.

• ➢ Provide a framework and process for evaluating the risks when multiple source control response options are available.

• ➢ Describe a method for integrating parallel source control responses into a an overall response plan (e.g., estimated timelines with critical path items and established thresholds for prohibiting activities deemed “high” risk.)

• ➢ Evaluate if the single ram capping stack is appropriate for use on high flow rate discharges through the use of engineering tools such as computational fluid dynamics (CFDs) models.

• ➢ Develop a standard operating procedure (SOP) within the Source Control Branch which includes an order of operations for the collection and transfer of information during an incident or exercise that allows for a more efficient response.

• ➢ Provide a minimum reporting standard for communicating information related to nodal and crossflow analysis (under shut-in and flowing conditions) that clearly identifies critical information/assumptions that will be used for subsequent decision making. Data should include but is not limited to: rates, gradients, fluid types, and crossflow. The reporting standard should also include considerations and behavior for 30, 60, 90, etc. days.

• ➢ Provide a minimum reporting standard for relief well and kill operations which emphasizes the importance of including sufficient details to allow a third party to duplicate the work. The reporting standard should also clearly indicate the boundary conditions that illustrate its limitations.

Hurricane Harvey caused significant flooding across the Houston area where a majority of operators and their IMT's are based. Hurricanes Irma and Maria devastated Puerto Rico and the Caribbean. Altogether, these storms put incredible strain on response resources in the Gulf of Mexico. The USCG and several other federal and state agencies dedicated significant resources to these responses. Contract IMTs were also heavily involved and preoccupied with response efforts during this time. As a result, OSPD paused its GIUE program for three months until the end of October in 2018.

When the OSPD GIUE program resumed in November of 2018, many response resources and contract IMTs were still being utilized for the hurricane responses which resulted in observed impacts to their performance during GIUEs. Reduced availability of response personnel limited the ability of IMTs to function efficiently and hampered communications. While BSEE recognizes and applauds the efforts of the response community in managing multiple hurricane responses, implementation of the previously identified and categorized lessons learned might serve to expand the preparedness capabilities of operators.

BSEE acknowledges where several observed strategies and tactics deployed by various operators contributed to their exceptional performance during GIUEs. As such, all operators should be encouraged to implement these recommended strategies and tactics in their response procedures.

• ➢ The safety of all GIUE participants remained at the forefront of response activities, both simulated and real-time.

• ➢ The incorporation of an Engineering Services Group was advantageous to the Source Control Branch. [Elaborate]

• ➢ There has been an excellent utilization of OSRO and other contractor representatives at the ICP.

• ➢ The Command and General Staff roles have been occupied by very experienced and capable people.

• ➢ GIUE players perform in their roles as though they were responding to an actual incident.

• ➢ The IMTs typically consider most of the risks involved with their response decisions.

• ➢ Great leadership has been displayed by most Section Chiefs who demonstrated proficiency in ICS.

• ➢ Assigning a courier (Runner) in the Logistics Section to distribute and monitor ICS 213s was a successful strategy for improving efficiency.

• ➢ Most IMTs utilized and referenced their existing response plans and procedures to quickly incorporate anticipated short and long-term actions for the exercise. Preparedness of this type and quality allowed those IMTs to proactively plan for and address routine issues before being prompted by exercise Controllers and their injects.

• ➢ Internal and external notifications were made in a timely manner and in accordance with the OSRP.

• ➢ ICS forms were used throughout the duration of most source control exercises to document operational and support aspects of the response. Their efficient use of these ICS forms helped to facilitate critical decision making by the IMT as well as the creation of the IAP.

BSEE - OSPD identified and collected internal lessons learned for the purpose of improving the GIUE program. Like most other government agencies tasked with conducting exercises, the number of BSEE personnel available to participate in the planning and execution of GIUEs is limited. Most BSEE employees participating in GIUEs are responsible for several other mission critical activities. BSEE personnel must often play in multiple critical roles during an unannounced exercise. An OSPD preparedness analyst can expect to perform the roles of Controller, Evaluator, and Player concurrently during a GIUE. Obviously, these personnel limitations present significant challenges.

In 2018, BSEE made significant adjustments to the GIUE program which included: focusing on more complex source control scenarios, larger GIUE planning teams, increased use of BSEE's subject matter experts (SMEs), multiple-day exercises, and the introduction of a rudimentary Simulation Cell. Improving internal training and defining high-level goals for source control activities has transformed the GIUE program in such a way that its planning and evaluation methodology now provides for the assessment of exercise performance against strategically identified benchmarks while also permitting greater flexibility. This methodology has proven to be more comprehensive than the previous approach which was limited to rigid evaluations of soft shut-in procedures. Updated performance goals have precipitated realistic complexity in OSPD's source control GIUEs and have provided operators with more plausible and challenging scenarios that offer fair opportunities for them to demonstrate their capability of responding.

As previously indicated, OSPD does not operate its GIUE program in a vacuum but leverages expertise throughout BSEE and from other state and federal agencies. The effort to coordinate and collaborate in that regard had enabled OSPD to expand the GIUE program's capabilities. At the same time, managing larger exercise teams has produced new challenges for BSEE as demonstrated by the internal lesson learned from the 2018 and 2019 fiscal years.

• ➢ It is often prudent and necessary for OSPD to invite and include source control GIUE participants from other BSEE offices and various external government agencies in the exercise planning meetings because of their stakeholder interests and subject matter expertise. However, not everyone who receives an invitation to the planning meetings is always able to fully participate in those meetings or the GIUE itself. Human nature is such that someone who receives an invitation to attend the initial planning meeting but is subsequently unable to fully participate in the exercise could become complacent and unintentionally or accidentally disclose details about the pending GIUE to the selected OSRP holder. Therefore, the Exercise Lead should not include any details about the anticipated date of the exercise or the identity of the selected OSRP holder in the invitation to the planning meetings in order to reduce the possibility of unintentional leaks. Furthermore, the anticipated date of the GIUE and the identity of the selected OSRP holder should only be initially provided to confirmed members of the Exercise Planning Team during the initial planning meeting.

• ➢ During the planning and execution of a source control GIUE, the Exercise Lead role is typically performed by the OSPD preparedness analyst who was assigned the exercise. Meanwhile, the Evaluator and Controller roles may be performed by other OSPD preparedness analysts, subject matter experts from other BSEE offices, or representatives from external stakeholder agencies. Even though the OSPD preparedness analyst serving in the Exercise Lead role has no official supervisory authority over the various representatives serving in the Evaluator and Controller roles, the Evaluators and Controllers should recognize and abide by the Exercise Lead's executive decisions regarding the planning and execution of the GIUE.

• ➢ Representatives from BSEE and other stakeholder agencies participating as Players in source control GIUEs should follow the chain-of-command procedures established for their roles in accordance with the ICS standards.

• ➢ Consider providing the BSEE representative serving in the SCSC role with at least one trained assistant or deputy SCSC. Otherwise, the SCSC may have difficulty collecting critical information from the Source Control Branch while also having to attend multiple ICS meetings throughout the day at the ICP.

• ➢ During several large source control GIUEs, operators initially assumed the exercise Controllers and Evaluators were also participating as players. This assumption caused delays in exercise play and potentially impacted the operator's ability to perform the exercise. BSEE has since purchased appropriately labeled ICS vests and provides exercise players with a list of the designated Controllers and Evaluators.

• ➢ When source control activities for a GIUE are taking place at a separate location from the ICP, consideration should be given to having Controllers and Evaluators available at both locations.

• ➢ When developing the source control GIUE scenario, the exercise planning team should consider the discharge rate and composition of the total fluids that could potentially flow from the well instead of relying completely upon the estimated daily WCD rate for oil identified in the OSRP or the Application for Permit to Drill (APD).

• ➢ All exercise planning team participants should have successfully completed the ICS 100, 200, 700, and 800 trainings as well as the FEMA IS 120 and 130 trainings. Repeat participants should also have completed the ICS 300 and 400 trainings. Any stakeholders actively playing a role in the exercise should also be appropriately trained to fill that role.

• ➢ The Exercise Lead should make every effort to ensure the Evaluator, Controller, and Player roles are not combined to avoid confusion during the exercise.

• ➢ Evaluators and Controllers should be assigned unique tasks in order to avoid any unnecessary duplication of efforts or the potential for conveying contradictory messages to the operator during the exercise.

• ➢ Develop individual guides for Controllers, Evaluators, and Players that clearly describes their unique roles, responsibilities, and expectations.

• ➢ Ensure the Evaluator guides include tables that identify the anticipated objectives, strategies, and tactics the players could potentially perform during the GIUE. A section for the Evaluator's notes should be provided below each of those identified benchmarks. Include blank tables in the guides where unanticipated benchmarks and relevant notes from the Evaluator can be added when applicable. Additionally, the Evaluator guides should include a description of the rating system that will be used for each documented benchmark.

• ➢ Ensure the Controller guides include tables that list benchmarks and exercise injects in the order they are anticipated to occur during the GIUE along with the relevant “truth” information players might need or request at those moments. A section for the Controller's notes should be provided below each of those benchmarks identified in the tables. Include blank tables in the guides where unanticipated benchmarks and relevant notes from the Controllers can be added when applicable.

• ➢ Print inject forms on colored paper which can be visually tracked more easily as it is processed within the ICP. Inject forms should have spaces for the IMT personnel who responded to each particular issue to print their names and assigned ICS roles. Additionally, inject forms should capture the date and time an inject was delivered and the date and time the IMT personnel satisfied the objective of the inject. Where possible, deliver specific subject matter inject forms to the designated IMT personnel who would normally receive and respond to the unique content of each inject.

• ➢ Individual training sessions for Controllers, Evaluators, and BSEE Players could better ensure the established roles and responsibilities detailed in the Controller/Evaluator Guides were properly understood prior to the exercise.

• ➢ The Exercise Planning Team should inquire with the appropriate subject matter experts for their assistance with the development of the exercise scenario and with the evaluation of the operator's performance during the GIUE.

• ➢ The exercise scenario memo, injects, and the Controller/Evaluator guides should be reviewed by all members of the exercise planning team at least 24 hours in advance of the GIUE.

• ➢ All Controllers, Evaluators, and Players should adhere to their roles throughout the exercise unless specifically instructed otherwise by the Exercise Lead.

• ➢ The Exercise Lead should minimize the number of duplicative Evaluators of an individual IMT Section. When there are a large number of Evaluators participating in the GIUE, a Lead Evaluator should be designated to assign subordinate Evaluators to the various IMT Sections or within an specific IMT Section.

• ➢ Exercise planning teams should consider using a Simulation Cell in large exercises or be prepared to provide “truth” in real-time during smaller exercises. Ensure the Simulation Cell is staffed with a sufficient number of Controllers.

• ➢ Controllers need to regularly update the Evaluators when deviations from the GIUE scenario occur.

• ➢ The Exercise Lead should review the ground rules with the QI or IC to minimize confusion and potential impacts to operations.

• ➢ Use the operator's incident reporting procedures and any notification forms provided in their OSRP when initiating an exercise.

• ➢ In response to a source control GIUE, an IMT's decision to simulate the evacuation of non-essential personnel from a MODU when the initial blowout occurred rather than fully evacuating the MODU was a delay that appeared to be inconsistent with the Offshore Emergency Evacuation Plan submitted with the APD. Within the simulated scenario, ensure that all necessary decisions are made to safeguard all personnel and consider previously developed emergency action and evacuation plans as necessary. However, OSPD is exercising the operator's OSRP and not the MODU's emergency response/evacuation plan. So, any decision involving the evacuation of the MODU would ultimately reside with the PIC and not the IMT. Since there are obvious benefits to exercising these aspects of a blowout scenario, OSPD should consider how this issue relates to the 30 CFR 254 regulatory framework.

• ➢ There are major hurdles and artificialities involved with a complex source control GIUE including decisions made by offshore personnel, situation status, and communication which are all simulated through the Simulation Cell. It should be acknowledged by BSEE that the expectation of IMT personnel to make decisions that would normally be made by offshore personnel, especially during the initial phases of a blowout, is a major artificiality. BSEE should explore the regulatory ramifications for perceived failures in this decision making activity.

The FY 2018–2019 GIUE years revealed significant lessons learned as a consequence of external influences and internal program adjustments. BSEE's distribution of these valuable lessons learned will hopefully serve to improve the preparedness of everyone who participates in oil spill response activities. OSPD is grateful for all of the BSEE, USCG, NOAA, USFWS, LOSCO, LDEQ, TGLO, TX RRC, PHMSA, and other stakeholders who participated in the planning and execution of these source control exercises. We also acknowledge the professionalism demonstrated by the BSEE regulated community and their spill response contractors for making sure safety remains a top priority within the offshore oil and gas industry.