Benefits and Limitations of Using Decision-Analytic Tools to Assess Uncertainty and Prioritize Landscape Conservation Cooperative Information Needs

The Landscape Conservation Cooperatives (LCCs) are a network of partnerships throughout North America that are tasked with integrating science and management to support more effective delivery of conservation at a landscape scale (U.S. Fish and Wildlife Service 2012). Originally organized within the U.S. Fish and Wildlife Service, the LCC concept was later adopted and promoted by the U.S. Department of the Interior as a way of enhancing the conservation of resources that cross jurisdictional boundaries (Millard et al. 2012). The LCC concept rests on the notion that to achieve effective conservation, multiple partners must work together to address problems that are too broad scale for existing partnerships to handle. One of the main goals that LCCs strive to achieve is better integration of diverse institutional and stakeholder needs to facilitate more efficient and responsive management of resources at a broad scale (Jacobson and Robertson 2012). To address the challenge of broad-scale conservation, LCCs need to develop a structured and transparent method for developing shared objectives and desired biological outcomes. Such shared objectives can then be used to help the LCCs assess where providing relevant scientific information and facilitating partner communication will be beneficial to its members.

One area of support that many LCCs have focused on recently is providing information to resolve uncertainties associated with management decisions made by their partners. By starting with partner decisions, LCCs will have a better understanding of the institutional hurdles that may limit a more integrated approach, and where commonalities in partner goals exist. The success of this approach, however, rests on at least two assumptions. The first is that effective broad-scale conservation is partially influenced by the state of knowledge about the consequences of different decisions. In other words, decision makers make better decisions when they have scientific information for predicting the potential outcomes of particular actions. The second assumption is that providing better information within a partnership context should lead to better communication and coordination of conservation decisions among the partners.

Although there are many structured frameworks that could be used to discuss uncertainty (e.g., Ayuub 2001) and complex policy and decision-making problems (Sterman 2000), we consider one approach in this paper known as “structured decision making” (SDM; Gregory et al. 2012). Although this approach is not focused on uncertainty per se, SDM (also known as decision analysis) does focus on specific decision contexts in which the components of the decision to be made, including the uncertainties, are disassembled and then reassembled in a structured way (Clemen and Reilly 2001). Such approaches have been used in a range of ecological applications (e.g., Maguire 2004; Gregory and Long 2009) and in determining management-relevant information needs within other LCCs (Woodward et al. 2014).

The focus of our paper is on highlighting some of the benefits and limitations of using SDM, or the decision-analytic approach, with LCCs. We discuss two case studies that involved the Plains and Prairie Potholes LCC (PPP-LCC) and their use of SDM to explore frameworks for choosing which information needs they should spend their research funds on. For each of the case studies we will briefly discuss how the decision problems were framed and how the problems were assessed. We will then follow up with our observations about what worked, where we found limitations, and thoughts on where this approach might be useful for LCC partnerships in the future.

The PPP-LCC spans a portion of the Northern Great Plains of the United States and Canada and is organized around two committees: a Steering Committee and a Technical Committee. The Steering Committee is the primary decision-making entity with regard to setting the strategic direction for the LCC and allocating resources. The Technical Committee is responsible for making recommendations to the Steering Committee about what topical areas to consider and what science should be supported by the partnership. The first case study we present focused on developing a prioritization tool for ranking information needs against LCC-based objectives. The purpose for this ranking tool was to develop a targeted set of science priorities within the LCC partnership. The second case study focused on identifying uncertainties associated with specific management decisions common to LCC partners, and determining which uncertainties were important in improving management outcomes.

For both case studies, we used decision-analytic methods consistent with those used by the U.S. Geological Survey and the U.S. Fish and Wildlife Service (e.g., Williams et al. 2009). The SDM approach and techniques are well documented in Gregory et al. (2012), Clemen and Reilly (2001), and Keeney (1992). In short, the SDM approach starts by framing the decision problem, eliciting measurable objectives (i.e., outcomes of interest and directions of preference), identifying the alternative actions that could be taken to meet those objectives, and often concludes with the use of tools to assess how the alternative actions perform across the objectives. These final assessments take the form of qualitative or quantitative analyses (Phillips 1984; Keeney 2004) and the results of the SDM assessment can be used to inform decision-makers.

The focus of this case study was on helping the PPP-LCC identify the range of missing information that individual partners felt precluded them from making informed management decisions. The intent of the PPP-LCC partnership was to address only the most important information needs with formal research projects. The problem was that information needs likely varied across the partnership in terms of how critical they were to decision making. The PPP-LCC wanted to focus on distributing financial resources to fund research projects that answered only the most pressing and broadly useful management questions. To guide the choice of which information needs deserved formal research projects, we developed a system for prioritizing needs in a way that reflected the values of the PPP-LCC partnership. The top-ranked needs would appear in requests for proposals that could be ranked and funded.

One tool that the Technical Committee considered using to address this question was a value-of-information analysis. Value-of-information analyses can be used to help determine which information is most valuable or influential in choosing among various management alternatives (e.g., Runge et al. 2011). In a formal value-of-information analysis, the performance of the best decision under uncertainty is compared with the performance of the best decision assuming the decision maker had resolved the uncertainty before making the decision. Thus, a value-of-information analysis quantifies the value of having information before making a decision (Skinner 2009). Such an approach typically requires a specific management problem and models linking system uncertainties with predicted management outcomes. However, through an elicitation exercise, Technical Committee members listed 34 information needs (Table S1). We anticipated that it would be too difficult to develop models to assess each of these uncertainties.

Instead, we used a simpler multicriteria decision-analysis approach (Keeney and Raiffa 1975) to rank the needs. We worked with the Technical Committee to develop objectives that all of the partners agreed reflected their priorities. They specified five main objectives, two of which had subobjectives (Table 1). For each of the identified objectives, we developed a constructed scale to quantitatively assess the effect of resolving each information need (Table 1). We then conducted a scoring exercise where the PPP-LCC coordinators scored each information need across the objectives using the constructed scale. Next, we worked through an exercise with the coordinators to weigh each objective using the swing-weighting technique (von Winterfeldt and Edwards 1986). This approach involves establishing a hypothetical alternative that performs poorly on all of the objectives. The person doing the weighting then picks a single objective that he or she would most prefer improving, or “swinging,” to the best value of performance. The participant assigns a score of 100 to that objective. He or she repeats the process for all of the remaining objectives, assigning scores to reflect relative preference for improving performance. Once completed, the scores are then normalized to sum to 1. For the process we followed, we first ranked the subobjectives, which all summed to 1, and then ranked the objectives, which also all summed to 1.

We used these weights to compute the weighted sum of the scores for each information need and used the scores to rank information priorities (i.e., higher score = higher priority). Ideally, this weighting and scoring should have been conducted with the Technical Committee, but time constraints prevented our working with them in this iteration of the ranking tool. However, they were given an opportunity after the analysis was complete to comment on the results. Overall, PPP-LCC coordinators weighed objectives associated with the importance of information in supporting partner decisions and the range of influence of information more heavily than objectives associated with urgency, opportunity, or novelty. After the exercise, the LCC coordinators presented these results to the Steering Committee, who asked that the list of 34 information needs be further summarized into eight broad categories (Table 2) and then rescored. The broader categorization did create some issues. First, as can be seen in Table 2, the difference in scores between the information categories was not large, which suggests that the process did not do a good job of ranking information needs because the objectives were not specific enough to pick up on trade-offs in performance. That is, the needs may have all performed similarly across the objectives. Although the categories of information need did perform similarly on some objectives, there were others on which the categories differed substantially in performance, but often for objectives that were given low weights. This suggests that there was some, albeit a small, trade-off in performance between the objectives. Second, the main limitation of this approach was that the detail of the information needs did not necessarily provide the Technical Committee with an understanding of how improved information could lead to more informed or coordinated decisions.

Despite these issues, the approach we outlined here did have some useful benefits. First, working through the process of developing objectives exposed the Technical Committee to the idea of thinking about ends before options. This was facilitated by the process of listing individual partner decisions, but then thinking about the end objectives those partners were trying to achieve when making those decisions. As the Technical Committee worked through the process of organizing this list of decisions and objectives it became clear that there were areas of overlap between the partners and that the PPP-LCC could develop Technical Committee-level objectives to help integrate across the individual partners. This helped the Technical Committee shift from a strong tendency to think about individual needs to thinking more about shared interests. Although the specific details needed to be generalized for this process, the Technical Committee did agree with the final results of the ranking in principle. The next case study contains much of the detail the Technical Committee felt was lacking from the previous case study, and outlines the process we went through to expose the Technical Committee to a value-of-information approach.

To address the main limitation of the first case study, we again enlisted the help of the Technical Committee to develop an example of how a detailed value-of-information approach could be used to assess the importance of resolving specific uncertainties in a management problem. Much of this case study was motivated by understanding how uncertainties, like those related to land use or management effectiveness, compare with uncertainties about uncontrollable variables, like future climate, in terms of improving decisions. Using the comprehensive list of decisions from the previous case study, we worked with the Technical Committee and identified invasive weed management as a high-priority decision that most LCC partners were engaged in. The Technical Committee was concerned with issues related to the allocation of resources between multiple invasive weed-control programs, especially control on private lands, which account for over 90% of the PPP-LCC landscape. To address these common concerns, we developed a description of the problem surrounding invasive species management, specifically, how to best allocate funding between control programs for multiple invasive weeds and control on public vs. private lands.

To begin, we elicited three fundamental objectives that the Technical Committee thought were important in invasive species management: maximizing native species populations, maximizing public satisfaction with invasive management programs, and minimizing program costs. We should note here that minimizing invasive species was not considered a fundamental objective, because it was assumed that the reason for removing invasive species was to increase or improve habitat for native species. Developing metrics for these objectives proved difficult because of a lack of information about the link between native species and invasive species removal. Likewise, participants had difficulty specifying how public satisfaction could be measured. We decided to measure the native species objective using the proxy attribute of minimizing the number of infested management units. We developed alternative actions for a simplified scenario in which a manager decided how to invest a limited budget between the management of two invasive plant species. The manager allocated resources between the two species, but also decided how much of the budget to allocate directly to managing each species on public lands or to incentive programs that encouraged private landowners to manage the species on private lands.

Given this scenario, we worked with the Technical Committee to build a conceptual model to describe the system. The conceptual model was used to develop an understanding of the major drivers and influences on invasive weed distributions. At first, this model included a wide range of potential uncertainties, but upon reflection, the Technical Committee was able to simplify the model so that it only included uncertainties that the Technical Committee felt were most important. These uncertainties included predicting the impact of land-use policy on agricultural practices, the efficacy of control on privately owned land, the influence of native plant communities, and climate-related variables like soil moisture and likelihood of droughts.

We then developed a quantitative state-and-transition model that allowed us to simulate optimal management decisions given the uncertainties listed above (Text S1). The model described a management scenario where there were 100 management units of 100 ha each, with 90% of the units assumed to be privately owned and 10% assumed to be publically owned. We assumed that there were two invasive weed species that were targets of control: leafy spurge (Euphorbia esula) and yellow toadflax (Linaria vulgaris). The model stochastically simulated invasive species spread and management over a 50-y time horizon. For the output of the model to be useful for a value-of-information analysis, we developed a performance measure to be estimated by the model. In this case, the performance measure was the number of years that at least 50% of the management units were free of invasive weeds. This was a proxy measure for the fundamental objective of maximizing native species populations. To address the other fundamental objective of minimizing costs, we treated cost as a variable constraint in the model.

The dynamics of the model described the management units as transitioning from an uninfested state to an infested state, or vice versa. We assumed that each unit could potentially be infested by either or both species. We further assumed that uninfested units in each year became infested as a result of dispersal from currently infested sites and that treatment in the form of spraying could send the infested unit back to the uninfested state. Developing this part of the model exposed additional uncertainties about how quickly species spread, and about the effectiveness of management practices. It also required being more specific about what information would be needed to simulate the infestation and control process. We consulted the literature and developed probability distributions to simulate the spread and establishment processes for each species. In some cases, where information was not readily available, we expressed parameter uncertainty as a uniform random variable over a wide interval. On the basis of model development, it became clear that some variables were extremely uncertain compared with others because of a lack of information (Table S2). The simulation model and analysis was based on the work presented in Moore and Runge (2012).

We used the state-and-transition model to assess the optimal investment decision (Figure S1) and to perform a value-of-information analysis to determine which of the uncertain parameters had the greatest influence on the performance of the management alternatives (Felli and Hazen 1998). We used two measures of this influence: expected value of perfect information (EVPI) and expected value of partial information (EVPXI). The EVPI is the amount of improvement in management performance when information about a system is known before making a decision compared with a decision made under uncertainty about the system. To estimate this improvement in performance, we computed the performance of the optimal allocation strategy assuming that all uncertainty in the model was resolved. This involved selecting the optimal strategy for each parameter combination and computing the average performance of the selected strategies (i.e., a decision made under certainty). Next, we computed the performance of the optimal strategy assuming uncertainties were not resolved. This required computing an average performance for each strategy and then selecting the optimal strategy on the basis of its average performance (i.e., a decision made under uncertainty). The difference between the performance of the optimal strategy under certainty and the optimal strategy under uncertainty was the EVPI. Larger values of EVPI indicate more expected improvement in performance. We computed EVPI over a range of possible budgets using the parameter values for 10,000 iterations of our simulation.

The results of our simulations showed that when information was known before making a decision, expected management performance improved (Figure 1). Recall that the modeled performance measure was the number of years that at least 50% of the management units were free of invasive weeds. The largest EVPI was 5.64 y, or about a 22% improvement in performance, which corresponded to a budget of $80,000. However, as budgets increased or decreased from that point, the difference in performance also decreased. This happened because small budgets limited the amount of management being done, so resolving uncertainty did little to improve already poor performance. Large budgets were used to invest in more control options, which meant the decisions could be less strategic, and uncertainty played a smaller role in determining performance.

The question that remained was, which uncertainties were contributing the most to the improvement in management performance? To answer this question, we conducted a sensitivity analysis and used EVPXI assuming a budget of $80,000. For the sensitivity analysis, we assessed the strength of association between values of each model parameter and performance for each iteration. We used a correlation coefficient (R) to measure this relationship and assumed that parameters with correlation values of at least 0.05 were important. Although these parameters might be important in the calculation of the performance measure, they are not necessarily important to decision making. Thus, we computed EVPXI for each of the important parameters, which measured the change in management performance when only one of the parameters was known and the other parameters were uncertain. Again, larger values for EVPXI indicate more value in resolving uncertainty. Table 3 gives the model parameters identified as important in the sensitivity analysis and the EVPXI analysis. The results of both the sensitivity and EVPXI analyses showed that although some parameters were important in predicting performance, others were more important in changing which decisions were optimal. In this case study, parameters associated with the effectiveness of management on private land and how species were likely to spread and establish were the most important uncertainties with respect to altering decision making.

This last conclusion, that not all uncertainties were equally important in terms of management decision making, was one of most important insights that LCC partners gained from this case study. Early on in the conceptual model development, the Technical Committee began to realize that although these kinds of decision problems may be complex and contain many uncertainties, there were often fewer important uncertainties that actually posed impediments to better management. Similarly, when trying to quantify aspects of the conceptual model, we realized that more specificity was often needed when trying to express relationships between uncertain variables, which further honed which aspects of uncertainty would be most useful in predicting management outcomes.

The PPP-LCC partnership has now worked through two structured decision-making processes and developed preliminary frameworks and familiarity with tools to identify, structure, and analyze management problems. Both were successful in beginning to identify the broad universe of decision types and major information needs of the partners, and in developing techniques for looking at more specific information and decision contexts across the partnership. The process also encouraged thinking that was less technically focused and more decision focused. This is an important point because the participants in these processes began with a common perspective among technical experts: they are knowledgeable about the major scientific uncertainties surrounding a topic, but they have less of a sense about whether resolving those uncertainties would matter to decision makers or influence the choices they make.

In the first case study, the results of the analysis, although highly generalized, did lead to greater transparency among the Technical Committee members about their broad-level information needs and about how to work together to develop shared objectives. In the second case study, the analysis taught the Technical Committee about a slightly different way to think about information and uncertainty. The second case study had a further benefit in that working through a more focused problem led to greater collaboration on projects between partners in the PPP-LCC. For instance, as a result of this collaborative thinking, the current focus of the PPP-LCC is almost exclusively on trying to understand the factors that drive changes in decisions about land use and how these changes affect how conservation is delivered on the landscape.

We should also point out that both case studies were focused on ranking, or prioritizing, information needs, but that ranking itself is not a decision. Rather, it is a decision-support tool for guiding choices about where to invest resources. For the first case study, the decision was choosing which information needs to develop into a request for proposals. The final decision was made on the basis of a simple decision rule to pick the top five ranked information needs. However, when proposals arrived, the objectives used to rank information needs were also used to rank the proposals. After ranking, the Technical Committee worked down the list of projects from highest ranked to lowest, and picked those that could be afforded. This was an informal process; a formal allocation model could have been developed to assess slightly more complex allocation decisions, such as developing a portfolio of projects across all of the information needs. However, at this stage of the process, the Technical Committee felt that the ranking tool was enough to help them in making these choices. We should note that an almost identical process of information needs selection was adopted using the results from the second case study.

The results from both case studies also pointed to an important trade-off between inclusivity and specificity in identifying information needs. For information to be most useful it needed to be specific enough that decision makers could understand how to use that information to predict or assess the effects of their decisions. Conversely, the PPP-LCC partnership was interested in expressing information needs that were general enough that a diverse set of land managers and conservation decision makers could agree on their importance. Although this was easily accomplished in case study 1, the generality of the results left the Technical Committee with little insight about how additional information might actually affect decision making. However, as case study 2 showed, even with a more specific focus, it was challenging to find a decision problem that remained useful and relevant to all partners while also being able to model the effect of uncertainty on real decisions. As a result, the analysis in the second case study focused on a general set of management alternatives in a specific management context.

An additional challenge inherent in the second case study was the mismatch between the expertise of the Technical Committee and the topic of the problem. Identifying critical uncertainties associated with natural resource and conservation management decisions requires focused thought and input from both technical and management experts. The members of the Technical Committee are all natural resource scientists or managers, but many do not operate at the tactical level. Although this was not immediately clear when we began the process outlined in the second case study, the Technical Committee later discovered that they were well equipped to identify priority management contexts (i.e., identify invasive species as a priority management issue), but did not feel comfortable acting as proxy decision makers for those tasked with actually controlling invasive species. The LCCs are often viewed as the link between managers and the science they need to make better decisions. As our results point out, engaging managers directly about how they operate in the field can only improve the process of narrowing down which information is needed to help them make better decisions.

Despite this, the processes we used in this paper did provide us with insights about management. Our first analysis culminated in a fairly general list of critical information needs and an assessment of those needs based on the opinion of experts, namely the Technical Committee. Our second analysis pointed out that information perceived as important in making a decision did not always lead to better management performance. This does not mean that such information is not valuable in management problems. Rather, it suggests that given a particular problem formulation, only some variables have influence over making optimal decisions. Realizing why not all information is influential in improving decision making is important for the LCCs to understand as they begin to support various decisions.

Up to now, the PPP-LCC primarily developed a broader-scale view of conservation problems by focusing on assessing the commonalities in what was limiting decisions made by its partners. Our work suggests that focusing on these types of decisions will continue to pose a considerable challenge, especially for LCCs with diverse partnerships. With that in mind, LCCs could consider focusing on the decisions over which they have more direct control. Developing broad-scale goals that represent larger cumulative outcomes may be more fruitful than focusing on what limits conservation delivery on a smaller scale. This may require setting aside specific partner objectives, decisions, and critical information needs to develop larger-scale objectives. Such a framework could then be used to coordinate partner actions by assessing how different partner decisions might contribute to the larger-scale objectives of the partnership. Assembling managers to articulate decisions could then be used as a basis for developing more coordinated strategies. This would not only provide a template for coherently integrating conservation delivery, but would also serve to engage a broader audience of stakeholders.

Please note: The Journal of Fish and Wildlife Management is not responsible for the content or functionality of any supplemental material. Queries should be directed to the corresponding author for the article.

Text S1. Description of modeling and analytical procedures for assessing the value of information associated with making investments in invasive weed control programs. The model and analysis were done as part of an uncertainty prioritization exercise with the Plains and Prairie Potholes Landscape Conservation Cooperative in 2012 in Bismarck, North Dakota.

Found at DOI: http://dx.doi.org/10.3996/032015-JFWM-023.S1 (28 KB DOCX).

Table S2. Uncertainties (or information needs) elicited from the Plains and Prairie Potholes Landscape Conservation Cooperative (LCC) at a structured decision-making workshop in 2011 in Bismarck, North Dakota. The workshop was focused on prioritizing uncertainties associated with LCC partner management decisions. Each uncertainty is listed along with the decisions that the uncertainty is thought to influence and the objectives that those decisions are thought to affect.

Found at DOI: http://dx.doi.org/10.3996/032015-JFWM-023.S2 (24 KB DOCX).

Table S3. Parameters used in a dynamic model describing the hypothetical spread of leafy spurge (LS; Euphorbia esula) and yellow toadflax (YT; Linaria vulgaris) across a collection of management units (P = privately owned units, C = publically owned units). Note that some parameters are fixed values, some are randomly drawn from statistical distributions (U = uniform, N = normal), and others are dependent on both. The model was developed as part of an uncertainty prioritization exercise with the Plains and Prairie Potholes Landscape Conservation Cooperative in 2012 in Bismarck, North Dakota.

Found at DOI: http://dx.doi.org/10.3996/032015-JFWM-023.S3 (16 KB DOCX).

Figure S1. Results from an optimization exercise performed with the Plains and Prairie Potholes Landscape Conservation Cooperative (LCC) in 2012 in Bismarck, North Dakota. These results were based on a dynamic model describing weed species invasion dynamics. The vertical bars represent the optimal allocation to different hypothetical invasive species control programs as a proportion of total annual budget (horizontal axis). The program could consist of investments in either publically owned management units (C) or privately owned units (P). Likewise, the program could be focused on controlling yellow toadflax (YT; Linaria vulgaris), leafy spurge (LS; Euphorbia esula), or both.

Found at DOI: http://dx.doi.org/10.3996/032015-JFWM-023.S4 (22 KB DOCX).

Reference S1. U.S. Fish Wildlife Service. 2012. Strategic Habitat Conservation, Landscape Conservation Cooperatives. U.S. Fish and Wildlife Service Fact Sheet.

Found at DOI: http://dx.doi.org/10.3996/032015-JFWM-023.S5 (532 KB PDF).

Reference S2. Williams BK Szaro RC, Shapiro CD. 2009. Adaptive Management: The U.S. Department of the Interior Technical Guide. Washington, D.C.: Adaptive Management Working Group, U.S. Department of the Interior.

Found at DOI: http://dx.doi.org/10.3996/032015-JFWM-023.S6 (38.2 MB PDF).

Reference S3. Woodward A, Liedtke T, Jenni K. 2014. Identifying resource manager information needs for the North Pacific Landscape Conservation Cooperative. U.S. Geological Survey Open-File Report 2014-1032.

Found at DOI: http://dx.doi.org/10.3996/032015-JFWM-023.S7 (1.1 MB PDF).

We thank the members of the Plains and Prairie Potholes Landscape Conservation Cooperative who participated in the workshops and provided feedback about the process for both workshops. We also thank Karen Jenni of Insight Decisions, LLC and Timothy Nieman of Decision Applications, Inc. for help in facilitating the workshop associated with the first case study, and the U.S. Geological Survey North Central Climate Science Center for providing support for the workshop used as the basis for our second case study. We also thank Andrea Ray with the National Oceanic and Atmospheric Administration Climate Prediction and Projection Pilot Platform for assistance with the model developed in case study 2. Finally, we thank D. Smith, M. Runge, and two anonymous reviewers for helpful comments on the manuscript.

Any use of trade, firm, or product names is for descriptive purposes only and does not imply endorsement by the U.S. Government.