FY 2003 Request for Studies proposal 200305800

Section 1. Administrative

Project location

Reasonable and Prudent Alternatives (RPAs)

Section 2. Past accomplishments

Section 3. Relationships to other projects

Section 4. Budget for Planning and Design phase

Task-based budget

Outyear objectives-based budget

Outyear budgets for Planning and Design phase

Section 5. Budget for Construction and Implementation phase

Task-based budget

Outyear objectives-based budget

Outyear budgets for Construction and Implementation phase

Section 6. Budget for Operations and Maintenance phase

Task-based budget

Outyear objectives-based budget

Outyear budgets for Operations and Maintenance phase

Section 7. Budget for Monitoring and Evaluation phase

Task-based budget

Outyear objectives-based budget

Outyear budgets for Monitoring and Evaluation phase

Section 8. Estimated budget summary

Itemized budget

Total estimated budget

Cost sharing

Reviews and recommendations

This information was not provided on the original proposals, but was generated during the review process.

Comment:

We can recommend proposal #10 for development of an evaluation model based on expert opinion as a stopgap measure in the short term. In the long run, only collection of better monitoring data as proposed in some of the other proposals to satisfy RPA Actions #184 and #182, and as will be identified in the course of development of an expert system, will allow detection of the effects at the population and ESU levels of a myriad of hatchery reforms in terms of their effects on extinction risk and/or recovery. Overall, the proposal is weak in directly addressing needs described in the RFS.

Direct answers to RFS questions.

Would the study result in the development of a standardized analytical approach for synthesizing the results and detecting the effects at the population and ESU levels of a myriad of hatchery reforms in terms of their effects on extinction risk and/or recovery?

The proposed approach is standardized but derived data will be mainly qualitative (subjectively determined by expert opinion) and in this sense, not strictly quantitative and analytical. The ability to apply the approach to a given hatchery and hence compare the different hatchery reforms in terms of effects on extinction risk and/or recovery at the population or ESU levels will vary with the data available. Fewer data means more reliance on expert opinion.

Will the study provide documentation and/or explanatory text for the analytical approach sufficient to allow other entities to readily use it to evaluate potential effectiveness of hatchery reform measures?

Yes and no. Others will be able to use the method, but use will depend on the availability of data and experts. It is likely that the work would provide documentation and explanatory text sufficient to support use by others, as the main players in the proposed effort have a track record of making information and approaches available to others. However, the outcome will be an expert system subject to continuous restructuring as new empirical information comes available or it would simply freeze today’s admittedly inadequate information as to outcomes of new approaches (reform). This would seem to make the project continuing, rather than a nine-month project with a clear useful end-product. The funding agency should carefully consider whether or not they wish to start what may turn out to be a 10-12 year project.

Primary Review Comments and Questions for Improvement of the Proposal

The authors recognize that, too few data are available to provide case-specific assessments. They propose an expert system approach that can use whatever data exist from ongoing processes, such as HGMPs, the APRE databases, and EDT analyses, to qualitatively assess risks and benefits. We agree that an expert system such as proposed is probably the only way to provide qualitative risk assessments of hatchery practices. The alternative is to wait 10-12 years for the completion of rigorous studies on the reproductive success of hatchery and wild fish to obtain more quantitative assessments.

The proposed approach is founded on the principles of probabilistic risk assessment and attempts to estimate risk as a function of the subjective probability that different events might occur and their possible consequences. The principal investigators propose to use “expert opinion” to derive lists of hazards, weighting factors for hazards, variables contributing to risk, and distributions of potential consequences. However, the proposal continues to use the term “risk” in other ways. To further confuse the terminology, it uses “hazards” interchangeably with “risks”. It refers to “risk-benefit” analysis without explicitly addressing benefits. These issues and definitions should be addressed and made clear before the project is funded.

It is an act of faith to assume that a consensus model based on expert opinion will in fact be a good model. The proposal should include methods for evaluating the quality of the resulting consensus model before funding. An itemized budget should be provided.

Comment:

Comment:

Comment:

The response provides reasoned answers to each question and good justification for the use of expert opinion as a short-term step in the risk analysis, although it is weak in describing the process of generating expert opinion in a way that would ensure consistency and usefulness. The proposed effort is useful as a stopgap measure until data are available, but the generation of “best guess” information from sources that have many potential biases is a far cry from the collection of actual data.

The response acknowledges that expert opinion is a second best alternative to having good data, but argues that in the absence of data, expert opinion is useful. The ISRP agrees that a model based on expert opinion will be useful. Further, a secondary benefit will be further identification of significant data gaps that limit the ability to meet the full intent of FCRPS BIOP Action #184: Synthesis of Existing Analytical Approaches, or Development of a New Analytical Approach, for Determining the Effects of Hatchery Reforms on Extinction Risk and Recovery.

The use of expert opinion as an approach to generating synthesized results is a matter of process, but the response does not provide specifics about that process. The authors assert that variability in expert opinion can be used as a proxy for parameter variability. This seems a tenuous assumption when considering the factors that contribute to variation in expert opinion: range of knowledge, context-specific information, vested interest, and selection bias. The response recognizes these sources of variation but says only that the solution is to guard against them where possible. How would this be done? The response does not thoroughly describe a process by which a group will develop probabilities of certain hazardous events, or quantify the magnitude of the impact (cost) of those events. The response refers to methods available to assess uncertainty in responses but does not elaborate on them. Additionally, the response does not indicate how, specifically, the use of expert opinion would be transparent.

Reviewers were surprised that the sponsors did not have suggestions on how to further test their proposed risk consensus model. The model could be used in a sensitivity analysis of a set of predictions, these could be the basis for new research or to find studies (of which there are several in this package) that could be used in these tests. If the consensus model is a short-term process, then the authors should be able to suggest means to develop the longer-term procedures. The recommendation to evaluate the quality of the consensus model is discounted as impractical.

Regarding the response to question 3 (continuous up-dating of information), the question posed by the ISRP and their response are both reasonable, everyone is acknowledging that we currently do not have adequate data for more quantitative models, so this is another process and we need to ensure that funding agencies and decision makers understand the dynamic nature of this work over the next 10+ years. Their adoption of “broader view of risk” as supported by the NRC (1996) report is good; this is a very readable and informative report. However, the response downplays the likelihood that new information will be different enough to change the basic conclusions of the initial expert opinion exercise. The ISRP continues to believe that usefulness of an expert opinion model will require periodic updates as new data are generated. Intuition of experts tends to change immediately upon learning new facts.

A point for clarification, Ken Currens is identified as a Co-PI but is not listed in the budget.

Comment: