Fraser River Sockeye Fisheries and Fisheries Management - Cohen ...

season forecasts that eventually correlate well with post-season abundance estimates cannot bereliable if they are highly biased early in the season when fishery openings need to bedetermined. Note that the Initial Statement of Work was vague on this to begin with. (iii)Basing assessments of reliability on four run-timing groups masks a great deal ofunpredictability and risk at the Conservation Unit or indicator stock level. Although suchanalyses may not be possible at the moment, there should be some qualification of reliabilityassessments based on recruitment aggregated across many stocks. (iv) Finally, assessment ofpre-season forecast reliability, in particular, is based on a small sample size and continuallychanging choices for pre-season forecast models. Thus, any assessment of past reliability maynot do well in predicting future reliability, or reliability under very different circumstances…inother words, the reliability metrics may not be reliable.LGL Response: (i) Our objective was to determine whether there was a statisticalrelationship between forecasted values and actual returns. If there was no suchrelationship (i.e., returns varied at random with respect to forecasts) we deemed theforecast to be unreliable. If the relationship was significant, we deemed the forecast toreliably track the general rises and falls of the actual returns, and then took further stepsto describe “how good” the forecast was based on regression and MAPE statistics. This is areasonable approach that has been used in other fisheries publications evaluating forecastperformance (e.g., our analyses were derived Haesaker, Peterman et al. 2008 N Am J FishManagement). We have added text to better describe how we relate regression parametersand MAPE estimates to (1) precision, where increased dispersion around the linearrelationship causes MAPE to increase and R 2 to decrease, and (2) accuracy, whereaccuracy improves when regression slope approaches 1 and intercept approaches 0. Webase our interpretations of forecast precision and accuracy on the following simple facts: aperfect relationship between forecast and return will have a regression slope=1,intercept=0, R^2=1.0, and MAPE= 0. As any of these values depart from optimal, eitherprecision or accuracy erodes. MAPE becomes useful when slopes depart substantiallyfrom 1 (an instance where R^2 would be very low), because MAPE describes dispersionaround the fit line independently of slope. This final point also explains why MAPE and R 2values do not always agree (e.g., Fig 9-10, as noted by the reviewer): MAPE can remainconstant (e.g., 25%), even if R 2 decreases rapidly with decreasing regression slope. Weagree with the reviewer that there are more complex ways to analyze these data, but westand behind our methods as an adequate and easily accessible approach to determinewhether there is indeed a relationship between forecast and return – significantly, weidentified several indicator stocks where no such relationship exists. Now that we haveprovided a relative comparison of each stocks’ forecast performance, future inquiry cansingle out individual stocks-of-interest and employ more rigorous and complex analyticaltechniques (e.g., Bayesian methods) to parameterize the forecasting error (i.e., parse andM-28