Fraser River Sockeye Fisheries and Fisheries Management - Cohen ...

Page 60: The authors make extensive use of two statistics throughout the document to assessaccuracy and precision, namely the Mean Absolute Percent Error (MAPE) and the Mean PercentError (MPE). MAPE is the mean of the series of absolute errors (AE) x 100, but the authors optto use the median instead of the mean reduce the influence of aberrant figures (fair enough).Each AE of MAPE is defined in Appendix F (p. F-2) as = |forecast-actual| / forecast. However,the equation commonly used for AE is different and = |actual-forecast| /actual(http://en.wikipedia.org/wiki/Mean_absolute_percentage_error). If so, the authors have itbackwards by using the forecasted value as the denominator instead of the actual value. Thissmall distinction is important. Let actual = 10 and forecast = 12. The AE as defined by theauthors is |12-10|/12 = 2/12 = 0.167. The correct definition yields |10-12|/10 = 2/10 = 0.200. Theauthors need to address this issue, and provide corrections, clarifications or both, because anerroneous use of absolute error values changes many trends and conclusions in their report.As for the Mean Percent Error (MPE), the authors use a median instead of the mean for the samereasons. The error term used = (forecast-actual) / actual (p. F-1), which is the correct term to use(see http://en.wikipedia.org/wiki/Mean_percentage_error). However, the authors seeminglyinterpret the trends backwards. The Fig. 11 caption for instance (on p. 69), it is noted thatpositive values indicate that forecasts under-estimate the actual abundance, but based on theabove equation, positive values are obtained when the forecast exceeds the actual value. If so, thecaptions need to be corrected.LGL Response: The reviewer advises us to make percent error relative to the actual returninstead of the forecasted value based on convention. The use of forecast or actual return asthe denominator in MPE and MAPE calculations is arbitrary (i.e., when forecast is thedenominator, results express forecasting error relative to forecasted values; when actualreturn is the denominator, results express forecasting error relative to actual returns). Weused forecast as the denominator and our results were described accordingly. However,the reviewer rightly pointed out an inconsistent reporting of the methods in an Appendix(i.e., text was wrong, calculations were right). To avoid ambiguity and potential conflictwith the norm, we have sided with the reviewer and now present errors relative to actualreturns. We have also edited the text accordingly: Instead of saying “the percent error inforecasted returns is X% higher or lower than forecasted values”, we now say “percenterror in forecasted returns is X% higher or lower than actual returns”. Although changesto the MAPE and MPE calculations have changed the scale of the errors (i.e., % errorvalues are inflated when forecasts over-estimated returns and deflated when forecastsunderestimated returns), which caused some changes in ranked error among indicatorstocks and run-timing groups, the resulting changes in MPE and MAPE values did notchange the overall trends in accuracy or precision over time nor have we had to modify theprincipal conclusions of our report.M-18