Fraser River Sockeye Fisheries and Fisheries Management - Cohen ...

Appendix F Pre-season forecasts: Statistical methodsfor assessing run-size forecast precision, accuracy,and reliability, including definitions of median percenterror (MPE), median absolute percent error (MAPE),regression models, and significance testing.Statistical AnalysisForecast accuracy, precision, and reliability were characterized by five metrics. MPE andMAPE describe the average errors in forecasts across years. These two statistics get ataccuracy and precision, respectively, but can be misleading if interpreted alone. Tofacilitate interpretation we also constructed a best fit regression line between the observedrun plotted against forecasted run (data were log 10 transformed to meet modelassumptions of normality and equal variance). The plot showing perfect forecasts inevery year would reveal all points falling on a 45° line originating from zero. Theregression analysis helps to determine quantitatively (as opposed to visual inspection) thecloseness of a real relationship between two variables (i.e., here we consider forecast andreturn values) to the ideal/theoretical 45°relationship by producing intercept and slopeparameter estimates, as well as, R 2 . Below we describe each of these five metrics (MPE,MAPE, intercept, slope, and R 2 ) and how they are used to judge the reliability, precision,and accuracy of forecasts. We stress that no conclusion about reliability can be can bereached by looking at any one metric in a vacuum; all must be consideredsimultaneously.What is MPE?Median Percent Error (MPE) is a measure of central tendency with respect to thedifferences between observed and predicted values. It describes the percent differencethat divides the data set equally—i.e., half the differences were greater than MPE andhalf were less. MPE is derived by first calculating the difference between forecasted andobserved return values (forecast return - observed return = error or E) for each year;second, converting those values into a percent relative to the observed value ([E /observed return] ×100 = PE); and third, taking the median across years (MPE). If aforecast is accurate (i.e., not biased) then positive errors tend to cancel out negative errorsacross years indicating that errors are for the most part random. In other words, a MPE =F-1