Annual Meeting - SCEC.org

SCEC Research Accomplishments | Report
(ANSS) catalog in California, the Japan Meteorological Agency (JMA) catalog for Japan, and the Global Centroid Moment
Tensor (CMT) catalog in the Western Pacific testing region. The CMT catalog is also used for global testing.
Today, 224 forecasting experiments are
being tested worldwide (Figure 79). Two
regions of current interest are New Zealand,
where 15 models were being tested by
CSEP at the time of the 4 Sept 2010 Darfield
earthquake (M7.1), and Japan, where 91
models were under CSEP testing at the
time of the 11 Mar 2011 Tohoku earthquake
(M9.0). The Darfield and Tohoku
earthquake sequences are being observed
by high-quality seismic networks, and they
may lead to a better understanding about
how such sequences could unfold along
other active zones like those in the western
United States.
The CSEP testing procedures follow strict
“rules of the game” that adhere to the
principle of reproducibility: the testing
Figure 78. CSEP testing centers and testing regions as of June 1, 2011, showing the
region, the authoritative data sources,
number of earthquake forecasting models under test in each region.
including the seismicity catalog, and the
conventions for model evaluation are established before and
maintained throughout an experiment. An experiment can be rerun
at any time by any researcher, and it will produce the same
results. All models submitted to CSEP are required to be properly
documented (preferably in the form of source code for the
executable model), and they can be calibrated using retrospective
data for each region; however, any data used for calibrating the
models retrospectively are not employed in model evaluations.
Figure 79. Number of earthquake forecasting experiments under
test in CSEP testing regions worldwide as a function of time from
the initiation of the SCEC Testing Center in September, 2007,
through November, 2010. Annotation indicate when new testing
regions were initiated.
Examples of CSEP Results
The catalog latency is variable but generally exceeds the updating
intervals of the shortest-term forecasts; therefore, current CSEP
testing is not strictly prospective. However, the model and any
updating methods are fixed; authors cannot modify or interact
with their models after an experiment has begun, and they are
not involved in conducting the statistical tests. Thus, the forecasts
are truly blind. Although the main focus is on this quasiprospective
testing, the reproducibility of CSEP experiments
provides a unique capability for retrospective testing.
Assessing the quality of a method is a multifaceted problem that can involve many attributes of performance (e.g., Jolliffe and
Stephenson, 2003). The attributes of forecast quality emphasized in the current CSEP testing procedures are reliability and skill.
Reliability is an absolute measure of performance; it evaluates the statistical agreement between the forecast probabilities of
target events and the observed frequencies of those events (e.g., the mean observation conditional on a particular forecast).
Skill, on the other hand, assesses the performance of one method relative to another. Measures of skill can be used to evaluate
a candidate method relative to a standardized reference method; e.g., a short-term earthquake forecast relative to a long-term
forecast. To be useful for operational purposes, a method must demonstrate both reliability and skill.
2011 SCEC Annual Meeting | 93