(BRAVO) Study: Final Report. - Desert Research Institute

Final Report — September 2004
In order to interpret the statistical results that were derived, it is useful to provide
benchmarks that reflect the state of the art for meteorological model performance.
ENVIRON Corp. has analyzed results from over 30 modeling studies to formulate typical
standards for meteorological skill (Emery et al., 2001), which are summarized in Table 9-1.
Because of limited data aloft, benchmarks have been developed only for the surface layer,
but they will also be used here as initial indicators of performance aloft.
Table 9-1. Ad hoc benchmarks of surface-layer meteorological model accuracy for air-quality
applications (from Emery et al., 2001).
Benchmarks
Temperature (C) Mixing Ratio (g/kg) Wind Speed (m/s) Wind Dir. (deg.)
MAE< 2.0 MAE< 2.0 RMSE < 2.0 MAE< 30.0
|Bias| < 0.5 |Bias| < 1.0 |Bias| < 0.5 |Bias| < 10.0
The corresponding BRAVO model performance statistics for MM5 were calculated
over the 36-km, 12-km and 4-km domains for each 5 1/2-day segment and were compiled in
tables by segment, intensive periods, month and for the full four-month BRAVO study
period (Seaman and Stauffer, 2003). That evaluation focused on predictions in the surface
layer, but also addressed wind predictions aloft. Since the final modeling performed in the
BRAVO Study used the 36-km meteorological fields exclusively, our discussion of the
evaluation results will focus on the 36-km fields and will compare them with the 12-km
fields. For consistency, results for both grid scales are averaged over the area of the 12-km
modeling domain, which is where the additional information for evaluating model
performance was most readily available. The average 36-km grid performance over the area
of the larger 36-km modeling domain is not addressed here, but may be different from that
over the 12-km domain.
Table 9-2 summarizes the performance of the MM5 predictions of the meteorological
fields at the surface and for winds aloft. These conclusions apply for both the 12-km and 36-
km grid scales over the geographic region encompassed by the 12-km modeling domain
shown in Figure 8-1; the overall performance with the 36-km grid differed when evaluated
over the entire 36-km domain. Values in red exceed the ad hoc performance benchmarks in
Table 9-1. Note, though, that benchmarks are provided there for only two performance
metrics for each variable.
Review of the information in Table 9-2 leads to the following major conclusions
concerning the performance of the MM5 model over the area of the 12-km modeling domain:
1. In most cases, the 12-km fields have the lowest errors for wind and they contain
sufficient detail to capture the regional flow. The 12-km biases in the wind
speeds and directions for all layers and in all model run segments are small and
fall within the ad hoc benchmark values for accuracy. The small values of MAE
for wind direction and RMSE for wind speed in the nominal planetary boundary
layer (30-1500 m AGL) and the lower free troposphere (1500-5000 m) are well
9-3