Thematic Accuracy Assessment Procedures. Version 2 - USGS

3.0 Field Methods (Response Design)
3.1 Considerations that Affect the Selection of a Source of Higher Accuracy
Many of the techniques for measuring uncertainty in mapped classes were developed for remote
sensing in order to provide users and producers with ways to assess the thematic accuracy of
remotely sensed land classifications. In this kind of accuracy assessment, it is common practice
to select a sample of observation sites and to compare the class that was assigned to each site by
the mapping effort being evaluated with that obtained using a source of higher accuracy than that
of the map. The source is usually a form of ground truthing that is obtained by human
observation and interpretation in the field.
A successful thematic accuracy assessment of a map requires that data be collected at reference
sites using a response design (Stehman and Czaplewski 1998). The response design is an
assessment methodology that produces reference data that can be assumed to be of significantly
higher accuracy than the map itself, although no assessment methodology can produce infallibly
accurate results. The methodology used in the response design should be independent from the
mapping results that are being assessed. The choice of an appropriate methodology of assessment
depends on three main factors: (1) accuracy of the method, (2) cost of the method, and (3) the
relevance of the method to the user. Sources of higher accuracy other than the most accurate
means possible may be used to increase sampling efficiency if the loss of confidence in the
identification is acceptable (Environmental Systems Research Institute et al. 1994).
During the course of the NPS Vegetation Inventory from 1994 to 2009, five methodologies have
either been considered and/or used. They are:
1. Collection of field data using the same methodology as the vegetation classification (e.g.,
establish a classification plot (or releve) at the site and then return to the office for analysis and
assignment of the site to a vegetation type).
2. Field observation and class assignment in the field by an expert in the vegetation
classification for the area. The expert is usually the ecologist that developed the vegetation
classification or similarly experienced person. Minimal other data are collected.
3. Field observation and class assignment in the field by a qualified and trained, but nonexpert
observer (as in Method 4); the observer also collects supplemental data on species
composition and cover and other parameters that would aid in identification of the type. The field
calls are assessed for matches to the map data. Supplemental data for the mismatches are
evaluated in the office by an expert (as in Method 2) and the field call is modified, as necessary.
The inherent assumption is that the field call is accurate, unless superseded by the office call. In
order to maintain the principle of reference data independence from the sample data, and to avoid
biases in the reference data determinations, experts in the office must make their determinations
from the supplemental data without only out knowing the sample data value of the site that is
being evaluated and without knowing the field observers’ assessment call. Both of these
principles are sometimes violated.
4. Field observation and class assignment by qualified and trained non-expert field observer.
The observer uses the classification materials (usually a field key to vegetation types) to make a
29