Lab 7: Accurate Against What?
Accuracy assessment, area estimation, and the cost of a self-test
Background
In Lab 6 we built a five-class land-cover classifier over the Salish Sea bioregion. The Random Forest map looked plausible and the visual comparison against the Salish Sea Atlas put it in roughly the right neighborhood. But “looks plausible” is not a defensible answer to the question every map invites: how accurate is it? In Lab 7 we take that map and ask the question quantitatively. The answer turns out to depend less on the map and more on the reference we choose to compare it against — and on the design of the sample we draw from that reference.
The discipline has a name for that dependency. In conservation biology there is a concept called a shifting baseline: every generation unconsciously accepts the environment they grew up with as “normal,” even if it has already been dramatically degraded. A 5 % increase in fish population sounds encouraging until you learn populations declined 70 % over the last few decades, and even that may be tiny compared to what existed 200 years ago. The number is meaningless without a reference, and the reference itself carries history and uncertainty. The same logic governs accuracy assessment in remote sensing. When we say a classified map is “accurate,” we mean accurate relative to something: a held-out training subset, a stratified probability sample, a published continental-scale product like NALCMS, expert visual interpretation. Each reference is a different baseline. Each gives a different number. Lab 7 makes that fact concrete by computing the same accuracy two different ways on the same map and watching them disagree.
The lab is built around a deliberate contrast. We compute two error matrices over the same Lab 6 Random Forest classification. The first uses the 30 % random holdout we exported from L6.2 — convenient, easy, and what most published student projects and many production reports actually do. We will frame this number as internal validation: a self-test against a random subset of the same labeled pool that trained the classifier. The second uses a fresh, independent sample we draw stratified by mapped class directly over the classified map and label point-by-point against high-resolution imagery. We compute Olofsson et al. (2014)’s proportional-area error matrix and place 95 % confidence intervals on the resulting overall, user’s, and producer’s accuracies. The two numbers will not agree. The gap between them is the central thing this lab teaches.
A small detour into kappa, the metric Cardille Ch. F7’s errorMatrix API returns by default. Kappa attempts to correct overall accuracy for the agreement we would expect by chance. The remote sensing community has broadly deprecated it — Campbell §13.4 summarizes the critique — because it is redundant with overall accuracy, tends to overestimate chance agreement, and underestimates the probability that any given pixel is correctly classified. We compute kappa in this lab because we will encounter it in published literature, but we treat it as a historical artifact and close the lab reporting overall, user’s, and producer’s accuracy with confidence intervals instead.
Part 1 — From Classifier to Accuracy
The Lab 6 Random Forest map exists as a colored raster. To go from a colored raster to a number — the map is X percent accurate — we need three things: a reference dataset (a set of locations whose true class we know by some method we trust more than the classifier), a spatial unit at which map and reference are compared (here, the 30 m Landsat pixel), and a rule for agreement (here, identity: the mapped class equals the reference class). The compact representation of all that comparison is the error matrix: a 5 × 5 table whose rows are mapped classes, whose columns are reference classes, and whose cell (i, j) is the count of points the map called class i and the reference called class j. The diagonal cells are agreements; the off-diagonal cells are confusions.
From the error matrix, three accuracies fall out (Campbell §13.4; Cardille F7 §7.2). Overall accuracy is the proportion of all points the map got right — the sum of the diagonal divided by the total. User’s accuracy for class i is the proportion of points the map called class i that the reference confirmed — the diagonal cell divided by the row total — and answers the question “if the map says Forest, how often is it actually Forest?” (the perspective of the map’s user). Producer’s accuracy for class j is the proportion of points the reference called class j that the map also called class j — the diagonal cell divided by the column total — and answers the question “of the actual Forest in the world, how much did the map find?” (the perspective of the map’s producer). Omission errors and commission errors are simply the failure-side rephrasings of the same numbers: omission for class j is 1 - PA[j], commission for class i is 1 - UA[i]. Three numbers, three perspectives — a single overall accuracy is never enough.
The L6.2 holdout is the simplest possible reference. We split the labeled training pool 70 / 30 before training, kept the classifier away from the held-out 30 %, and reserved the holdout exactly for the moment we are in now: classify those points with the trained Random Forest, build the error matrix, read the accuracies. Script L7.1 does exactly that, and it produces a number — typically in the high 80s — that we will print and inspect. But before we celebrate, we should be honest about what the number means. The holdout was drawn from the same labeled pool that trained the classifier. It inherits any class-balance, geographic, or spectral biases that pool carried. It does not constitute an independent probability sample over the mapped surface. It tests how well the classifier reproduces the labels of points it never saw, which is a different question than how well the map represents land cover in the world. Olofsson et al. (2014) §1.1 are explicit on the point: a defensible accuracy assessment requires a probability sample whose units are independent of the data used to train the classifier. Random subsetting of the training pool does not satisfy that requirement. The number is useful — it is what most reports actually do, and it is the natural baseline — but it is a self-test, and we will treat it as such.
Part 2 — Designing a Defensible Reference Sample
A defensible accuracy assessment starts not with a classifier but with a sample design. Olofsson et al. (2014) §2 walk through what that means: the reference units must be drawn by a probability sample — every unit in the bioregion has a known, positive chance of being selected — and the sampling plan must be documented so that someone reading the accuracy report can evaluate it. The choice that matters most for an undergraduate land-cover map is what to stratify by. Stratification means dividing the bioregion into non-overlapping subpopulations and drawing reference units from each one separately. The natural stratification when the goal is per-class accuracy is the mapped classes themselves — we draw a fixed number of reference points from each mapped class and label them. This guarantees that even rare classes (here, Snow/Ice on Mt. Baker, Mt. Olympus, and Glacier Peak) receive enough reference points to compute a meaningful per-class accuracy.
The alternative — simple random sampling without stratification — is what happens when we draw 75 points from across the bioregion uniformly and accept whatever class distribution we get. For a class that occupies roughly 1 % of the mapped surface, simple random gives us about one reference point in expectation. One point is not a sample; it is an anecdote. The user’s accuracy for Snow/Ice computed from one point is undefined or has a confidence interval so wide as to be useless. Olofsson §2.1.1 is unambiguous: simple random fails for rare classes. We will see this directly in EX1, when we re-run the assessment under simple random and watch Snow/Ice drop out of the matrix.
For the stratified sample we use 15 points per mapped class, 75 points total. The choice of 75 has two justifications worth stating. The first is operational: hand-labeling each point against high-resolution imagery takes roughly 20–40 seconds, putting the total labeling burden at about 30 minutes inside a single class period and leaving time for the analysis in Part 4. The second is design parity: Lab 6’s L6.2 holdout produces approximately 75 points (50 per class × 5 classes × 30 % split), so matching the stratified sample to 75 keeps the L7.1-versus-L7.3 comparison clean — only the sample design and the reference independence vary, not the number of reference units. Equal allocation (15 per class) is also a pedagogical choice. Olofsson §2.1.2 recommends a compromise allocation — boost rare classes above proportional, then allocate the remainder roughly proportional to class area — which gives smaller standard errors on overall accuracy at the cost of larger ones on rare-class user’s accuracy. We use equal allocation for clarity here; the compromise allocation is part of the EX1 discussion.
Worth saying out loud: the Olofsson worked example in §5 uses n = 641. Production-grade accuracy reports routinely use 500–1000. Our n = 75 will produce visibly wide confidence intervals. That is by design — wide intervals make the cost of a small reference sample visible rather than hiding it behind a single point estimate.
Part 3 — Manual Reference Interpretation
This part of the lab is a hands-on procedural workflow that has not appeared in any prior ESCI 442 lab, so we walk it through one step at a time. The workflow turns the 75-point exported CSV from EX1 into a labeled reference set by filling in the reference column (integer 0–4) for each point, where the integer is the true land-cover class as we judge it from high-resolution imagery.
Setting up the imagery in ArcGIS Pro
The exported CSV contains point_id, longitude, latitude, mapped_class, class_code, and a reference column initialized to −9 (a sentinel meaning “not yet labeled”). Import it with Map tab → Add Data → XY Point Data; set X field = longitude, Y field = latitude, coordinate system = GCS WGS 1984 (WKID 4326).
The Lab 6 classification was built from Landsat 8 imagery composited between 29 June 2020 and 15 September 2020. To compare each reference point against imagery from roughly that same growing season, apply the World Imagery time filter: right-click the basemap layer → Properties → Time tab → check “Show the layer when the map is within a fixed time extent” → set the Visible Time Extent to 6/29/2020 – 9/15/2020 → click OK.
If you do not have access to ArcGIS Pro, Google Earth Pro (free) provides a historical imagery slider — click the clock icon in the toolbar and drag the slider to summer 2020 — which gives you equivalent date-matched high-resolution imagery without requiring an ArcGIS license.
Labeling each point
For each of the 75 rows, navigate to the point using the longitude and latitude values, inspect the high-resolution imagery in the immediate 30 m Landsat-pixel neighborhood, and decide what class it actually is. The decision rule is the same as for Lab 6 training-point collection: assign the class that dominates the 30 m neighborhood around the marker. For ambiguous points — mixed-pixel boundaries, forest-farmland edges in the Cascades foothills — make the best judgment call and assign the more dominant class. Replace the −9 sentinel in the reference column with the appropriate integer (0 = Forest, 1 = Developed, 2 = Water, 3 = Herbaceous, 4 = Snow/Ice). Olofsson et al. (2014) §3.5 recommend recording an interpreter-confidence flag for each point; we do not implement that here, but worth keeping in mind: every reference label is itself an act of interpretation.
The figures below show examples of the labeling judgment for three commonly confused class pairs.
Editing the asset properties
To add the ref_class property: in the asset table, click the gear / edit icon next to any property column header to open the property editor. Add a new property named exactly ref_class with type integer. For each of the 75 rows, type the integer 0–4 corresponding to the class you decided in the previous step. Save the asset. The Asset Manager automatically commits the edit; the labeled asset is now ready for L7.3 to read.
Re-uploading to the labeled-asset path (alternative)
If editing 75 rows in the GEE table view is unwieldy, download the CSV export from your Drive folder and label it outside GEE. The exported file contains point_id, longitude, latitude, mapped_class, class_code, and a blank reference column — you only need to fill in that last column (integer 0–4 for each row). Two good options:
Spreadsheet (Google Sheets / Excel) — open the CSV, use the longitude and latitude values to find each point on Google Maps or Google Earth Pro, and type the reference class code directly in the
referencecolumn.ArcGIS Pro — go to Map tab → Add Data → XY Point Data. Set X field =
longitude, Y field =latitude, coordinate system = GCS WGS 1984 (WKID 4326). The 75 points appear as a feature layer over your basemap. Set the World Imagery time filter as described above (Properties → Time → 6/29/2020 – 9/15/2020), then edit thereferencefield in the attribute table for each point. When done, export the layer to CSV (Right-click layer → Data → Export Table).
Once labeled, re-upload the CSV to GEE via Asset Manager → New → Table Upload and name it lab7_strat_sample_labeled (or your chosen path). The result is identical to the in-browser edit: a FeatureCollection with both classification (mapped class) and reference (your interpreted class) on each feature.
Part 4 — From Counts to Area, with Confidence
The proportional-area error matrix is what makes Olofsson’s framework actually buy us something beyond Cardille’s accuracy() call. The conversion is one line of arithmetic: p̂[i][j] = W[i] × n[i][j] / n[i]·, where W[i] is the mapped-area proportion of class i and n[i]· is the row total. The interpretation is more interesting than the formula. Each p̂[i][j] is the estimated proportion of total bioregion area in mapped class i and reference class j. The whole matrix sums to 1.0. The diagonal sum is the overall accuracy. The row marginals are the mapped class proportions (the W[i] we started with). The column marginals are the reference-based proportions of each class in the bioregion — and those reference-based proportions are different from the mapped proportions whenever the classifier has commission or omission errors. That difference is what enables corrected area estimation.
Olofsson Eqs. 5–7 give variance estimators for OA, UA, and PA under the stratified-by-map-class design. The variance equations are mechanical applications of survey-sampling theory; the formulas in their full algebraic form are in Campbell §13.8 and in the lab’s master script for those who want to read them. What matters for interpretation is the result. Each accuracy estimate now comes with a 95 % confidence interval — OA = X ± Y — which tells us how much sampling variability is in our point estimate. At n = 75 and a reasonably accurate classifier, the OA half-width is typically in the 4–8 percentage-point range. That is wide. It is wide because the sample is small, and it is wide for a reason: an accuracy report that says “OA = 87 %” without a CI is hiding the fact that the true value could be anywhere from 80 % to 94 %. The CI is what makes the number scientifically usable.
Area estimation is the second-order payoff. If we counted pixels of the mapped Forest class and multiplied by the pixel area, we would get a mapped area — but mapped area is biased by the classifier’s commission and omission errors. The reference-based estimator (Olofsson Eq. 9) uses the column marginals of the proportional-area matrix to give an unbiased estimate of each class’s actual area in the bioregion, with its own confidence interval (Eq. 10). For a class with high commission error, the reference-based area estimate will be smaller than the mapped area; for a class with high omission error, it will be larger. The L7.4 script reports both numbers side by side.
The final move in Part 4 is the McNemar test. Lab 6 trained both CART and Random Forest classifiers on the same training data. The Lab 6 lab document asked us to compare them visually. Lab 7 asks the same question quantitatively: are the two classifiers’ error patterns statistically distinguishable on the same reference set? Campbell §13.9 gives the McNemar statistic — z = (b − c) / √(b + c) where b and c are counts of points one classifier got right and the other got wrong — and a 95 % significance threshold of |z| > 1.96. The L7.4 script prints b, c, the z-statistic, and the significance flag. The result tells us whether the visual difference between CART and RF is real (statistically significant on this reference) or whether it is within sampling noise.
Part 5 — When Sampling Design Fails
The independent exercise in Lab 7 re-runs the same accuracy assessment under simple random sampling — 75 points drawn uniformly across the bioregion with no stratification by mapped class. Olofsson §2.1 predicts what will happen: the points fall in classes roughly in proportion to mapped area, which means Snow/Ice (~ 1 % of the bioregion) receives roughly one reference point, and Developed (~ 2–4 %) receives roughly two to three. The user’s and producer’s accuracies for those classes are either undefined or have confidence intervals so wide as to be useless. The simple-random error matrix tells us, in concrete terms, what stratification was actually doing.
The pedagogy here is not that simple random is bad in some abstract way. It is that sample design choices have consequences and those consequences are not theoretical — they show up as missing rows in the error matrix. The full discussion of why Olofsson recommends a compromise allocation rather than equal allocation connects EX1 directly to the design choice for your own final-project accuracy assessment.
Key Estimators
The equations below are the computational core of the Olofsson (2014) framework. Each one maps directly to a line in the L7.3 / L7.4 console output, and each has a counterpart in Campbell §13.8.
Use the table below to record your own count-form error matrix from the L7.3 console output. Rows are mapped class; columns are reference class; diagonal cells are agreements.
| Forest | Developed | Water | Herbaceous | Snow/Ice | Row n | UA | |
|---|---|---|---|---|---|---|---|
| Forest | |||||||
| Developed | |||||||
| Water | |||||||
| Herbaceous | |||||||
| Snow/Ice | |||||||
| Col n | |||||||
| PA |
Proportional-area element (Olofsson Eq. 4). Each cell is scaled by the mapped-area weight \(W_i\) of its row class, so the whole matrix sums to 1.0:
\[\hat{p}_{ij} = W_i \cdot \frac{n_{ij}}{n_{i\cdot}}\]
where \(n_{ij}\) is the sample count in row \(i\), column \(j\), and \(n_{i\cdot}\) is the row total.
Overall accuracy — the diagonal sum of the proportional-area matrix:
\[\widehat{OA} = \sum_{i=1}^{q} \hat{p}_{ii}\]
User’s accuracy for mapped class \(i\) — the proportion of reference points in that row confirmed by the reference (equal to the raw sample proportion because \(W_i\) cancels):
\[\widehat{UA}_i = \frac{n_{ii}}{n_{i\cdot}}\]
Producer’s accuracy for reference class \(j\) — the proportion of all reference points the map correctly found for that class:
\[\widehat{PA}_j = \frac{\hat{p}_{jj}}{\displaystyle\sum_{k=1}^{q} \hat{p}_{kj}}\]
Variance of OA (Olofsson Eq. 5) — used to place a 95 % CI on the overall accuracy estimate:
\[\hat{V}(\widehat{OA}) = \sum_{i=1}^{q} W_i^2 \cdot \frac{\widehat{UA}_i\,(1-\widehat{UA}_i)}{n_i - 1}\]
Corrected area of class \(j\) (Olofsson Eq. 9) — reference-based, unbiased estimate of the true areal extent of each class:
\[\hat{A}_j = A_{\text{total}} \sum_{i=1}^{q} W_i \, \frac{n_{ij}}{n_{i\cdot}}\]
95 % confidence interval half-width for any estimator \(\hat{\theta}\) with variance \(\hat{V}(\hat{\theta})\):
\[\text{CI}_{95\%} = 1.96\,\sqrt{\hat{V}(\hat{\theta})}\]
McNemar’s z-statistic (Campbell §13.9) — tests whether two classifiers differ significantly on the same reference set, where \(b\) counts points Classifier A got right and B got wrong, and \(c\) counts the reverse:
\[z = \frac{b - c}{\sqrt{b + c}}, \qquad |z| > 1.96 \;\text{is significant at}\; \alpha = 0.05\]
Guiding Questions
ESCI 442/542 — Introduction to Remote Sensing | Western Washington University