Exercise 10UDP 422 Advanced GeoSpatial Analysis

Exercise 10: Accuracy Using Kappa

To review the role of accuracy and precision, and sources of error in the urban landscape

Data needed:

fileexc10.zip from the website

Files include:

  1. exercise10.mxd
  2. Land_cov_sm: Land cover data for a small area of Seattle.
  3. Naip2006_sub_utm10n_nad27.img: Ortho photo for the same small area of Seattle.

Note: All files are projected in UTM NAD27.You will probably need to repair the data source in exercise10.mxd!

In this exercise you will explore and analyze a small subset of a classified Landsat™satellite image by measuring and interpreting error. You will learn how to calculate error using an confusion matrix. This process allows you to calculate how accurately a satellite image has been classified.

Definition of the Kappa Index

In this exercise, you learn how to calculate the Kappa Index, also known as Cohen’s kappa, which is a measure of inter-rater agreement (how well do two[1] people/methods of/approaches to agree on land cover classification?).

This statistic acts as an indicator of the extent to which the percentages of correct values from an confusion matrix or error matrix are due to “true” agreement versus “chance” agreement. As true agreement approaches 1 and “chance” agreement approaches 0, k approaches 1 (the ideal case).

Importantly, kappa differs from (overall) accuracy. Overall accuracy is the ratio ofthe sum of all correctly classified cells to the total (diagonal elements of the confusion matrix divided by total). Kappa incorporates instances where the two observers do not agree(the non-diagonal elements of the confusion matrix) in addition to instances where they agree.

Conceptually, κ can be defined as:

Note that there is disagreement about if or how well ‘chance agreement’ is represented in this model[2]—but that is beyond the scope of this class!

Land Cover Class Description
Value / Class / Definition
1 / Mixed Urban / A combination of urban materials and vegetation
Predominantly low- and mid-density residential
2 / Paved Urban / Surfaces with > 75% impermeable area
Includes high-density development, parking lots, streets, and roof tops
3 / Forest / Surface dominated by trees
4 / Grass, Shrubs, Crops / Agricultural fields, golf courses, lawns, and regrowth after clearcutting
5 / Bare Soil / Land that has been cleared, rocks, and sand
6 / Clearcut / Clearcut forest that has not had significant regrowth; very dry grass
7 / Water / Lakes, reservoirs, and streams

You will use ArcGIS Spatial Analyst to do this exercise. Be sure that the Spatial Analyst extension is checked inToolsExtensions.

A: Creating the confusion matrix/ contingency table:

  1. Open the ArcMap document exercise10.mxd
  2. Note: when you open the ArcMap document you will need to set the data source for each layer to the data sources you unzipped.
  3. There are two data files included here, a subset of a land cover grid with 6 classes and an aerial photograph corresponding to the same area in Seattle, WA[3]. Four sample grids have been drawn onto the land cover using a 3 x 3 cell grid. They are labeled with numbers and correspond to the four classes listed in Table 1: Mixed Urban, Paved Urban, Forest and Water. We will use these sample units to check the classification accuracy of the four classes. Notice that three classes - bare soil, grass, and clearcuts- are being left out of our accuracy assessment because there are no good locations to sample these cover types in this photo.
  4. Starting with the Mixed Urban grid (top of map), zoom into the general area and turn off the land cover layer. Be sure to zoom out far enough to see the photo image clearly without it being too pixilated (about 1:2,000 works well).
  5. Now look at the piece of the aerial photo visible within each of the nine cells, and consider the definition of ‘mixed urban’ (“a combination of urban materials and vegetation”). How many do you think should be classified as mixed urban? If more than 50% of a grid cell matches your assessment of mixed urban, then count that grid cell as truly mixed urban. On the other hand, if more than 50% looks like another land cover type (water, paved, forest), then count that grid cell as one of these other land covers.

Enter your estimated counts for each class under the Mixed Urban column of the matrix below (Table 2). The Sum of the column should equal 9 because we have 9 sub-sample units per class sample.

  1. Do the same for the following 3 classes: Paved, Forest, and Water. Count the pixels by class and record the sums in the Confusion matrix.
    Confusion Matrix:

Predicted Class
Mixed Urban / Paved / Forest / Water / Total (Σ)
Actual Class (from image) / Mixed Urban
Paved
Forest
Water
Total (Σ) / 36

Q1: Fill out the aboveConfusion Matrix.

Q2: Calculate total number of correct pixels. Total number of correct pixels is the sum of elements along the major diagonal (grey cells). From this, calculate the overall accuracy by dividing the total correct pixels(sum of the diagonal) by the total number of pixels which is 36 (4 sample units * 9 sub-samples).

Q3: For one of the cells you classified differently than the computer did, take a screenshot and briefly explain why you chose to classify it the way you did. If you didn’t classify any cells differently than the computer, briefly explain your reasoning.If there are some cells that appear to be grass/crops/shrubs (which are not options in the confusion matrix), how did you classify these? Why?

Q4: The User’s (Consumer’s) Accuracy are errors of commission (including a pixel in the predicted classification when it should be excluded), calculated by dividing the # of pixels correctly identified in a class by the total number predicted to be in that class. Producer’s Accuracy are errors of omission (excluding a pixel that should have been included in the predicted classification), calculated by dividing the # of pixels correctly classified as a class by the number that should be in that class, as determined by your analysis of the image. A great resource is available here:

Calculate both the User’s Accuracy and the Producer’s Accuracy for each class and write these numbers either in the table, or summarize them below. Please present your answer for both the Producer’s and User’s accuracy as a fraction (2/14) and a decimal (e.g. 0.298).

Predicted Class
Mixed Urban / Paved / Forest / Water / Total (Σ) / Producer’s Accuracy
Actual Class (from image) / Mixed Urban
Paved
Forest
Water
Total (Σ) / 36
User’s Accuracy

B: Calculate the Kappa Index:

Table 3:Confusion matrix comparing image referenced class to computer-based classification (Note: this is different data than the confusion matrix you generated earlier)

Predicted Class
Mixed Urban / Paved / Total (Σ)
Actual Class (from image) / Mixed Urban / 68 / 12
Paved / 7 / 112
Total (Σ)

We can calculate κ using this information. This will require you to calculate the expected cell frequencies (expected agreement) and compare it with the observed agreement. The table above is the observed agreement. To calculate the expected agreement, you need to calculate the probability that they would either both classify a cell as ‘mixed urban’ randomly or that they would both classify a cell as ‘paved urban’ randomly. Each of these can (separately) be calculated by:

Where [predicted class] is either Mixed Urban or Paved. Summing these estimates will give the probability of random agreement. This can be generalized to include more than two classes (so you could calculate the kappa for 4+ classes such as for the confusion matrix you generated), but we’re keeping it simple here so you can see the basic idea behind the kappa statistic.

Please see these websites for more help:

Q5: Calculate the overall accuracy (observed agreement) and the Kappa Index for the data in Table 3. Please show your work (e.g. write out how you calculated the probability of random agreement).

Is the total Kappa coefficient above or below the overall accuracy?

Answer questions for highlighted questions 1-5. Submit your assignment March 13, 2014.

- 1 -

[1] For more than two, look at Fleiss kappa, a generalization of Scott’s pi (similar to but not the same as Cohen’s kappa).

[2] E.g.:

[3] The western shore of Greenlake, to be precise.