Chapter 9.2 - Image Segmentation¶

Image segmentation describes approaches used in the identification of image regions or image objects. This process can be used to simplify or focus the analysis of images that often have very high-dimensionality. For example, CONUS-spanning, high-resolution geoscience datasets like GridRad and Level II Radar Data, MRMS, and GFS can have hundreds of thousands or millions of “dimensions” in the form of pixels/grid points. While having the rich (i.e., tons of information) data available is a benefit for society, we often will use data mining techniques to reduce the dimensionality and make it easier to process. The vast majority of modern AI/ML tools perform either and explicit or implicit simplification step. For example, even though convolutional neural networks and large language models appear to ingest raw images and/or text, the [“black box”] portion of these models applies complex transformations to the data so that it is fit for ingesting into the model. Using the ingestion analogy, this is similar to how you can eat an apple and your body is able to break it down into its most important parts and get rid of the rest (luckily, this is not a biology class, so I do not have to go any further!).

Chapter 9.2.1 - Image segmentation workflow¶

Basic image segmentation is largely based on empirical rules created by domain experts (i.e., expert systems). Segmentation algorithms take advantage two features in gridded Geoscience data: 1) grids have well-defined uniform spacing; and 2) grids have a well-defined and physically-derived intensity. With these two attributes, you can identify areas, shapes, distances, and other distinguishing features of regions and/or objects that might be found in gridded Geoscience datasets. For simple tasks, you might organize pixels into two categories:

1. foreground / positive: these pixels / grids belong to an object of interest.

2. background / negative: these pixels / grids do not belong to an object of interest.

A typical workflow looks something like this:

1. Identify a gridded dataset you would like to segment

2. Study the metadata, caveats, and assumptions

   • Projection: this might make distances, shapes, and areas more difficult to translate to “pixel space”

   • Intensity: if the intensity is reported in mm and you assume in, you could experience a catastrophic error in your analysis.

   • Is the study period sufficient? Is the study area sufficient?

3. Create a high-level view of what it is you are trying to identify in the gridded data (either based on testing, literature, or both).

   • What are you actually trying to identify?

   • What are the features / attributes of this object?

   • How can you tell the difference between this object and the “background”

4. Translate the high-level view into “grid space”.

   • What intensity value defines the edges of a region?

   • Does an object need a minimum area, length, maximum intensity, etc.?

Chapter 9.2.2 - Thresholding¶

As we explored in a previous chapter, image thresholding is the process of identifying pixels / grids that meet a certain criteria. Defining a threshold is extremely important for the segmentation process and this choice can dramatically change your analysis. This approach should be determined based on a solid understanding of the problem, as well as existing literature and/or best practices.

Data Driven Thresholding

You can automate the thresholding process by using an approach like Otsu’s Method. This method uses the histogram of image intensity values to determine the ideal threshold based on that image. Much like what we learned about how Decision Trees are created, this method tries to break the pixels into two different groups (background and foreground) based on a tested threshold. In this case, however, it tries to maximize the between-class variance, which depends on the the mean and the count of pixels above the threshold (i.e., all foreground pixels) and the mean of pixels below or equal to that threshold (i.e., all background pixels):

$\sigma_b^2 = w_0 w_1 (\mu_0 - \mu_1)^2$

where:

• $\sigma_b^2$ = between-class variance

• $w_0$ = proportion of pixels assigned to the background class

• $w_1$ = proportion of pixels assigned to the foreground class

• $\mu_0$ = mean intensity of the background pixels

• $\mu_1$ = mean intensity of the foreground pixels

Examine the image

The following code plots a greyscale image that has pixel values between 0 - 15. These are quantized ranges of reflectivity values with a conversion of dBZ = pixel_value * 5. The maximum range of values is 0 - 75 dBZ. In the image below, you can see that there is an oval-shaped blob of higher pixel values surrounded by pixel values near zero.

Examine the histogram

This gives us an initial idea of what threshold would work best. In this case, it seems somewhere between 10 and 55 would be an ideal split to get the most variance between groups:

We can explicitly test this by looping through threshold candidates and testing the standard deviation of foreground and background pixels. We want to maximize the between-group variance.

We can create a function to calculate the between-group variance for one threshold:

And then we can run the loop and test each threshold. The highest variance is at 15 dBZ.

Finally, we can use this variance to define foreground and background pixels in the image:

We can also compare it to the built-in threshold_otsu from skimage:

Manual thresholding

While the otsu method has some utility, particularly when you do not know what threshold to use, it lacks some generalizability. Since it is data driven, the histogram of the image used to define the threshold may not be representative.

In many storm tracking applications, a threshold of 35 dBZ is a common choice for identifying thunderstorm rainfall rates vs. stratiform rainfall rates or no rainfall.

Use the widget below to apply various thresholds to the image. What threshold seems best for identifying the most intense (reds and orange) rainfall rates?

Chapter 9.2.3 - Segmentation¶

The segmentation process uses the thresholded pixels as candidate pixels between which regions / blobs / objects can be created. The basic idea is that any “foreground” pixel could be a part of an object. Typically, “background” pixels are not considered. This can make the process of segmentation easier because objects are typically sparse within a larger image. In other words, only a small fraction of the original pixels should be foreground pixels.

One of the most popular segmentation approaches used in sklearn is label and is located at sklearn.measure.label. This process connects activated pixels together to form contiguous groups. The user provides a thresholded image and a connection strategy that determines how a pixel is determined to be “touching” another pixel:

1. connectivity=1: the pixel only looks at the two pixels to the left and right (x dimension) and above and below (y dimension). In other words, pixels that touch its edges.

2. connectivity=2: the pixel looks at edges and diagonal touching.

The differences can be seen below using some sample data that only has diagonal neighbors (leftmost image).

If you use a connectivity of 1, each pixel will be its own cluster, which results in 9 clusters. If you pick a connectivity of 2, all of the pixels are associated with the same cluster.

We can apply label to the radar image and modify the threshold and connectivity to see how that changes the segmentation result.

Chapter 9.2.4 - Watershed¶

The Watershed segmentation approach is another very popular approach in the Geosciences. It is based on the idea of geophysical watersheds which geologically define how drainage occurs in rivers and streams in an region. This approach defines regions by finding local maxima (or minima) that meet some initial threshold, and then building out the region until it reaches the secondary threshold by “flooding” the region. This approach is useful for when there are regions within a region that need to be distinct. For example, “thunderstorm cells” can be connected via pixel bridges, but are defined by their convective core where the most intense precipitation occurs.

The maximum extent region is typically defined using the thresholding step. The min/max of the basin is then given as an additional threshold to the watershed method or automatically identified using local min/max identification algorithms.

Below is sklearn.segmentation.watershed applied to the same image as above. You can see that within regions that were connected above, there are now distinct sub-regions. The distance transform image is a representation of how far a pixel is from the closest basin center. You can see how it looks like topography. Pixels will “drain” away from the max values in the distance transform (mountain peaks) to the basins (valleys). Pixels on either side then “belong” to the respective basins.

Try it yourself

Bring in a gridded dataset from your term project and see if you can apply one of the segmentation approaches demonstrated above. What challenges did you encounter? Subjectively, did it do a good job? Is there some value?