P3D v5 create accurate water polys using scenProc?

[arno]

Hi,

To be honest the weeks since I have returned from vacation have been very busy with both work and family things, so I have not been able to sit down and work on this new feature a lot. I have some good imagery with NIR channel laying around that I can use for testing so that should not be the challenge.

 

[arno]

Hi,

Motivated by this reminder (and sorry for the other users who's bug now has to wait a little longer ), I had another look at my progress with water detection tonight.

Let me explain you what I am doing. I am adding additional steps to the texture filter so that you can implement different water detection strategies. One of them in the Normalized Difference Water Index (NDWI) and the other is the Two Step Urban Water Index (TSUWI). The NDWI is slightly simpler to use, but the downside is that it sometimes detects areas with lots of shadows or flat roofs as water as well. The TSUWI seems to perform better for that, if you can get the right thresholds tuned for the imagery of your area.

Below are some screenshots of test images that show the differences. As you can see they vary in how much water got detected, how many false detections there are, etc. I don't know what you want to use the detected water for in your scenery, is it to make the watermask? In that case it is probably better to miss a little water than to have a lot of false detections.

Sample image	NDWI detected water	TSUWI detected water

 

[Clutch Cargo]

That looks very promising. It will be interesting to see how it handles areas of different water colors, areas that have light reflections, or white water like rapids or just fast moving water.

 

[arno]

I'll see if I can find some samples with different water colors, although you already see in the samples above that the sea and the inland wetlands have different colors and are detected.

Both indices that I use here not only use the RGB channels of the image, but also the NIR channel. So it is not purely color that is used to find the water.

How are you planning to use the water output? For your watermask or as polygons in other processing?

 

[Clutch Cargo]

I plan to use it for both.

Right now, using my above screenshot, both the USGS and OSM show this lake water covering over the sandy beach shoreline... not real bad but as the water moves into those little inlets it gets worse. So with your new process, I would not only get accurate edgelines for the waterpoly I would also create exact matching watermasking polys to rid of vegetation. Right now I am creating those in a different method and then I add the water masking shapefiles into the scenProc script. If they both can be created at the same time that would be a huge time savor. Currently I am also using a buffer to expand the watermask a little to ensure it keeps vegetation from showing up on the water.

Maybe that could be an option in the step... just thinkin' out loud here. And an option to export as an image (b&W), geo-referenced tiffs would be nice, and as shp files.

 

[Clutch Cargo]

So I am starting to test the new water detection. I see in your announcement that you can create a watermask (exactly what I seek), however I cannot export one? I can export polygons nicely to review its outcome. I am sure it is something simple I am missing.

my script:

ImportGDAL|B:\CHICO\Section_162\Images\CHICO_AA26.tif|*|AUTODETECT

DetectFeatures|FTYPE="RASTER"|B:\simwestCAL\scenPROC 3 Texture Filter Editor\Water Detection\WaterDetect-UWI.tfc|String;Type|water|NONE|DONTPROCESSHOLES

ExportOGR|Type="water"|ESRI Shapefile|B:\CHICO\Section_162\CHICO_AA26\Water\UWI\CHICO_AA26_WATER.shp|water

(I've tried three different steps below to export a b&w watermask type tiff)

ExportGDAL|FTYPE="RASTER"|GTiff|CHICO_AA26_WATER|B:\CHICO\Section_162\Images Water|NONE <--- gave me a copy of my original color image
ExportGDAL|FTYPE="POLYGON"|GTiff|CHICO_AA26_WATER|B:\CHICO\Section_162\Images Water|NONE <--- gave me no output
ExportGDAL|Type="water"|GTiff|CHICO_AA26_WATER|B:\CHICO\Section_162\Images Water|NONE <--- gave me no output

 

[Clutch Cargo]

Couple of other questions:

In order to use UWI method I need a 4-band image. Currently I see I can only download NAIP imagery in 3-band color and Infrared images, but not a 4-band (contacting USGS about this).

1. Can the Texture Filter Editor use both separate color samples and IR samples to configure or must be a 4-band to work properly?
2. If it needs a 4-band, will the scenProc step ExportGDALCombined combine the two types of images into a 4-band?

 

[arno]

Hi,

I'll reply about the script when I'm behind my pc again (too hard to typ a script on my phone).

A texture filter has only one input image, so it can not merge different files. But the ExportGDALCombined step can indeed do that.

In the past I have downloaded 4 band USGS images via wms. But not sure if that's available for each state.

 

[arno]

Hi,

The script should be like this:

ImportGDAL|B:\CHICO\Section_162\Images\CHICO_AA26.tif|*|AUTODETECT
ExportGDAL|FTYPE="RASTER"|GTiff|CHICO_AA26_WATER|B:\CHICO\Section_162\Images Water|NONE|B:\simwestCAL\scenPROC 3 Texture Filter Editor\Water Detection\WaterDetect-UWI.tfc

The DetectFeatures step is to make vector features, but if you apply the texture configuration file directly in the ExportGDAL step the output will be the modified raster.

I guess I could also add another step that allows you to load a second raster file in the texture filter. But in general your texture filter is flexible and not linked to a specific file. So would it work to use a filter like FROMFILE="*_CIR.tif" to load a colored infrared as second image? For the primary image you would then for example use FROMFILE="*_RGB.tif". Something like that would only work if you only load a limited number of raster input files per tile. But it saves you an extra step to write the combined image to disk first.

 

[Clutch Cargo]

Thanks Arno for the example. I will try that but first I need to combine my two rasters (RGB +IR) into one which I use the following:

ImportGDAL|B:\CHICO\Section_162\Images\CHICO_AA26.tif|*|AUTODETECT
ImportGDAL|B:\CHICO\Section_162\Images Infrared Butte\CHICO_AA26_IR.tif|*|AUTODETECT
ExportGDALCombined|FROMFILE="CHICO_AA26.tif"|FROMFILE="CHICO_AA26_IR.tif"|A1;A2;A3;B0|GTiff|CHICO_AA26_4BAND|B:\CHICO\Section_162\Images Infrared Butte|NONE

Pretty straight forward but no results. I have tried a multiple of combinations with the 'A1;A2;A3;B0' but scenProc crashes. Actually, it just closes without any error message? So I have to reopen it to run again?

Question 1) - As you see, trying to create a single 4-band image to use in the TFE. Why does the step parameters in ExportGDALCombined point to the path of a folder and not export to a single image? Am I to expect a lot of LOD images that I would need to combine? When I did try adding SplitGrid|LOD13|* then scenProc does crash with the following "system argument out of range exception"

 

[Clutch Cargo]

As a follow-up to my above response -

In order to take full advantage of the TFE steps of UWI or NDWI water detection I will need a 4-band image. So far from my research, the USGS only offers (as of 2018) only 3 band natural color or NIR as separate downloads. (what is "wms"?). I have even just emailed our contact at the USGS about 4-band availability. In the meantime I thought I would try combining the bands so I could fully test the water methods. I also re-read what you said here:

"I guess I could also add another step that allows you to load a second raster file in the texture filter. But in general your texture filter is flexible and not linked to a specific file. So would it work to use a filter like FROMFILE="*_CIR.tif" to load a colored infrared as second image? For the primary image you would then for example use FROMFILE="*_RGB.tif". Something like that would only work if you only load a limited number of raster input files per tile. But it saves you an extra step to write the combined image to disk first."

Yes, I understand the flexibility of the filter. FYI - for my processing, I am spanning across approximately a 105 square kilometer area per one '.tfz' file for every 25 of my image files , so I need to create a wide-ranging filter to handle the variety of water types/colors that could be found in that size of an area. This has worked quite well for vegetation detection. Hopefully I can achieve similar results with water.

Your idea of adding another step to input a 2nd raster seems like that would save a lot of work of having to combine bands. One step for a natural color image and the 2nd for NIR. And again, both of these would include separate "sample.bmps"? Maybe 10 or so for natural color and 10 for NIR? Would that be feasible? If it worked as I imagine I could then see the results inside the TFE in real time as I make adjustments to the steps.

 

[arno]

I'm not behind my pc now, so I'm not sure if the export combined step starts counting bands at 0 or 1. I'll check that later.

As for two images in the texture filter editor, I did not plan to have multiple samples, but you are right that might be an issue. Let me think what would work there.

 

[Clutch Cargo]

Appreciate the interest and help.

 

[arno]

I did some more thinking, its probably better to add a step that merges in memory instead of directly exporting with gdal. Then you can merge the two files and directly feed the result into the texture filter. I'll try to add something.

 

[arno]

Hi,

In the step that merges the raster features the bands start counting at 0. So most likely A3 above is not valid if you loaded a 3 band RGB image.

I have also started implemented the step that does the merge in memory without saving it to disk.

 

[Clutch Cargo]

Yea, I was just trying any of the bands to see if I could get any output. I do that sometimes when I am experimenting with a new step I have not used before to get an understanding of what I am working with. Such as your mentioning of "merging in memory". I take it that saves processing time/space or something? You know best. I'll try whatever you decide with.

 

[arno]

Did you check the manual as well? It usually contains examples of how to use a step

To do the merge in memory saves you the need to first save the merge image to disk and then load it again to use it in a texture filter. So I think it will optimize your workflow. The new step has been added to the latest development release btw.

 

[Clutch Cargo]

Oh wow, I will hit this first thing Monday morning (actually probably tomorrow as I am eager to test!). Yes, I check the manual deeply as part of my ritual.