P3D v5 Water Detection idea (work-around)

[Clutch Cargo]

Hey Arno, here’s an idea for water detection. More of a work-around that may be quicker and prove worthy while true water detection continues development. What I have noticed is a lot of water is placed in the shadows of mountains, hills or steeper slopes. Not always, but maybe a good percentage (80% maybe?).

What if we used DEMs to filter out unwanted water shp polygons based on the percentage of a slope? For example, scenProc removes all water shp files that have <10% slope (user configurable). Not sure if you would need a distance or radius that determines in that area we have a slope that is greater or less than 10% but you get the idea.

So for now we would run TFE for water detection as we are doing now. We now have our water shapefiles, granted with many errors. From there we could create something like the following:


ImportGDAL|B:\CA\Section_64\Images\CA_BV47_DEM.tif|*|NOREPROJ|String;CENTER|elev
ImportOGR|B:\CA\Section_64\Masks Water\CA_BV47_WM.shp|*|*|NOREPROJ

AddAttributeSampleRaster|FROMFILE=”CA_BV47_WM.shp”|FROMFILE=”CA_BV47_DEM.tif”|slope|10

ExportOGR|WatType="water"|ESRI Shapefile|B:\CA\Section_63\Masks Water\CA_BW46_WM.shp|water


Of course I have not thought this through completely, but I am hoping you get the idea what I am thinking from above to create this type of step. I have around 30 days to find a way to get a higher percentage of correct water bodies (Ahhhh!) . Your thoughts in testing this idea in scenProc?

 

[arno]

Hi,

If you would have a shp of high slope areas you could indeed use the boolean step to subtract them from the result.

Are you proposing that scenproc can generate such a shp from a DEM file?

 

[Clutch Cargo]

"If you would have a shp of high slope areas you could indeed use the boolean step to subtract them from the result."

I would first need to create the shp files first, but shapefiles do not inherently know what slope they are at. We would need to tell them via an attribute?

"Are you proposing that scenproc can generate such a shp from a DEM file?"

My idea was to take the results of TFE water detection, which in my use I created shp files. So now I have the shapefiles of water but I do not know if they are flat or at a slope. I would then use a matching DEM image to determine what slopes they are at and the step would filter out shapefiles that do not fall within the amount of slope... maybe that is the boolean part of the step or a separate step. Maybe below will help clarify my idea:

1. Import the shape files (already created with TFE) and import the matching DEM imagery.
2. Add an attribute(?) to the shp files as to what slope each shp is based on the DEM.
3. Use a Boolean step to filter out those shps that exceed the % the user desires.
4. Export the results as a new/modified shp file (at lease that is what I need as I turn turn those shps into WM tiff files).

What I don't understand is how one determines the slope % from a DEM image. Would that mean scenProc needs to create two points and measue at what distance to determine the slope of a shp. Or perhaps a radius? That's where I need your smarts on this.

 

[Clutch Cargo]

Found this.... maybe it would be of help?

"Generate a slope map from any GDAL-supported elevation raster:"
gdaldem slope <input_dem> <output_slope_map>
[-p] [-s <scale>]
[-alg ZevenbergenThorne]
[-compute_edges] [-b <band>] [-of <format>] [-co <NAME>=<VALUE>]... [-q]

gdaldem — GDAL documentation

gdal.org

 

[arno]

Yes, that would work. You can make a slope map with gdal (I have used that before) and then use that map in scenproc to sample.

 

[Clutch Cargo]

I was hoping something like the above could be created as a step inside scenProc?

Also, how about a filter to make out polygons of a determined size or smaller... ie <5m. While a slope filter would help for those waters on hillside shadows, the more I study the artifacts there are also 100's on flat croplands, inside towns a cities that are not on a slope. I did a test in QGIS where added a _WM layer and then calculated the $area for each polygon. Here's what I did.

Typical results of waterpolys after I ran through TFE. With waterpolys in the blue it looks like TFE did a good job.

But when I select all waterpolys you can the the 100s of artifacts. Many in the town, on crops or even patches of dirt.

So in QGIS I created a Field and got the size ($area) of each poly. Simple function to select polys under a certain size and deleted. Here are the results:

That cleaned out most of the tiny, unwanted polys yet left most of the small ponds and canals of water. What normally takes up to an hour to "clean up" would get this down to maybe 5 minutes. Could scenProc implement a step like so? THat way I would not have to keep going back & forth among apps. I could see this filter step be applied right after the TFE water detection (like slope), and produce results in shapefiles and tiff if wanted.

 

[arno]

Hi,

You can use the FAREA attribute already to get the area of the feature. So that can be used easily in filtering.

I have put the slope calculation on the wishlist of features.

 

[Clutch Cargo]

THANK YOU!

So, I am testing with FAREA not fully understanding how to use it.
1. FAREA Type = Double. What is a Double? What does that look like? 0.0, 0:0;0:0 ?

2. Do I first use it as a way to add it like an attribute like so?

AddAttribute|FROMFILE="CA_BX45_WM.shp"|Double;FAREA|*

All I got was "0's" in the results, so I do not think so. So scenProc automatically processes this behind the scenes?

3. Since you have it listed under the section of Filtering in the User's Manual, I am assuming I would be using it in a processing step. Not sure how I would write the step as I am not comparing it to a second object /layer/file? ... so far:

FilterFeatures|FROMFILE="CA_BX45_WM.shp"|FAREA < 5m|mode|distance

I know this is incorrect... we'll keep studying the manual for clues.

Update, ...now I am thinking it should be more of something like:

AddAttribute|FROMFILE="CA_BX45_WM.shp" AND FAREA <XX|Integer:WaterSize|(Square meters of waterpoly)

What I am thinking I need is to know the size of each poly so I can determine what is the best size for an overall filter. Not just filter out the polys automatically from a given size as each area I work may need adjusting as to what is the best filter size. Ha... might be losing you on what I am thinkng.

 

[arno]

Hi,

1. FAREA Type = Double. What is a Double? What does that look like? 0.0, 0:0;0:0 ?

The attribute has a value that is the area of the polygon in square meters. So if you use a test like FAREA>10 you would only get polygons that have an area of more than 10 square meters.

2. Do I first use it as a way to add it like an attribute like so?

AddAttribute|FROMFILE="CA_BX45_WM.shp"|Double;FAREA|*

No, it is a special attribute that is automatically calculated for each feature. So there is no need to add it first.

 

[Clutch Cargo]

I have been conducting more test with water detection and just wanted to update some observations. My best results seem to be using NDWI (water) as opposed to using UWI Urban Water and USI Urban Shadow. With that I believe I came very close to achieving "perfection" in detection. Let's look at a few screenshots:

Here's results showing water with the steps shown further below in another screenshot:

Now same area with highlighted water polys. As you can see getting some artifacts in the mountains.

Lets take a close up of those little polys.

As you can see they are very small. I would think I should be able to filter those out using the following step:

DetectFeatures|FTYPE="RASTER"and FAREA>200|(path to tf2 file)|NONE|DONTPROCESSHOLES

200 square meters is huge! I would think none of those artifacts would be there. I tried a few different sizes >100, >200, >400 but makes no difference in the results. I have also tried the filter this way using the following step at exporting stage:

ExportOGR|WatType="water" AND FAREA>200|ESRI Shapefile| (path to export shp file)|water

No difference. Am I using FAREA correctly?

THe other observation is when I am in TFE look at the detection:

The river looks pretty good and not a single artifact there. And that is using the step SVM so this should look like the finished result I would think. I then looked as few other sample images and they too look really good:

yet the finished results look very different with lots of artifacts as previously discussed in the past. Could it be the algorithm itself and not so much needing more samples? I was only using about 15 samples and only appointed sample points to six of them. Typically I may use 40-50 sample images.

Question... Lets say I have loaded three sample images. I apply a few sample points and move on to sample image 2. When I add sample points to sample 2 does that mean that sample image one will also be affected by the new samples or is it from sample 2 onward?

It's interesting, I have noticed after I process an sample image I may get a few water polys that pop up where there is no water. So I add some red points (no water) on top of them to get rid of them. I reprocess and that works, they disappear but then some additional polys appear in a different part of the sample where there was none before. It seems odd that if I am adding only negative sample points that water would appear in an area that was previously void based on past sample point from another sample image.

It's all good... just the latest... feeling the urge to jump back into this?!?!?!

 

[Clutch Cargo]

This image was posted in another thread discussing creating DEM polys. Looking at that black line that I am pretty sure is a river, do you think DEMs could be used as an additional step or filter in scenProc or in TFE?

 

[arno]

Hi,

I have been conducting more test with water detection and just wanted to update some observations. My best results seem to be using NDWI (water) as opposed to using UWI Urban Water and USI Urban Shadow. With that I believe I came very close to achieving "perfection" in detection.

I have added these different indices, so that users can try them and see which one works best for their images. I don't think you can say that one is always better than the other. The UWI and USI combined theoretically should be better in avoiding that shadows are detected. But reality might be different

Now same area with highlighted water polys. As you can see getting some artifacts in the mountains.

Those are probably shadows, I have seen that in tests I have been doing as well.

As you can see they are very small. I would think I should be able to filter those out using the following step:

DetectFeatures|FTYPE="RASTER"and FAREA>200|(path to tf2 file)|NONE|DONTPROCESSHOLES

No, you are now applying the area filter on your input raster polygons. You want to run the area filter over the detection output. So that would be a FilterFeatures step after the DetectFeatures step, where you use the polygon area as filter.

ExportOGR|WatType="water" AND FAREA>200|ESRI Shapefile| (path to export shp file)|water

No difference. Am I using FAREA correctly?

Filtering like this at export should also work. Did you check that these polygons are separate polygons and not part of a bigger multi-polygon?

yet the finished results look very different with lots of artifacts as previously discussed in the past. Could it be the algorithm itself and not so much needing more samples? I was only using about 15 samples and only appointed sample points to six of them. Typically I may use 40-50 sample images.

Usually this means that in the areas where the results are different, the characteristics are slightly different from the samples you used. So in those cases I try to add more samples from the areas that don't work so well. I would assume that if you run the part of the image where the sample come from that you get good results.

Another different might be that the segmentation of the image gives slightly different results when you run it on the big image. But that should not give huge differences in most cases.

Question... Lets say I have loaded three sample images. I apply a few sample points and move on to sample image 2. When I add sample points to sample 2 does that mean that sample image one will also be affected by the new samples or is it from sample 2 onward?

Not sure if I understand. You add the sample points per image. Then the SVM step is trained on all the different sample points that you provided (so all sample points from all images are combined into the dataset that is used to train the algorithm).

It's interesting, I have noticed after I process an sample image I may get a few water polys that pop up where there is no water. So I add some red points (no water) on top of them to get rid of them. I reprocess and that works, they disappear but then some additional polys appear in a different part of the sample where there was none before. It seems odd that if I am adding only negative sample points that water would appear in an area that was previously void based on past sample point from another sample image.

That is very well possible. You are just giving more positive and negative points to the algorithms. Depending on the parameters of the algorithm it will try to define the right classes. But even by adding only negative points, the class definition of what is positive can also change.

It's all good... just the latest... feeling the urge to jump back into this?!?!?!

Still quite busy with loads of ModelConverterX issues. So it won't be in the next few weeks for sure. But the more you post, the more likely that you will distract me one day

 

[arno]

Hi,

This image was posted in another thread discussing creating DEM polys. Looking at that black line that I am pretty sure is a river, do you think DEMs could be used as an additional step or filter in scenProc or in TFE?

You would have to examine the DEM data. The black parts are most like tagged as "no data", or sometimes there are separate values for "no data" and for "water". But I don't really see how this could help with the detection, you could just use the polygons extracted from the DEM as your water data of course.

 

[Clutch Cargo]

Thanks for all the detailed replies. Back to testing!

 

[Clutch Cargo]

Arno! Time to "sweet talk" you again to back into water detection . It's been 4 months since I started this thread. We really need your help on this topic. Our project has stalled because of this (bummer). While I know the digging into the true water detection is a bigger project, I am thinking adding a slope % filter might be a good enough work around the current issues of water detection to allow us to move forward. Any way I can help I am more than happy to, but I need you to take the first steps. Thank you so much for all the help you provide.

 

[arno]

Hi,

Still most of my brain power goes to the MCX features I am working on now, but let's see if we can add this extra step easily. I have some questions on how you intend to use it.

1. Does your DEM data and imagery data have the same resolution? If not, something has to be done to make sure that the slope information is available at the same resolution.
2. Each texture filter takes one input image as input (this can have multiple layers). So probably the slope information has to be added as for example a 5th channel to that image so that you can use it in the filter. Or filters should be modified so that they can take more than one input image.
3. Elevation data is typically not stores as a byte (range 0 -255), but normally as a float or int. The teture filter only works on bytes. So I think this means there needs to be a step outside of the fitler to calculate the slope. If we store that in degrees it would fit in the byte range again. And then inside the filter that information can be used.
4. Do you expect scenProc to calculate the slope or do you plan to do that in an external tool (e..g GDAL) beforehand?

So the most logical approach to this I see now is to calculate the slope in a scenProc step. Then either merge the result of that with your image before you pass it to the texture filter or the texture filter will need to be modified to allow an additional input.

Does this should like a workable workflow for you?

 

[Clutch Cargo]

I had no idea you were still so heavily involved in MCX still, which is good as we use it too, but I appreciate you taking the time to perhaps put something in place for me to experiment with.

1. No, the resolutions of DEM and Imagery are not the same. Typically, we use 60cm per pixel on our imagery, that is our default. However, there has been request for finer resolutions so we may go to 30cm or even 15cm for smaller sized projects. For DEM we are using 30m per pixel. Quite a difference. We may at some time go down to 15m per pixel but for now 30m works good for us. It would be nice if scenProc can determine the resolution of both DEM and imagery on its own but I would not be against where one would have to manually input the resolution of both in the step itself, if that's what you need.
2. If I am answering correctly, we import one 60cm image and a matching DEM image, same size but different resolutions, per scenProc script. In my scripts, I have never imported more than one image at a time, that may be due to the area we work with (approximately 30 square kilometers), or just because it makes it easier for me to automate running 25 scripts (25 images) at a time. For other developers, they may have different methods.
3. That sounds good to me, ha! Beyond my knowledge, but it sounds like it would require an additional step which is fine by me.
4. I would like scenProc to calculate the slope in a step and for the user be able to adjust the filter (i.e. If I put in 4% as the filter, any slope greater than 4% would be deleted) What I do not understand is how scenProc would determine how large of an area is used to determine the slope for that area?

I think your approach to the step(s) required is quite workable for me.

 

[arno]

Hi,

I think it might make more sense to calculate the slope (or roughness) outside of the texture filter and then create a polygon for the area that is acceptable for water (e.g. by using a threshold). That polygon can then be used as mask for the texture filter. I think that's more efficient than getting the DEM into the texture filter itself.

 

[Clutch Cargo]

Sounds good, I am open to try any approach. You say it's more efficient than getting the DEM into the texture filter itself. Curious, is that due to the resolution difference or concern on the size of the DEM file? FYI - Our DEM files are very small, only 238kb.

 

[arno]

Getting it into the texture filter would mean the resolution has to match, so the up sampling will use quite some memory. And it would also make the filter much more complex to work with and test if multiple images are involved in using it.

I'm currently away from my FS PC for a few days, but after that I can check how to add the DEM processing step.