Texture filter editor machine learning steps changes

[arno]

In the latest scenProc development release there are a number of changes to how the machine learning steps work. This post gives an overview of them:

• Besides the SVM machine learning step that has been available for a while, a new one has been added now. This one uses the Artificial Neural Networks – Multi-Layer Perceptrons (MLP) algorithm for the machine learning.
  While working on detecting of water data, I had the impression this algorithm gives better results. But I’m not an expert on the differences of these steps and they both have a lot of attributes to tune them. So I guess you best see for yourself which one works best for you.
• When you change attributes of a machine learning step or add sample points, the machine learning algorithm is not trained automatically every time. This was quite annoying as the training can take long with many sample points. Now the step will be rendered in red in the step overview and you have to manually trigger the retraining of the step with a button on the toolbar when you are ready for it.
• Besides training machine learning steps with sample points, it is now also possible to train them with raster data. So if you have reference raster data of the required classification or if you have good vector data so that you can make such reference data you can use this as an alternative way to train the algorithm. The machine learning steps have two optional inputs for this, for the labels and the reference data. If any of those two inputs are not connected in your filter the sample points are used for training. Be aware that training on raster data typically takes longer as it includes a lot more training data.

Continue reading...

 

[Clutch Cargo]

Hope to try it in the next couple of days. Is there a sample script for this in the latest scenPROC manual? If so, I did not find it. If there is not one can you post a sample script here, sort of a default script as where to start? Maybe one with sample using sample points and the 2nd using the images themselves if there is any difference. Thx.

 

[arno]

Hi,

I did you send a PM with the test script I used on your sample image. You can use that as a start.

 

[Clutch Cargo]

I started testing you latest version starting with vegetation detection first. So it seems the previous version of saved tf2 files are not compatible with the new version? At first glance I see SVM and the merge steps have change cause my old scripts to crash. Noticed two issues which may be related to the new version.

1) - I noticed now when I have a split screen, add some points and click to process nothing was being updated based on my points. I discovered I have to now move on to the next image then back track to the previous image to see the results. Did not have to do that before as the results were instantaneous.

2) - After complete my points and saving, I tried to run a detection process (again, this is for veg and not the new water detection). I receive the following error on this step:

" Detect Features The machine learning step training is outdated, you probably want to retrain it in the texture editor first."

So have my previous script of steps is now obsolete? I take it these are due to the new steps not working with my old scripts.

I will study what you supplied me. Thought I could use my old script for veggies... guess not. I will report back after I dive into your samples.

 

[arno]

I started testing you latest version starting with vegetation detection first. So it seems the previous version of saved tf2 files are not compatible with the new version? At first glance I see SVM and the merge steps have change cause my old scripts to crash. Noticed two issues which may be related to the new version.

These steps have changed indeed, but it is supposed to automatically upgrade the script. So what issue do you have?

1) - I noticed now when I have a split screen, add some points and click to process nothing was being updated based on my points. I discovered I have to now move on to the next image then back track to the previous image to see the results. Did not have to do that before as the results were instantaneous.

That's not supposed to happen. What has changed is that you manually need to trigger the training.

2) - After complete my points and saving, I tried to run a detection process (again, this is for veg and not the new water detection). I receive the following error on this step:

" Detect Features The machine learning step training is outdated, you probably want to retrain it in the texture editor first."

So have my previous script of steps is now obsolete? I take it these are due to the new steps not working with my old scripts.

For your old script you can probably ignore this warning, but when you have worked on the script it's an indication that you changed parameters or added sample points without training the algorithm again.

 

[Clutch Cargo]

Did more testing this weekend with the newest (what I call the new water version), though I was still testing for vegetation.


1) Like stated previously, I can confirm that adding points and processing does not show immediate results. Have to move onto the next or previous image, then return to the image I was working on to see the results.

2) Here's a sample of my steps from the scenProc previous version - my pretty standard script for veggies:

I ran this script and it worked fine, no errors. I see the new water version scenProc updated two steps, the MergeImages and SVM step to this:

I then added an extra sample image and a few points and saved as a different tf2 file just to make sure. Ran the scenProc detection and still get the same error as before:

" Detect Features The machine learning step training is outdated, you probably want to retrain it in the texture editor first."

So script with old steps on previous scenProc is fine, script with modified steps with new version results in not able to complete processing.

 

[arno]

1) Like stated previously, I can confirm that adding points and processing does not show immediate results. Have to move onto the next or previous image, then return to the image I was working on to see the results.

What kind of results do you mean here? Do you mean the extra sample point you added is not rendered on screen until you swap images? Or do you mean that you do not see that the detection takes the new sample point into account? The last is by design, as the training is slow it does not happen after every minor change automatically anymore.

2) Here's a sample of my steps from the scenProc previous version - my pretty standard script for veggies:

View attachment 93634

I ran this script and it worked fine, no errors. I see the new water version scenProc updated two steps, the MergeImages and SVM step to this:

View attachment 93635

That looks fine. Merge images and SVM have indeed some additional (optional) inputs. But the images show that your old script has been converted fine. I don't see any issues in that.

I then added an extra sample image and a few points and saved as a different tf2 file just to make sure. Ran the scenProc detection and still get the same error as before:

" Detect Features The machine learning step training is outdated, you probably want to retrain it in the texture editor first."

Did you press the Train ML button in the TFE after you added the extra image and the sample points?

 

[Clutch Cargo]

1) "Or do you mean that you do not see that the detection takes the new sample point into account?" Yes, that is what's happening. Ok, I was not aware of the Train ML button, so I added a couple more sample images. I then added a few more points. In the previous version, I would then click just about anywhere and it would update incorporating the new points. Now I see that does not work just to confirm. So I click on the Train ML button. It turns blue for a moment then when it turns off the blue color I suspect that is when the processing has ended. So I did that. Still, I see no changes on my new points (in this case green adding a few tree areas). I click to the next image and then back. Now I see the changes.

However I also noticed, if I do not click the Train ML button and move to the next image and then back, the processing has been updated? Is it designed to update automatically if I move on to the next sample image?

By the way, what does "Train ML" stand for?

I think I like the idea of the button. That way I can go through several images and then process them all together one time. Not waiting every time I add a point to two.

2) Ok, so these changes are correct, good to hear.

3) Got an unhandled exception when I tried to save to a new filename.

 

[Clutch Cargo]

Update crashed again when running scenProc with my new TF2 file. See screenies below:

I did send in the error report.

Can the new TF2 file I created be used on the previous version of scenProc?.

 

[arno]

By the way, what does "Train ML" stand for?

It stands for traing machine learning. SVM and MLP are both machine learning algorithms that need to be trained.

I think I like the idea of the button. That way I can go through several images and then process them all together one time. Not waiting every time I add a point to two.

That's exactly why I changed it, because often I had to wait minutes for the training, while I still wanted to check some other samples.

3) Got an unhandled exception when I tried to save to a new filename.

Do you have the details of that error as well?

Update crashed again when running scenProc with my new TF2 file. See screenies below:

View attachment 93641

I did send in the error report.

Unfortunately there seems to be an error with the error reporting, as I did not get it. But hopefully with this information I can find something already. Does your image in the script have the same number of bands as the ones used for training?

Can the new TF2 file I created be used on the previous version of scenProc?.

Yes/no. If you use the new MLP step or the updated Merge images step then it will not work in the old version. If you don't use these steps it will work.

 

[arno]

I have fixed the issue with the error reporting now (I hope), that will be part of the next development release.

I did try to Google for the OpenCV error and it seems that it can come from the KMeans clustering function when the input image has less colors/pixels than the number of pixels specified. Could it be there are some areas you are processing that have very small images or so?

 

[Clutch Cargo]

I had a feeling it had something to do with bands. I have d/l two ways for the latest area I am working on: First way is separate RGB and IR downloads, then merging them into a single 4band sample image. It seem to work in detection but not final processing. I would get the error in the image I posted above. Could it by when merging them they were a pixel off or something casuing the error?

My 2nd method of d/l is how I normally do it via our API. So it is a 4band image to begin with. But it too is getting the same error as posted above?!? This does not make sense? It did process one time fully and completely but now... nothing. Obviously a little confused. Now this is with version v3.2.0.0 r8909683 DEV 6/1/2024, which I believe is the last version before the new Train ML version.
So I have been going back to this one cause I could not get the new one to work. Ha, now the old one isn't working. Guess I should try the new version again and report back.

 

[Clutch Cargo]

Update - back to the TML version. Created a new template with the updated Merge Images and SVM steps. I added one 4band sample image (image that downloads as a 4band). What caught my surprise is TFE is already showing where veg should and should not be. I have not added a single point and it looks great?! I have not clicked on Train ML either. Is this the new way?

So if I click on TML button it does nothing without a point added, so I guess I need at least one. So I did, let it run and then saved the tF2 file. Then started the scenProc detection. Once again, getting the:

1:37 PM DetectFeatures Warning The machine learning step training is outdated, you probably want to retrain it in the texture editor first

So that is where I am still stuck on the new version. Again going back to previous is cause the other error message...AHHH!

 

[Clutch Cargo]

Further update. Just an FYI, I must have mixed up my 4band download and/or sample images. Re d/l and now TFE IS working correctly. So apologies thinking it was an issue on scenProc's part. So I can continue using the old version until you release your latest with TML.

 

[Clutch Cargo]

One further update and I only post it here because it is giving me the error message in the image I posted above. I run TFE detecting in batches of 25 with three scripts running at a time of those 25. This is how I always done it for the past couple of years. This time out of the 25, eight of them crashed. Never seen it with some working and some not since they use the same tF2 file, the same all new downloaded 4band images and sample files and the same scenProc script. Only difference is files are saved in different locations. Wierd eh? This is not your new TLM version so I guess it would not be worth it ask you to look at it. Just wait for your new release. Unless you want me to send a log report? But again, it is that red error message line and the accompanying little error handler window that opens.

 

[arno]

I had a feeling it had something to do with bands. I have d/l two ways for the latest area I am working on: First way is separate RGB and IR downloads, then merging them into a single 4band sample image. It seem to work in detection but not final processing. I would get the error in the image I posted above. Could it by when merging them they were a pixel off or something casuing the error?

I had another look and I think it is more likely that the number of pixels is smaller than the number of clusters. What value did you set for the number of clusters? Maybe SplitGrid left a small 1x1 or 2x2 pixel image in some cell near the edge of your image. I think I can easily make the step robust for this.

Update - back to the TML version. Created a new template with the updated Merge Images and SVM steps. I added one 4band sample image (image that downloads as a 4band). What caught my surprise is TFE is already showing where veg should and should not be. I have not added a single point and it looks great?! I have not clicked on Train ML either. Is this the new way?

Nope. Did you start with a completely new template or did you start from an old one? Sounds like some old training data might be left.

So if I click on TML button it does nothing without a point added, so I guess I need at least one. So I did, let it run and then saved the tF2 file. Then started the scenProc detection. Once again, getting the:

1:37 PM DetectFeatures Warning The machine learning step training is outdated, you probably want to retrain it in the texture editor first

So that is where I am still stuck on the new version. Again going back to previous is cause the other error message...AHHH!

Yes, you need at least two sample points (one positive and one negative) for the training to work. After the training has finished the ML step will no longer show with the red-ish background in the step window (it will have a white background). If you then save your TF2 file you should not get this warning anymore.

 

[Clutch Cargo]

Just to be safe, I created a new blank TFE "canvas". No sample images or points. I re-created my script and then saved that as my template.

Oh, so it may be clustering and not the bands. That makes me feel a little better as I wasn't how I could manipute the bands.

Regarding the Splitgrid, my K-Means Cluster settings are the same as the default settings:
Attempts 5
Cleanup Iterations 1
ClusterCenter PPCenters
Name
NumberOfClusters 8

Maybe changing the number of clusters on those that crashed may help? I'll try a few settings.

 

[Clutch Cargo]

This morning I tried various settings for the K-means however, nothing appeared to work. Same reported issue as above. Noticing you mentioned the Splitgrid I looked at mine. I had it set as SplitGrid|LOD15|* . To be honest I could not remember why I had it set for that (thinking it may of had something to do when I was testing processing holes). So I changed it to SplitGrid|AGN|* a pretty common setting I believe as one of your examples. Used a fresh TFE template and guess what, using the new TML scenProc it now got past the "needs to be trained warning". So I finally saw it processing. Unfortunately, even this version came up with the same crash (red line error + the error handling window), that I am seeing in the previous version of scenProc. So a little at a loss of what to try next? Should I upload the image & samples causing this and my script for you to try?

 

[arno]

I'll modify the step as well to automatically reduce the number when there are less than 8 pixels. That should fix the error.

 

[arno]

Did you get the same errors this time or new ones?

I'll try to fix the kmeans issue tonight and then also push a new development release with the error reporting working. Then it's easier for me to see where things go wrong.