P3D v5 Unhandled Exceptions and a possible bug found

[Clutch Cargo]

Here is a new list of UE's I have encountered today. I believe this has also lead me to find a bug in TFE. I continued to get a repeatable UE every time I clicked on one of my samples to view in the preview panel. (UE3, UE4,and UE5). I can view a 4-band RGBN tiff, a 3-band RGB bmp but NOT a 3-band RGB tif. Crashes every time. Wanting to test the Back Projection method with 3-Band tiffs instead of a 4-band tiffs but right now a no go. I will try in the meantime a 3-band bmp. I have attached three of my samples, each in a different format.

Unhandled Exception Errors experienced today.

UE1 - I was working on a TF2 file. Saved it and left it open. I then went to scenProc, and ran the script that had that particular TF2 file in it. After processing I went back to the TFE and clicked on a Dilate step icon. That is when the error occurred. This happened two times in a row. I will close the TFE this time, run the script and then reopen TFE.

UE2 – Opened a formally saved TF2 file that had no sample images. Added 21 sample images and saved the TF2 with the same name (updated file). I then clicked on the first image and that is when the UE occurred.

UE3 – Immediately after UE2 I closed scenProc and reopened same scenProc script. Opened TFE and selected the former TF2 file that previously had the UE2. Now I immediately got another UE. The sample images are there but the steps on the bottom panel are missing. In place of them is a big red “x” across the panel. Closed TFE (and with it goes scenProc). Just tried opening again same TF2 file. Another UE (details are in UE3). File is corrupt now.

UE4 – I deleted the TF2 files from UE3 and create a new TF2. I added 16 images, saved again and then once again when I clicked on the first image to see in the preview panel I once again got a UE.

UE5 – exact same as UE4, the second I click on a sample image (1200x1200px) 3-band RGB I get UE. Looks like the issue is repeatable.

 

[arno]

Hi,

Here is a new list of UE's I have encountered today. I believe this has also lead me to find a bug in TFE. I continued to get a repeatable UE every time I clicked on one of my samples to view in the preview panel. (UE3, UE4,and UE5). I can view a 4-band RGBN tiff, a 3-band RGB bmp but NOT a 3-band RGB tif. Crashes every time. Wanting to test the Back Projection method with 3-Band tiffs instead of a 4-band tiffs but right now a no go. I will try in the meantime a 3-band bmp. I have attached three of my samples, each in a different format.

I have seen that before. You can use a GeoTIFF images as a sample, but if you use a normal TIF image for some reason GDAL gives an error indeed. So either make sure its a GeoTIFF or use a different format like BMP or PNG.

UE1 - I was working on a TF2 file. Saved it and left it open. I then went to scenProc, and ran the script that had that particular TF2 file in it. After processing I went back to the TFE and clicked on a Dilate step icon. That is when the error occurred. This happened two times in a row. I will close the TFE this time, run the script and then reopen TFE.

It is reporting one of the sample files can not be found. I will see if I can reproduce this. Maybe having it open in the editor and running the script interferes with each other, let me check that.

UE2 – Opened a formally saved TF2 file that had no sample images. Added 21 sample images and saved the TF2 with the same name (updated file). I then clicked on the first image and that is when the UE occurred.

The error says that the file is in use by another application and therefore can not be read.

UE3 – Immediately after UE2 I closed scenProc and reopened same scenProc script. Opened TFE and selected the former TF2 file that previously had the UE2. Now I immediately got another UE. The sample images are there but the steps on the bottom panel are missing. In place of them is a big red “x” across the panel. Closed TFE (and with it goes scenProc). Just tried opening again same TF2 file. Another UE (details are in UE3). File is corrupt now.
UE4 – I deleted the TF2 files from UE3 and create a new TF2. I added 16 images, saved again and then once again when I clicked on the first image to see in the preview panel I once again got a UE.
UE5 – exact same as UE4, the second I click on a sample image (1200x1200px) 3-band RGB I get UE. Looks like the issue is repeatable.

These three are all related with the file format as mentioned above I think.

 

[Clutch Cargo]

UE3-UE4-UE5 - All my TIFFs are all geoTiffs. I have been testing 3-band geotiffs and 4-band geotiffs but have not been tracking errors against those.
Question: Can the sample images and the "base image" (the full-size image it is being detected against). can these be of different formats? Like the main image it geoTiff and the image samples are bmps? This I think would only matter if I am using a Back Projection method but I am just curious?

UE2 - I have kept open QGIS and I remember you said it could cause issues. Could be related to UE1 as well. I will make sure I close QGIS before running scripts or even with TFE open to limit issue variables. I will also always close TFE before processing for now.

Another set of UEs I have encountered today posted below. This is before I was closing QGIS or TFE so we'll see if the next set is fewer or none at all. Also, these are using the NDWI water detection method.

UE6 – Created a TF2 for water detection using the NDWI method using 4-band samples against a 4-band image. This is using the sample setup as found in Figure 6-29 in the scenProc Manual. I changed the value of the Threshold Binary step then clicked on the Erode step icon. That is when this UE occurred.

UE7 – I added step Icon, it was alone in panel and not connected to any other step. Deleted then deleted it and the UE occurred. I closed file without saving.

UE8 – Opened previous TF2 file and just clicked back and forth on some of the step icons then I got the UE.

UE9 – Opened a previously saved TF2 file. This is the NDWI method. I disconnected a couple of steps to try a a new step not included in the sample figure 6-29. I added a ‘Split Into 4 Images’ step. I connected it into the path and then clicked on the icon step after it. UE occurred.

Which leads me to this question. Can all steps available be added in a path for a certain method, or a wrong step will cause a crash. Not knowing what I can add or not add leaves me to trial-n-error but with so many UE’s not sure if it is my selection or a bug?

 

[arno]

UE3-UE4-UE5 - All my TIFFs are all geoTiffs. I have been testing 3-band geotiffs and 4-band geotiffs but have not been tracking errors against those.
Question: Can the sample images and the "base image" (the full-size image it is being detected against). can these be of different formats? Like the main image it geoTiff and the image samples are bmps? This I think would only matter if I am using a Back Projection method but I am just curious?

Humm, I have only seen this error before when I used non-GeoTIFF TIFF files. Let me see if I can reproduce it somehow.

Yes, you can use different formats for the samples and the image you feed from the script. Actually in the editor it is not so important that they are georeferenced (unless you are burning vector data into them), so a PNG or BMP file will work fine. When feeding from the script they need to be properly geo-referenced of course.

UE2 - I have kept open QGIS and I remember you said it could cause issues. Could be related to UE1 as well. I will make sure I close QGIS before running scripts or even with TFE open to limit issue variables. I will also always close TFE before processing for now.

If you had the sample open in QGIS as well that might be part of the problem.

UE6 – Created a TF2 for water detection using the NDWI method using 4-band samples against a 4-band image. This is using the sample setup as found in Figure 6-29 in the scenProc Manual. I changed the value of the Threshold Binary step then clicked on the Erode step icon. That is when this UE occurred.

I can't exactly see from the error what is going on here, some object is a null pointer it seems.

UE7 – I added step Icon, it was alone in panel and not connected to any other step. Deleted then deleted it and the UE occurred. I closed file without saving.

Also a null pointer. The Limit3 step is used to drop additional bands and make the image 3 band. Is that what you planned to do and was the input a 4 band image? Although I should check why there is an error of course, it should be robust for incorrect inputs or other things.

UE8 – Opened previous TF2 file and just clicked back and forth on some of the step icons then I got the UE.

Once again some null pointer. Hard to see from the error what happened exactly.

UE9 – Opened a previously saved TF2 file. This is the NDWI method. I disconnected a couple of steps to try a a new step not included in the sample figure 6-29. I added a ‘Split Into 4 Images’ step. I connected it into the path and then clicked on the icon step after it. UE occurred.

Some index is out of range, are you sure the input image has 4 bands when using the Split4 step?

Which leads me to this question. Can all steps available be added in a path for a certain method, or a wrong step will cause a crash. Not knowing what I can add or not add leaves me to trial-n-error but with so many UE’s not sure if it is my selection or a bug?

Yes, in general you can link most steps to each other. In practice some steps make a 1 band output for example and then feeding them into a step that expects multiple bands will not make sense. In the changes I am coding now I have colored the connectors to make it more clear what kind of input or output each connector has/wants. The validation of the filter should also take care of these things.

When I use the editor I don't get so many errors, but I guess I am always using it in the same way. I should make sure it is more robust. I will try to do some testing (miss abuse) of the tool see if I can make it more robust. But I am also very busy with the machine learning improvements at the moment, so I need to see when I have the time to look into this (I don't want these two developments to give conflicts with each other).

 

[Clutch Cargo]

I noticed the number of UE's has diminished now that I am closing QGIS as I work with TFE. I will still track. I hope my images are not the source of some of these UEs. I am creating my own RGBN images and samples. I have attached a sample. Would you mind taking a quick look to verify that what I created is a correct 4-band image?

 

[arno]

Hi,

Thanks for the extra details. Which files did you have open in QGIS? Because once you add a sample image it should be copied to a different folder and I don't assume you opened files from the texture filter temp folder in QGIS?

 

[arno]

I noticed the number of UE's has diminished now that I am closing QGIS as I work with TFE. I will still track. I hope my images are not the source of some of these UEs. I am creating my own RGBN images and samples. I have attached a sample. Would you mind taking a quick look to verify that what I created is a correct 4-band image?

Yes, the sample image you attached is 4-band and the last band seems to contain NIR indeed.

 

[Clutch Cargo]

Good to hear on my image. I can't recall which files exactly, however I believe it was vegetation shp that I was reviewing after I ran scenProc with a TFE tf2 file in the script. Your right, doesn't really make sense why that would cause a UE in TFE. I was thinking maybe QGIS was keep files opened or should I say "locked" unless I removed them from QGIS and then saved QGIS after removing them. I'll test more next week.

 

[arno]

I have made some changes already that should make the texture filter editor more robust. I will put them in the next development release. I want to test a few more situations that you described first however.

 

[Clutch Cargo]

This is to confirm that 4-Band and even 3-band TIFF images crash TFE. I can repeat the UE every time. Yet a 4-band or 3-band bmp has no issue with UE (it does seem to have a different issue which I will make a separate post. I think you said you saw this issue in the past? I have attached samples of 3 and 4-band TIFF, 3 and 4-Band bmp and the UE.

UE10- was simple, I added a couple of 4-Band TIFF samples. Clicked on one and UE occured. Did this around 3-4 times, UE every time. Clicked on a bmp or 3-band tiff, no issue.

 

[arno]

Thanks, I'll try to reproduce it here with those files.

 

[Clutch Cargo]

UE11 - I had one TF2 open and wanted to open a different TF2 file. I clicked OPEN and intermediately got this UE.

 

[arno]

You get a lot of file in use errors it seems. These are a bit hard to debug. Either some other process is accessing the files (e.g. a virusscanner or so) or it must be scenProc itself that is still using the files. But I don't get this kind of errors so often when I work with the texture filter editor here. I'll see if I can reproduce it somehow.

 

[arno]

This is to confirm that 4-Band and even 3-band TIFF images crash TFE. I can repeat the UE every time. Yet a 4-band or 3-band bmp has no issue with UE (it does seem to have a different issue which I will make a separate post. I think you said you saw this issue in the past? I have attached samples of 3 and 4-band TIFF, 3 and 4-Band bmp and the UE.

That's in inline with what I posted before. If you use TIFF files, make sure they are GeoTIFF files. The GDAL library seems to have an issue with normal TIFF files and throws this error. I'll have another look to see if I can find a workaround for this. But for now I either use GeoTIFF or PNG files in most cases.

 

[Clutch Cargo]

Yea, thx for clarification... forgot that you mentioned it. My tiff samples are not Geo referenced. I take it you are using some GDAL script to create your to keep reference? I've been pumping mine out in Photoshop so I lose reference once I save it. I don't know how I could reference them against the original image unless its pixel by pixel... Think for now I will concentrate on 4-band bmps.

So bmps should give the same accuracy\results like tiffs or I really should see no difference? Just don't mix different sample types.

I'll keep checking myself on what I might have open or "in use". In fact the next UE I will stop and see what is open.

 

[arno]

I think I have a fix for the issue with normal tiffs now, but I need to run some more testing to be sure nothing is broken.

 

[Clutch Cargo]

For me, I am fine using either type now that I have created scripts to produce one or the other. I am mostly interested in whatever produces the best results. In the mean time, I will test this week with 4-band bmps. Thx for looking into/addressing the issue.

 

[arno]

Hi,

The latest development release that is online now should make the texture filter editor more robust. I am not sure if I have been able to fix all the errors you got, but I hope most of them are solved in this new release. Let me know what you still find.

 

[Clutch Cargo]

Oh, this looks interesting. Did a quick glance at it. Dare I say I see some machine learning with this update? I see a new 'Sample Points' tab and it looks like an updated manual on this section. But alas, no two preview panels? I will be diving into this on Monday. Thx!

 

[arno]

The machine learning is in there as well, but it's not fully finished yet, that's why the manual has no sample or other details on how to use it yet.

I haven't worked on the two preview images yet.