Build 2624 Airport List Turning Up Empty

[scruffyduck]

New Build to test this problem

Download here:

http://www.scruffyduck.co.uk/files/sde_build_2634.zip

Things to note:

this build is intended to be used to test the scanner problem

Install it away from any existing SDE installs

Set you FS and BglComp Folders. Also set the units to make sure that you have a full settings.xml file.

Open the scanner and click Scan as before. Note that you will get a progress bar that should run twice. The first is the default scenery folder and the second is your addon scenery folder. After the second run of the progress bar it will close and the scan button will stay greyed for a while. This is when the tree of objects is being transferred to the window. It is slow.

Finally you will get the scan button back and the the tree (or not!).

Click on View Log. There is going to be a lot of stuff in there. Here is some information to help interpret it.

Note I have scanned an FS9 installation but that should not make any difference and you should see similar stuff for FSX

There are two section headers in caps that signify which folder is being scanned. An entry like this

D:\Program Files\Microsoft Games\FS9\scenery\Same\scenery\ap923260.BGL

Decompiler Output...

. 34472 646 10 ... NameList
Name List Extracted

signifies that a file has airports in it and they have been extracted

An entry like this:

D:\Program Files\Microsoft Games\FS9\addon scenery\static objects library\scenery\ez-extra-objects-1.bgl

Decompiler Output..

signifies a file that has no airport data in it (should only turn up after SDE starts scanning the Addon Folder

An entry like this:

D:\Program Files\Microsoft Games\FS9\addon scenery\LF2 - Madisonville\scenery\LF2_VTPL.BGL

Decompiler Output...

BGL ID 1 not correct

signifies that SDE found a bgl that is not BglComp based (should only turn up in the Addon Folder section and is related to FS9 files)

At the end you should see something like this

Processed 235 files

Building Tree with 23769 entries

Tree Built

SDE processed 235 files in the Addon Scenery Folder Tree and had 23769 airports to put into the tree. The tree got built!

You numbers may be different.

If you see anything else the it might signify the problem Also if the run stops short somewhere it might show us the file (or maybe the one before it) that is the problem

 

[Eddie]

Hi Jon,

I did everything that you wrote in the correct order and------


It worked.

Have the tree with all the airports listed.

Great job and I didn't have any other problems.

Thanks,

Ed



Now this is weird, I just tried the other version and that now works correctly.

 

[Eddie]

Update.

Before, I forgot to add back the addon scenery folder. After I added the folder back into FSX, the tree would not show as before. Removing the addon folder and your program would work.

I narrowed the problem to Justin's Genesis mesh. And now what I am getting is a "out of memory" error.

Could it be that because his files are so large they're using up all the memory?


Ed

 

[Gypsy Baron]

Results....favorable!

Update.

Before, I forgot to add back the addon scenery folder. After I added the folder back into FSX, the tree would not show as before. Removing the addon folder and your program would work.

I narrowed the problem to Justin's Genesis mesh. And now what I am getting is a "out of memory" error.

Could it be that because his files are so large they're using up all the memory?


Ed

To add to Eddies results, I found that I had to remove my
USA_10M_Mesh folder, a 48-BGL folder, 2.31 GB in size.

With that folder installed I got "Out of Memory" errors with
the debug build of 2364.

I monitored memory usage as the scans progressed.
After the "Scenery" folder scan, memory use had peaked
and stood at about 235MB. When the Add On Scenery
folder scan began, memory usuage began to climb rapidly
The scan seems to go from "bottom up" alphabetically and
the USA_10M_Mesh folder was the 2nd or 3rd one scanned.
When memory usage reached about 1.365 GB, I got the error.

Removing the USA Mesh folder resulted in a good scan
and completed airports list. Surprisingly, the memory
usage peaked at a level HIGHER than the error scan! ( ????? )

With the USA Mesh folder still removed, I used the "real"
2364 install and, indeed, the scan completed with a full
airports list!

The out of memory, or I should say, memory usage seems
to be tied to the number of "BGL ID 1 not correct" events
encountered. I have alot of FS9 BGL's, many of which are of
"the old style"...sca's compiled to BGL's, plus many AFCAD2
files that I haven't yet run through SDE to get the XML code to
recompile.
In any case, we have found a "smoking gun" . Now for a
"silver bullet" and we'll be set

My solution now is to move ALL my mesh folders into my
"Addon Scenery 2" folder which resides in a different
directory, outside the main FSX folder. That way SDE won't see
these folders.
Also, I noted, in my first attempt to "hide" 3 folders from
SDE I just created a "SAVE" folder in the Addon Scenery folder
and then moved my "USA_10M_Mesh folder and two other
scewnery folders into the "SAVE" folder.
SDE scanned the "SAVE" folder and the folders within,
finding the "scenery" folders there...3 levels down!

I have a habit, when debugging, of creating these "save" folders
or "disabled" folders in the same folder that I'm working
with. I future I shall do a better job of "hiding" them from
the outside world

I'm attacking a ZIP file that contains the log outputs of three
scans. One "good" and two "Out of Memory" scans.
Hopefully they will contain all the info you need to further
analyze what is happening.

Thanks so much, Jon, for taking the time to create this
"special" debug version.

Paul

 

[scruffyduck]

Update.

Before, I forgot to add back the addon scenery folder. After I added the folder back into FSX, the tree would not show as before. Removing the addon folder and your program would work.

I narrowed the problem to Justin's Genesis mesh. And now what I am getting is a "out of memory" error.

Could it be that because his files are so large they're using up all the memory?


Ed

It's possible that memory is the problem - can you use task manager to see what memory is being used as the scanner runs?

Do things work if you temporarily move those files?

Is the scanner log reporting the out of memory error ? if so can you post that bit of the log?

At the moment SDE grabs the whole file before deciding what to do with it - I will think about just reading the header before deciding whether to get a file or not.

 

[Eddie]

This is an application error.

If I remove the files, the program runs fine.

The program stops responding and there is no error in the log file.

I have been trying to attach a screen shot of the error but I can't get it to work.

Here is the url for the screenshot.

http://www.fsdeveloper.com/forum/ne...01&posthash=72785cd57314906cc590e57622a602f8#


If this doesn't work , let me know and I will email you the attachment.

Ed

 

[Eddie]

Jon,

Check your email.

Thanks

 

[scruffyduck]

Eddie got the mail and replied. This looks like SDE has tried to read a file that is too big for the memory available. The scanner is also still leaking memory in the version you have. I have changed the code so that the program will not try to load bgl files that are not compiled by BglComp.

If we continue to get problems with large files I will need to look at another way of reading them.

How big is the file that is giving the problem?

 

[Eddie]

The file where the program stops is 219,832,587.

This is the first large file where I get the error message.

Ed

 

[José]

Extra FSX Programs

Hi, All

For those that are getting errors with SDE scan, caused by others programs installed, I suggest to move to a folder outside FSX root, and re-activate.

My MeshX is installed outside FSX root, so there is no problem and I do not need it to be read by SDE.

Regards,

José

 

[scruffyduck]

Hi Jose

That is certainly a good way around the problem. I am also hoping that some of the changes with the engine will avoid reading these big mesh files. I am looking at ways to read only the first section and then decide whether to load the lot. Seems to me that a 220Mb file is too big in any case


Ed

Does the scan work without these big files in place?

 

[Eddie]

Yes, the scan works fine up until it hits a file starting about 218,000,000.

I have tested most of Justin's files and anyting smaller then about 215,000,000 are read fine. As soon as the program starts reading a larger file, the memory shoots up until I get the error message.


Hope this helps,

Ed

 

[scruffyduck]

Thanks Ed

I think we can be safe in assuming that the problem you and Paul have is due to trying to parse very large files. I will change the code to try and avoid this happening