BGLDEC - A resample BGL DECompressor

[theisomizer]

BGLDEC - A BGL DECompressor for resampled BGL files

BGLDEC is a utility that decompresses BGL files that are output from resample.exe.
A BGL file is decompressed into multiple binary or bitmap files that contain the raw data as it used by the simulator.
This includes elevation data, terrain photos, landclass, waterclass, season, region, and population density data.

New Version: BETA 0.9.4 -
* Better INF Writing - Still not complete
* Better performance in some cases with large BGLs
* More decompression bugs fixed
* New! MS Flight Support! TRQ2 is basically the same format, but with 32bpp support and detail mask support for orthophotos.

Version: Beta 0.9.3
* Fixed! Numerous bug fixes - mostly memory management, blend mask, and decompression algorithm improvements. Several WIP features in preparation for 1.0.
* Changed! BMP mode now does colorization for additional types.

Version: Beta 0.9.2
* Fixed! -STITCH options on 257x257 chunks now only output (cols|rows*256)+1 vs cols|rows*257 pixels. This is equivalent to PixelIsPoint=1 in resample .infs.
* Changed! Rewrote algorithm to stream output in-place one row at a time. This uses far less memory and it won't crash on very high-res files.
* Changed! BMPs on Windows have a 2GB limit, if it detects a file will go over this it will warn and output that level as .bin (RAW) instead.

Version: Beta 0.9.1
* Fixed! Bug with reading input segments multiple times for photo subsections, resulting in much higher memory usage in certain cases

Version: Beta 0.9.0
* New! Added -STITCH option to the existing output types (BMP-STITCH, BMPMASK-STITCH, BIN-STITCH), which combines all tiles for a level into a single image
* Fixed! Error in PTC skirt for elevation types - all rows were erroneously shifted by one pixel upwards when converting 256X256 to 257x257
* Fixed! Minor Error in PTC that manifested as missing water mask on photo chunks
* Fixed! Much more conservative memory management - only decompresses chunks when needed - much better performance on large files
* Changed! Non-blended (regular BMP) photo tiles are now output as native 16 bit bitmaps instead of converting to 24 - smaller files, much faster
* Changed! No longer output INDEX files - these are not useful for anything other than the quadtree in the terrain engine
* Changed! BGLstats file is no longer output - any tiles that are used for multiple variations will have those concatenated into the filename
* Changed! "Missing Data Mask" is really a blend mask, these have been renamed <TileName>-MASK.bin


Please read the attached .txt file!

No longer dependent on any sim runtime. Can be used irrespective of sim version.

This is a beta version, download as an attachment below.

Make sure you have installed the VC++2017 runtime (x86 version): https://go.microsoft.com/fwlink/?LinkId=746571

Please let me know of any issues you experience in this thread.

 

[theisomizer]

Updated to version 0.2.1

It can now decompress all raster sections that are processed by resample.exe including landclass, regions, imagery, etc.

I have also fixed several issues with the decompression itself as well as reduced memory usage to support more complex BGL files.

Also, all files are now output by QMID so they can be meaningfully processed. The file names are in Level-U-V.bin format. (Remember QMID levels are LOD + 2).

 

[nz255]

Wow that's pretty amazing, watching with interest...although I don't know what I'd use it for

 

[theisomizer]

Thanks! New version 0.4.2 has been posted. This has better performance, support for variations and alpha masks, and can write aerial imagery chunks to human-readable bitmaps.

Now I'm focusing on bglstitch, a tool to merge various bin files into a single .raw + inf file.

Remember to run BGLDEC from your root sim folder (where all of the .dll files are) if you are decompressing aerial imagery or elevation data! The other types will work without this.

 

[fs1]

Great job.

I was looking into this as well and came out with a similar solution - PTC to raster. I am wondering how useful a tool like this could be for the developer's community?

Any insights would be great to hear.

 

[dave hoeffgen]

I just tried this tool, sadly the only thing it does is crashing immediately.

I placed it in my root fsx folder and gave it some dlls it was asking for.

Am I doing anything wrong?

Edit: I tried from inside P3Dv3 aswell and also in xp compatibiltiy mode. No change

 

[denali]

I think you forgot to compile this in release mode.

I've PMed you about this project and I'm really looking forward to hearing from you.

 

[denali]

I keep getting a missing c runtime debug dll missing error. I can't seem to fix it just by getting the dll and putting it into system32.

 

[arno]

Interesting thread Sean. I'll check out this new tool. Could open some possibilities for scenProc, like sample roof colours from the photo bgl.

 

[Golf-HotelDelta]

It created 8 256x256 pixel very low resolution bmps before crashing.

Initial bgl in TmfViewer:

Zoomed in:

Output from Bgldec:

 

[arno]

For the missing DLL files, it seems the tool was compiled in debug mode, since it is referencing the debug version of the C++ runtime files. I think these are in general only installed if you have Visual Studio installed. Sean, maybe you can make a release build?

 

[denali]

interesting. I have visual studio installed on a virtual machine. Maybe I should try it there.

 

[denali]

could it be that it resample uses compressed bitmaps?

 

[denali]

Actually, it really doesn't matter what resample uses; It matters what FSX/P3D use. If they can be read in the FS then it may open up the possibility of more flexibility in output of a redone resample.

 

[arno]

I'm not sure I understand your last two posts. This tool is a decompressor for the terrain bgl format. So it's a reverse resample and in that way it isn't relevant if FS can read it, it makes output for us as developer.

 

[theisomizer]

Oh boy! Glad people are using it.

I've been offline and not actively working on this since September. I moved across the country again for work, second time in 2 years! Enjoying New York City now and things are finally calming down enough now that I am able to get back to work on this.

Arno, and Denali, you are correct in that I need a release build that is not dependent on the VC++ libs, complete oversight on my part. I'll get that done ASAP - that should fix a lot of errors for everybody. Also I will uplift to use the DLL's from P3D 3.1. Every time they release a new build the offsets change.

I am still working on removing the dependency on the simulator runtime. The, ummm, let's call it "re-engineering" of the PTC functions ended up being a much more difficult task than I expected. There were a lot of compiler optimizations turned on, making my job that much more difficult.

 

[arno]

Hi,

Happy to hear you are still working on it. I can understand that moving across the country gives other priorities for a while

Since you say there is a dependency on the FS build, would we need different versions for FSX and P3D?

 

[theisomizer]

Yes, it would need new versions. It would actually need the offsets from every build of the sim, as every recompile of the sim moves things around. So FSXA, FSXSP2, P3D 2.5, 3.0, 3.1 etc would all have different offsets. The last version posted here was built against P3D 2.5 hotfix 946.

Unfortunately, it gets more complicated than this, and this is going to be more difficult than I thought. I've spent the past couple days seeing what is involved in uplifting this to P3D 3.1, and it would appear that in the v3.x branch they have refactored some of the code I was hooking into out of the executable. Obviously there is still support for decompressing PTC in there, but is refactored to use a slightly different data structure and will take some work. Also it was compiled with VS2013 instead of VS2010, further changing the output.

I could go ahead and post a release build here, but it will only work with the P3D 2.5 libraries for any file containing PTC-compressed sections (most elevation and aerial imagery). Thoughts on whether this would be useful to anybody?

This is why I had been focusing on finishing on implementing my own PTC decompression methods before I paused working on this. Once that is done, there will be no dependency on the simulator at all, and bgldec can be run in complete isolation. However, this PTC work may very well be the most complex project I have ever taken on.

Let me do some groundwork over the next few days and see how involved all of this will be to get this working for 3.x.

 

[arno]

Hi,

I agree with you, this doesn't seem very practical, especially if every build is different. So then you would end up with a dozen of versions of the tool or so.

 

[theisomizer]

New Version 0.5.1 has been posted.

BGLDEC has been updated to support the P3D v3.1 runtime. I was able to overcome the technical challenges mentioned above. Some other minor improvements and fixes are included as well, see the attached user manual.

I am still working on my PTC implementation, hopefully I can get this working soon. I am fighting some wicked bugs with it. The headers are working but the second I hit the first band the stack corrupts. Assembly language is nasty stuff, I am very thankful I do C# and node.js at my day job.

To Arno's point, this is still not an optimal solution, but hopefully it is usable for most until I can make more progress. I'll keep you guys posted.