• Which the release of FS2020 we see an explosition of activity on the forun and of course we are very happy to see this. But having all questions about FS2020 in one forum becomes a bit messy. So therefore we would like to ask you all to use the following guidelines when posting your questions:

    • Tag FS2020 specific questions with the MSFS2020 tag.
    • Questions about making 3D assets can be posted in the 3D asset design forum. Either post them in the subforum of the modelling tool you use or in the general forum if they are general.
    • Questions about aircraft design can be posted in the Aircraft design forum
    • Questions about airport design can be posted in the FS2020 airport design forum. Once airport development tools have been updated for FS2020 you can post tool speciifc questions in the subforums of those tools as well of course.
    • Questions about terrain design can be posted in the FS2020 terrain design forum.
    • Questions about SimConnect can be posted in the SimConnect forum.

    Any other question that is not specific to an aspect of development or tool can be posted in the General chat forum.

    By following these guidelines we make sure that the forums remain easy to read for everybody and also that the right people can find your post to answer it.

BGLDEC - A resample BGL (and CGL) DECompressor

Messages
144
Country
italy
Thanks Dick. I should have an update ready to go in about a week that can write vector data.
It's been a while since CGLdec (0.2.130.0) doesn't give any news.

In the meantime, there are some things I would like to share here. Wanting to create the stats.txt I encountered some difficulties, but in the end I managed to get a file of 2,335,417 lines.

I can't set in the parameter -o "M: \CGL" only accepts -o "M: \"
Immagine 2022-05-08 143510.jpg

Immagine 2022-05-08 141601.jpg





Another problem with the command
cgldec extract -f "H: \ MSFS_2020 \ Official \ OneStore \ fs-base-cgl \ CGL \ 032 \ dem010.cgl" -t dem -p 032010 -q 013113 -o "M: \"

there is no output.


Immagine 2022-05-08 160619.jpg
 

rhumbaflappy

Administrator
Staff member
Resource contributor
Messages
5,932
Country
us-wisconsin
I think the first problem is that you need to create the CGL folder in M: before you add anything to it.
The second problem is that -p 032010 -q 013113 doesn't exist. I think it would need -p 032010 -q 0131 so...

cgldec extract -f "H: \ MSFS_2020 \ Official \ OneStore \ fs-base-cgl \ CGL \ 032 \ dem010.cgl" -t dem -p 032010 -q 0131 -o "M: \"
 
Messages
284
Country
us-newyork
Hi Giovanni, Dick is correct on the above. The tool does not create the folder, make sure it exists. For your target file above, the max quadkey level is 10, you can see this by dumping all quads in that file:
Code:
cgldec decompress -f "C:\FS\MSFS\Official\Steam\fs-base-cgl" -t dem -p 032010 -o "C:\CGL"
As a convenience you don't need to specify the full filename when using the prefix, just the base CGL directory, so this is equivalent to Dick's command (with my paths):
Code:
cgldec extract -f "C:\FS\MSFS\Official\Steam\fs-base-cgl" -t dem -q 032010013113 -o "C:\CGL"
They have made a significant amount of changes to the sim internals in the past year, which can be discouraging, but a recent quick check shows that vector tiles are thankfully still compatible with only very minor binary differences.
It's been a while since CGLdec (0.2.130.0) doesn't give any news.
Time makes fools of us all. Unfortunately I don't have much news to share, as life has often gotten in the way, other than that writing vector tiles is in progress, and something that I very much still want to complete. It actually is a significantly more complex task than initially anticipated, however it is still possible.

For unrelated reasons, for anybody that uses bgldec, I have replaced the PTC compression library with a fully open source decoder which means flightsimlib is now platform independent and bgldec is almost stripped of closed code.

Hopefully I will have more to share on cgldec and MSFS soon.
 
Messages
284
Country
us-newyork
A quick progress update: Good news and bad news!

The good news is I can read at least building footprint geometry from arbitrary bldo/bldn tiles. These are called ".bld" files in Blackshark parlance, and the file format is an unbelievably convoluted proprietary mess.

Screenshot 2022-08-12 170410.png


The bad news: first, there is a lot of additional metadata there that is going to be even more convoluted to decode. I can perhaps get the code to a point where I can write some arbitrary bytes in some of these areas and observe the effect in-sim, but this is really going to be a slog. But first, there's an even more pressing problem:

Screenshot 2022-08-12 170446.png


The 16-bit fixed point grid is stretched/skewed over the tile bounds (and as you'd expect give the above, in longitude the tiles overlap, and have a gap in latitude). The amount of stretch/skew depends on Lat/Lon of the tile, but is not 0 at the equator as you would expect if this was related to projection distortion. I've tried countless projection techniques and I can get close, but have not found an algorithm yet that perfectly fits without manually tweaking a single tile, which obviously does not scale. This doesn't affect vec* tiles, which project fairly cleanly to web mercator - however the sim reads these directly instead of the PGG engine, two different codebases. To compound this, it's not really just a 16-bit grid, the algorithm selectively expands vertices to 32-bit at tiles bounds (for normalized values below 0.0, or greater than 1.0) to handle buildings with coordinates that overlap a tile boundary.

I'm going to keep investigating and hopefully figure something out. But I've been doing terrain programming for years, even decades now, and believe me I am stumped.

Anyways, thanks for your patience and cgldec updates to follow soon!
 
Messages
4
Country
china
A quick progress update: Good news and bad news!

The good news is I can read at least building footprint geometry from arbitrary bldo/bldn tiles. These are called ".bld" files in Blackshark parlance, and the file format is an unbelievably convoluted proprietary mess.

View attachment 83473

The bad news: first, there is a lot of additional metadata there that is going to be even more convoluted to decode. I can perhaps get the code to a point where I can write some arbitrary bytes in some of these areas and observe the effect in-sim, but this is really going to be a slog. But first, there's an even more pressing problem:

View attachment 83474

The 16-bit fixed point grid is stretched/skewed over the tile bounds (and as you'd expect give the above, in longitude the tiles overlap, and have a gap in latitude). The amount of stretch/skew depends on Lat/Lon of the tile, but is not 0 at the equator as you would expect if this was related to projection distortion. I've tried countless projection techniques and I can get close, but have not found an algorithm yet that perfectly fits without manually tweaking a single tile, which obviously does not scale. This doesn't affect vec* tiles, which project fairly cleanly to web mercator - however the sim reads these directly instead of the PGG engine, two different codebases. To compound this, it's not really just a 16-bit grid, the algorithm selectively expands vertices to 32-bit at tiles bounds (for normalized values below 0.0, or greater than 1.0) to handle buildings with coordinates that overlap a tile boundary.

I'm going to keep investigating and hopefully figure something out. But I've been doing terrain programming for years, even decades now, and believe me I am stumped.

Anyways, thanks for your patience and cgldec updates to follow soon!
Thanks for the update, you are still working on it. Can you open sourced the code and let us know how to figured this out?
 
Messages
859
Country
australia
A quick progress update: Good news and bad news!

The good news is I can read at least building footprint geometry from arbitrary bldo/bldn tiles. These are called ".bld" files in Blackshark parlance, and the file format is an unbelievably convoluted proprietary mess.

View attachment 83473

The bad news: first, there is a lot of additional metadata there that is going to be even more convoluted to decode. I can perhaps get the code to a point where I can write some arbitrary bytes in some of these areas and observe the effect in-sim, but this is really going to be a slog. But first, there's an even more pressing problem:

View attachment 83474

The 16-bit fixed point grid is stretched/skewed over the tile bounds (and as you'd expect give the above, in longitude the tiles overlap, and have a gap in latitude). The amount of stretch/skew depends on Lat/Lon of the tile, but is not 0 at the equator as you would expect if this was related to projection distortion. I've tried countless projection techniques and I can get close, but have not found an algorithm yet that perfectly fits without manually tweaking a single tile, which obviously does not scale. This doesn't affect vec* tiles, which project fairly cleanly to web mercator - however the sim reads these directly instead of the PGG engine, two different codebases. To compound this, it's not really just a 16-bit grid, the algorithm selectively expands vertices to 32-bit at tiles bounds (for normalized values below 0.0, or greater than 1.0) to handle buildings with coordinates that overlap a tile boundary.

I'm going to keep investigating and hopefully figure something out. But I've been doing terrain programming for years, even decades now, and believe me I am stumped.

Anyways, thanks for your patience and cgldec updates to follow soon!

Sean your work is incredible. I'm looking forward to being able to compare the building footprint data in sim, especially around monument valley to the stuff MSFT released on github, as there seemed to be a few false positives in the github data but not in the MSFS dataset.
Also curious if a neighborhood near me has the building footprints actually encoded as no autogen buildings show up in some locations. I noticed this at Rosamond Skypark, no buildings in sim but when I turned on dev mode to rework the airport the autogen buildings appeared.
 
Messages
859
Country
australia
Hey Sean, I was just digging through the files and for the landclass you're correct that they're WEBP format. It's RIFF is WEBPVP8.

You can open the data directly in Google Chrome as it uses the same file format as Google Maps. But you can also rename the .dat as .webp and Windows will generate thumbnails but the files can also be opened in Adobe Photoshop and any other software that can read .webp files.

So far I've seen the landclass classifications of:

(0,20,0) green
(0,50,0) green
(0,70,0) green
(0,100,0) green

on the few files I've opened. Would love to find a way to convert the quadkeys back up to the files so they can be loaded into GIS software.

Here's a screenshot of the landclass files:

landclass.jpg
 
Messages
10
Country
indonesia
Hello all...

First I must say thank you for this helpful utility.

I have few simple questions. I have tried to decompile default P3D landclass (worldlc.bgl) and some very old landclass addon to either .bin or .bmp and its corresponding .inf file. Can someone help me how can I convert those files to .raw file so I can append it to sbuilderx or edit it in Photoshop? I have no problems recompiling the output .bin and .inf files to bgl with resample, I just don't understand how to modify the landclass.

Thank you for your help.

Sorry for my bad english
 
Messages
284
Country
us-newyork
Hi @PK-LKP! If you rename the .bin file to .bil and use the .hdr file it spits out, you should be able load it in QGIS: https://www.qgistutorials.com/en/docs/open_bil_bip_bsq_files.html

Likewise, the file itself should be compatible with SbuilderX .raw if you rename it, although I haven't tried this workflow myself.

Does that meet your needs? I could explore adding Photoshop Large Document Support if not, but it is not a pleasant format to work with.
 
Messages
10
Country
indonesia
Hi @PK-LKP! If you rename the .bin file to .bil and use the .hdr file it spits out, you should be able load it in QGIS: https://www.qgistutorials.com/en/docs/open_bil_bip_bsq_files.html

Likewise, the file itself should be compatible with SbuilderX .raw if you rename it, although I haven't tried this workflow myself.

Does that meet your needs? I could explore adding Photoshop Large Document Support if not, but it is not a pleasant format to work with.
Thanks for the reply. I have tried renaming the output.bin to output.raw directly and opening it in photoshop, then inputted image width and height with numbers of nRows and nCols from output.inf file and it loaded perfectly! Now I just have to change the RGB value and recompile it. Thank you very much for your help.

PK-LKP

worldlc.PNG
 
Messages
10
Country
indonesia
Hi again!

I have one more question. Is it normal that extracted bmp from photo scenery has some color degradation (e.g. some dark area become completely black)?

Thanks.
 
Messages
284
Country
us-newyork
Hi @PK-LKP - what does the photo BGL look like in TMFViewer? The BGLDEC output should exactly match the image in TMFViewer - if that also is pure black it is as a result of the lossy image compression algorithm and you could try to rebuild as lossless or a smaller quantization step (CompressionQuality setting in the .inf). If the two do not match, it is a bug with BGLDEC - you can PM me the BGL and I can take a look.
 
Messages
10
Country
indonesia
Thanks again for the reply!

The bgl I'm trying to decompile is default TNCM island (/scenery/cities/StMaarten/StMaartenAerial.bgl). I have attached two images for comparison below.

Original bgl in tmfviewer:

tncm_bgl.PNG


Extracted (highest LOD) bmp in photoshop:

tncm_bmp.PNG


As we can see some of green and blue color became pure black. If I recompile it it looks similar to the bmp not the original bgl regardless compression quality value.
 
Messages
284
Country
us-newyork
Hi @PK-LKP, thanks for the report! I can reproduce with the TNCM bgl and it looks as above. It looks like this is a bug in the decompression algorithm where it is not reading a single band of coefficients. I will debug and fix this and post a new build.
 
Messages
284
Country
us-newyork
Bug fix version 0.9.9 is up in the top post. Please note this now requires the VS2022 redistributable which you likely already have installed if you are using MSFS or P3D6: https://aka.ms/vs/17/release/vc_redist.x64.exe

This is pretty embarrassing - I fixed this bug a while ago in flightsimlib and never posted a new version of BGLDEC here with that. This is an interesting one if you're interested in image compression- PTC encodes in a YCoCg color space for better compression performance. While the luma bitstream is always present, there are instances where either and/or both of the chroma bitplanes are empty. If that is the end of the data stream for the chunk, the code worked fine, but if the first was empty and there was subsequent HF data that was missed. This is a valid case and was not present in my test data set previously (it is now).

https://github.com/seanisom/flightsimlib/commit/6763027d63b2062b9ceea0526a198fc655e76c7f

I have some other fun things that I'm working on for bgldec 1.0, cgldec, and more but I don't want to hold this fix up for that and will share more soon.
 
Last edited:
Messages
10
Country
indonesia
Wow that's pretty quick response! Thank you very much sir! I will try it soon. I hope the fix will be useful for others too.
 
Top