MSFS20 Everything about CGL generation [Custom DEM works]

[Breadeater]

We agree on the generic formula and that my writing can leave much room for reader interpretation and disagree on how the upscaling should be done, I think.

I am using pregenerated target LODs, each LOD made directly from source data. Just to waste some time -__-

Will try to explain the subdivision process better tomorrow.
Regarding testing the interpolation the game does, if you set your game terrain level of detail to the lowest, zoom in and move closer/further from the terrain, you can see very well the accumulation of interpolation error. Doing that allows allows using tile debug to measure altitudes of the actual vertices.

Did so in the above image some time ago, apologies for the readability
Circled values are from a base layer, not interpolated, other values are interpolated (once).
I hope you can read the 14.06something middle of the circled 15 values, when interpolated 5+ times, without adding corrective data in the delta layers, it will "dissolve" the mesh like in your first test image, so it will look like the filter is larger I think.

this 4x4 is annoying because you need the neighboring tiles to be able to interpolate at the edges.

With what I came up there is no need to use the neighboring tiles. But I will have to explain more.

It's probably what you mentioned above as being Lanczos. It might also just be bicubic?

Yes, that's what I meant, I think I saw something in a "corner step response" that did not fit bicubic, but I will have to double check. And as you said, it's really not the most important thing. "Nice to have" level of detail.

So: In order to go beyond Level 12 you're using 32 bit QuadKeys and something in the cgl header?

Yes, cgl header byte 0x1C should be 1 instead of 2, QuadKeys 32 bits long and what I designate in my cgl layout generation code as "levelchangevalue", should be
268435456, instead of 4096.

 

[mrbump]

Hi again
I doubt, that we both found different correct (or nearly correct) solutions. I think it's rather the same solution found out via different paths.
So let's see:
OpenCV documentation:
https://docs.opencv.org/3.4/d4/d1f/tutorial_pyramids.html shows the gaussian kernel used
https://docs.opencv.org/3.4/d4/d86/group__imgproc__filter.html#gada75b59bdaaca411ed6fee10085eb784 describes the pyrUp function

Now, when upsampling (x2) and filtering at the same time, 3/4 of the information are missing ("injected rows and columns of zeros" in the pyrUp function documentation). So to apply the gauss kernel, I differentiate the cases and accumulate only the values that are actually available:

The first case is really just a simple average of the four neighbours.
The next two cases (really the same case rotated by 90°) need to make use of the weights from the kernel (could be reduced to 1/6/1).
And well, I just realised, that I don't handle the last case according to the kernel (therefore in brackets). For vertices that exist in both LODs I don't filter at all.
As this would likely change the appearance and the result of my algorithm looks pretty much spot on I'm guessing, that's what Asobo does as well.

Thanks for the information about the header info. I'll give it a try soon.

 

[Breadeater]

It indeed seems that our filters produce exactly the same result

The linear interpolation I apply on the edges doesn't produce perfect results though. Good enough when used in real terrain I think as there is no gaps.

 

[theisomizer]

Unfortunately I have been slammed at work and have some catching up to do here

Thanks for the info about chunked LOD - I'll have to have a closer look at it. I'm not quite sure yet how minimal error downsampling ties in with elimination of unnecessary lods yet. So far I've seen it as two separate steps.

Green arrows are transitions/algorithms that I have in some form.
- Elimination of unnecessary lods is what I have planned in the "prune/optimize" step. Discard differential tiles with neglectable amplitudes starting at the leaves. The MSFS default offline data seems to do that as well, as many CGLs have less that 341 tiles (5 Lods: 1+4+16+64+256). I guess this would become more interesting once we can get to higher LODs. At the moment it's probably mostly used for large water surfaces.

Yes, these are orthogonal processes. However, I don't think the MS LOD scheme is as much about reducing unneccesary LODs due to minimal error, as much as it is about reducing file size and bus bandwidth/VRAM for this intermediate data. It seems like a consistent, conscious design decision to skip every other LOD in both raster and vector data. In terms of Chunked LOD or anything else based on an error metric, you would simply double your error metric per level.

It is such a pleasure to see someone citing their sources (and in the correct format too)!

I spend a lot of time in research papers / journals and have APA ingrained into me

Yet, for large parts of western switzerland (close to the french border) the CGL source is used for the lower LODs where closeup it switches to the LOD(s) streamed from the cloud.

Yes, going beyond lvl12 is possible, only needs minor changes to the code.

The quadkey format "2" (28 bit quadkeys) is indeed necessary. Additionally, an Asobo dev has mentioned in the dev forum regarding SAI tiles (imagery) that you must provide the entire pyramid to the highest LOD provided by Bing or Azure in an area in order for the sim to properly override in all circumstances. The only publicly accessible way to determine this without processing cm2(coverage map) tiles (WIP) is to inspect representative tiles in-sim using the tile debug tool.

I think I figured out how the game actually creates it's mesh based of the CGL data:

This looks correct to me as well. This is called elevation "residuals". It is very interesting to me that they are not using Chunked LOD as FSX/P3D, but that does require an offline computation step to optimize (performed by resample). With uniform residuals disk compression is efficient. When rendered more data is stored in VRAM and it uses more triangles, but hey triangles are relatively cheap these days, and getting even cheaper when LOD selection can be done in a mesh shader.

Currently, only replacing is possible. But as the format is known a tool could be made to convert contents of VEC+VECN cgl to a shapefile(s) for example, edited and converted back.

This is awesome! I have almost fully reversed the vecXXX.cgl files (except one 8 bit flags field on roads), see my other thread: https://www.fsdeveloper.com/forum/t...e-bgl-and-cgl-decompressor.433789/post-883045. I have put docs on the geojson output here: https://github.com/seanisom/flightsimlib/wiki/CGL-vecXXX-File-Format, but like you have run short on time for documentation and have not updated with the binary format yet. I will try to finish documenting that and post the source code this week. It seems we are very close to being able to roundtrip this!

TileAbs[x] = Upsample(TileAbs[x-1]) + TileDiff[x]

This makes sense, since it is using residuals. While taking the highest LOD and recursively applying the same Gaussian kernel up to the root is going to introduce a significant blur. However the method of starting from the root and applying a weighted incremental sampling to upsample higher LODs seems correct. It seems you've both made great progress in trying to reverse engineer the actual filter.

I would not be surprised if they referenced the proland code when building this, as it is the canonical way to do residuals. That similarly applies a uniform kernel while collapsing edges: http://proland.imag.fr/doc/proland-4.0/terrain/html/index.html#sec-residual However in terms of filters the MSFS binary has functions named "UpsampleDemBilinear" and "UpsampleDemBicubic" in the DEMDataMarker CGL residular parser. Do with that information what you will.

As this would likely change the appearance and the result of my algorithm looks pretty much spot on I'm guessing, that's what Asobo does as well.

Are you guys saying this matches how the levels are sampled in the default CGL files? Or just each other's implementations? I would be very curious if we can fully figure out the MSFS algorithm, as that opens the widest possibilities in the absence of an SDK!

 

[mrbump]

Are you guys saying this matches how the levels are sampled in the default CGL files?

I'm not quite sure what you mean. In order for a test pattern to look correct when loaded in the sim it must have been saved using the correct inverse process. So as the test patterns show up correctly I guess we can deduce that the algorithm is correct. What happens in the edge rows might need to be looked at more closely. I'd be very relieved if breadeater's assumption there is correct and we don't have to involve the neighboring tiles. It's quite likely considering that the FS probably just picks those tiles out of the files that are needed.

However in terms of filters the MSFS binary has functions named "UpsampleDemBilinear" and "UpsampleDemBicubic"

These might be used for the "in between" points. I have noticed lately that when quickly zooming about in my mesh, so that you can observe the loading of new LOD tiles, the mesh is initially displayed faceted and after a few seconds it changes to rounded.
BTW: The overshoots the bicubic filtering creates in rugged terrain actually turn out to be a major pain - they just look stupid on sharp edges ("sawtooth ridges" for instance). I noticed that the Asobo mesh in the french alps suffers from this effect less, but they are also going 2 LODs higher and there doesn't seem to be much added detail to show for it. So it might be that they just oversample and blur to prevent the overshoots. Alternatively the mesh could be optimized to compensate for the overshoots, but that would be silly, right? Well I guess blurring IS a form of compensating for subsequent sharpening.


Anyway, I completed a level 13 tile that is close to my heart (well, about 1.5m below it when I'm standing) as a showcase model without blurred LODs. It does still have quite a few issues but it nevertheless turned out well enough that I thought it would be a shame to keep it to myself. So I uploaded it to flightsim.to - hope that's ok with you guys.
(https://flightsim.to/file/15382/cgl-dem-switzerland)

 

[Sasa]

Hi Guys,

Seems a miracle to me this thread was idle for such a long time after what you achieved here!

Just downloaded the tools from Github. Very well documented and I got everything (including the dependencies all available/installeable via PIP) installed and the first test area running within a few hours.
Some observations:

• When using a Global Mapper Grid (GMG) as a source instead of GeoTIFF, the elevations in the preview window (and resulting DEM) are off heavily and fluctuate somewhere between 20.000 and -5000 Meters. Not sure exactly what goes wrong here as with Global Mapper the GMG is a lot easier to work with and saves disk space. After I converted the sources to GeoTIFF 32-bit everything worked fine.
• Not sure if this something that can be addressed at all: There is a noticeable transition between default DEM and custom DEM in my testing area:
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  

Looks much better from a distance than heightmaps (which seem to flatten mountain peak LODs to a degree when seen from far away).

 

[=Hollywood=]

Okay I'm starting to try and get up to speed with this, stuck and finding all dependencies and getting them installed, but most of that is due to being tired.

What I'd love to do is do up a video tutorial once I've learned the ropes, trying to work on a small island I visited as a child with awesome 1m dem that is DTM, seems for this country Asobo has used the 5m elevation data which is cool, and nice for most instances except there's a few sheer cliffs in certain areas hence my longing to finally see them in sim. Really wish that Asobo released a simple tool like we had for FSX.

It's a shame that the Nool tool is no longer available, thanks a lot to those who took advantage and didn't pay for a commercial license. I've been dreaming of this day for 32 years, and now I have the source data and not sure if I can get a DEM CGL generated. Fingers crossed.

EDIT: Finally up to speed *mostly* with all the above, so it seems that MSFS can render a max of approx 5m dem. I am curious about testing out my 1m data for a small island just to see if there's any difference visually at runtime as the local area is already 5m dem data from Asobo (totally surprising).

 

[=Hollywood=]

Basically (This is in Global Mapper)

1. Load elevation data (Workspace projection EPSG:4326)
2. Correct elevation data to egm2008

Okay so getting up to speed with this, egm2008 is the 2008 Earth Gravitational Model which it seems has an effect on the modeling of the elevation data in sim. Here's a video example of how it affects stuff.

https://www.reddit.com/r/MapPorn/comments/u1e7g6

EDIT:

I found a way to make this a bit easier for myself, the only step I haven't figured out a non-python method of step 2 for correcting elevation data for egm2008, but I think I briefly saw something earlier as I was researching. My steps so far based off your guide:

Basically (This is in Global Mapper)

1. Load elevation data (Workspace projection EPSG:4326) - Done
   
2. Correct elevation data to egm2008 - Still Learning / On Hold
   
3. Generate "centerline" that goes north to south - What I'm doing here is right clicking on the geotiff in the Overlay Control Center and choosing "BBOX/Coverages" and then choosing "YES - Create Rectangular Areas", then selecting the generated polygon in the main window and right clicking and selecting "Crop/Combine/Split Functions" and then selecting "Subdivide Quadrilateral Area" then under Grid choose "Number of Rows = 2" and "Number of Columns = 2", this will generate your top center and bottom center vertex points as well as a centoid vertex point (which should come in handy later for reverse engineering tests)
   
4. "Expand" that line to rectangle - just done in a different way in step 3 above, but can be done if you already have a center line by choosing the center line and right clicking and choosing "BUFFER - Create Buffers Around Selected Feature(s)...", "Number of Buffer Zones for Each Feature = 1" and checking "Only Create Rectangle Buffers for Each Edge Segment" and if you measured the horizontal distance of your bounding box (divided by 2) you set "Fixed Distance of = 1/2 of X meters"
   
5. Export area inside that rectangle to 256x256 to (float32) BIL - Here I export to Elevation Grid Data to GeoTiff Elevation 32bit floating point samples rand check generate TFW and PRJ ready to flip the data horizontally
   
6. Flip the data "left to right" - Open up the geotiff in photoshop and flip horizontal and save
   
7. Generate "space separated" list of elevation data - Next re-import the geotiff in GlobalMapper and "Export Elevation Grid Format" and select "Arc ASCII Grid" and ensure "Generate PRJ (Projection) File" is selected on (in case you need to open .arc grid again but regardless global mapper will ask for the projection if you miss it and select Geo WGS 84 on GM import if needed
   
8. Replace data in intermediate xml with that generated data - Open the *.asc file you just generated and copy ALL the numbers after the lines denoting "ncols, nrows, xllcorner, yllcorner, cellsize and nodata_value" and place that into your XML file (I'll do up a video tutorial for this whole process)
   
9. Replace coordinates with "centerline" start and end points - To do this I go to the center vertical line I generated earlier, to get the values export the vector line via "Export Vector Format" and choosing "Arc Unregenerate Lines" and it will save as a .DAT file which is a simple text file you can open in notepad or text editor. The top line has lon/lat for the top vertex, the next line has lon/lat for the bottom vertex. example: 146.15 -17.92 / 146.15 -18.01 (again I'll create a video to help show how this is done)
   
10. Replace width with rectangle width in meters. - From memory this should be the entire width of the entire horizontal distance of the bounding box rectangle (I need to verify, otherwise it's like stage 4 (bounding box width divided by 2)
    
11. Reload intermediate file in MSFS and check if it works - On Hold as I'm still re-installing MSFS and am unable to test

Now my next phase is to test if we need to manually split the bounding boxes into 256x256 squares or if we can supply the whole DEM as one chunk and the SDK subdivides the data. If it is the former I'll work on building shp file grids for each of the Bing quadrangles to make it easier to create the bounding boxes around data.

 

[FoxtrotScenery]

After many hours of tinkering, I would also like to add to this discussion. I've spent the past two days thinking of a similar method to what Paavo described in his thread of the MSFS toolkit. Thus I grabbed the packages for gdal for python and some more tools and got to work. I can currently import a EPSG:4326 image and convert it to a rectangles with heightmaps in a user specified resolution. It's still fairly slow (currently takes 2 minutes to cover .. square kilometres at 1m resolution on a fairly old mobile i7) and has no command line interface (only works from inside VS Code) or UI. However, it allowed me to take my custom terrain which was sculpted and baked to a DEM through Blender and QGIS into MSFS without much hassle.

Here's my method so if I have mistakes maybe some of you can spot them:
1) Load raster image using gdal and use its boundary and user specified resolution to determine the amount of needed rectangles with heightmaps of 256x256. The user specified resolution is compared to the average resolution per metre of the raster file to avoid oversampling and thus unnecessary waiting time. The final chosen resolution is the greater of the two. The image is than scaled accordingly so I get a raster which has sides of multiples of 256. This was done to make math easier at first, but a more modular approach is need for optimisation in the future.
2) The raster is converted into an array containing the elevation values of the source data.
3) This array is than corrected to EGM2008 geoid model
4) The final corrected elevation array is split into chunks of 256x256
5) The shapes of the rectangles is determined using the previously calculated amount of rectangles needed and the boundary coordinates of the input raster.
6) Finally, the XML file is generated. Currently I have to copy the resulting XML output into my project XML manually, but that's an easy fix.

This resulted in the following:

First the input mesh as sculpted in Blender:

The resulting DEM baked from the Blender model:

One of the rectangle as generated by the DEM Tools addon I am working on:

Hopefully I can work on this some more as it could be a very helpful tool once its checked to be working correctly as well as more optimised. Anyone who is more interested in the idea or if someone has some more data for me to test, shoot me a message or leave a reply here!

 

[B757200]

I want to do this for my island Aruba,

How would i go about finding or generating this for my small island?

 

[B757200]

The part which seems to trow me off is where to a put the country/island to generate?

 

[B757200]

Or perhaps someone and test it for my island and share it with us?

Aruba, caribbean

 

[superspud]

Before you do anything else you'll need to check if there's a good enough source for DEM data.

A lot of countries may only have elevation maps from general satellite passes that might be too low resolution for anything detailed.

A lot of them seem to be around the 30 metre mark which means entire hills might not be present. 10 metre or less would make a better result. For an entire island 5 metres or lower would probably be too much info. I would go down to 1 for a small area.

As for the CGL stuff, I've yet to see a guide that was remotely comprehensible to anyone who didn't already know a lot about it. Which isn't me.

 

[B757200]

Before you do anything else you'll need to check if there's a good enough source for DEM data.

A lot of countries may only have elevation maps from general satellite passes that might be too low resolution for anything detailed.

A lot of them seem to be around the 30 metre mark which means entire hills might not be present. 10 metre or less would make a better result. For an entire island 5 metres or lower would probably be too much info. I would go down to 1 for a small area.

As for the CGL stuff, I've yet to see a guide that was remotely comprehensible to anyone who didn't already know a lot about it. Which isn't me.

Thank you for your reply it helps a lot.

I have the IT knowhow to install the app but missing the knowhow to find the dem data is there a website i can see if Aruba is worth doing? Maybe it has bad data.

Cheers

 

[superspud]

Just search for - Aruba DEM data. Or digital elevation map or any equivalent.

The best free sources are almost always government sources. A lot of governments won't bother to make info like that available.

Some of the commercial stuff might be very good as well but can be thousands of dollars.

I've found most of the generalised world stuff to be so low res it's not really usable. Or you might be lucky and someone maintains something specific for the Carribbean.

 

[vezirgiannis]

I dont know if you can help me. I want to smash my computer!

I followed the tutorial from github but after running the
python3 elev_wgs84_egm2008.py

i getting errors

C:\OSGeo4W>python3 elev_wgs84_egm2008.py
C:\OSGeo4W\apps\Python37\lib\site-packages\osgeo\gdal.py:315: FutureWarning: Neither gdal.UseExceptions() nor gdal.DontUseExceptions() has been explicitly called. In GDAL 4.0, exceptions will be enabled by default.
warnings.warn(
ERROR 1: PROJ: vgridshift: could not find required grid(s).
ERROR 1: PROJ: pipeline: Pipeline: Bad step definition: proj=vgridshift (File not found or invalid)
ERROR 1: PROJ: vgridshift: could not find required grid(s).
ERROR 1: PROJ: pipeline: Pipeline: Bad step definition: proj=vgridshift (File not found or invalid)
ERROR 1: Too many points (441 out of 441) failed to transform, unable to compute output bounds.
0

Any help? Or another way how to convert dem.tif?

Thanks in advance!

 

[Bavarello]

I dont know if you can help me. I want to smash my computer!

I had the same problem the first time I tried to run this script... I left it there for a few years (so I understand you). Finally, I decided to come back to this again... having the positive result.
Let me start by saying that I am by no means a Python programming expert... I was helped by a competent friend.

C:\OSGeo4W\apps\Python37\lib\site-packages\osgeo\gdal.py:315: FutureWarning: Neither gdal.UseExceptions() nor gdal.DontUseExceptions() has been explicitly called. In GDAL 4.0, exceptions will be enabled by default.
warnings.warn(

This warning does not affect execution of the script. If desired, it can be eliminated with a change to the script.

ERROR 1: PROJ: vgridshift: could not find required grid(s).
ERROR 1: PROJ: pipeline: Pipeline: Bad step definition: proj=vgridshift (File not found or invalid)
ERROR 1: PROJ: vgridshift: could not find required grid(s).
ERROR 1: PROJ: pipeline: Pipeline: Bad step definition: proj=vgridshift (File not found or invalid)
ERROR 1: Too many points (441 out of 441) failed to transform, unable to compute output bounds.

This error is due to the lack of a specific library inside the Python folder.
I had to install some finally everything works.

0

this is nothing more than the (incorrect) count that the script should have made of the files modified in a given final folder.

Cheers

 

[trespassvr]

Hi All! I gave this a shot with some USGS 10m DEM data and only was able to get valid Level 6 data. Level 7 to 12 didn't seem to go well. It also doesn't seem to decompress with cgldec (as a way to check validity). Does anyone have any tips to get a CGL built? I'd like to pack in the dense data like in Italy and Switzerland examples. I've got it going fine with the heightmap tricks courtesy of @FoxtrotScenery -- Thanks!

 

[theisomizer]

Hey @trespassvr, can you DM me the file that won't decompress with cgldec? Thanks!

 

[Bavarello]

Dear forum members, I would like to share with you the topic I recently introduced in the MS/Asobo forum, regarding the priority between custom CGL files and Azure DEM data. After several attempts and investigations, I have successfully used a program to generate CGL files up to level 14, ensuring a resolution of 10m for my terrain mesh.

Experience and approach followed:

• I have configured the manifest.json file by including the package_order_hint parameter and indicating the DEM/CGL files in the globally_overriden_base_sim_files.
• Despite the configurations, the simulator does not seem to always recognize the custom files, probably due to the priority of the DEM data provided by Azure.

Considerations:

Working with CGL files offers a significant visual improvement for the terrain mesh, but presents significant challenges, such as:

1. Data compression and formatting;
2. Correct integration of CGL files into the simulator;
3. Creating higher levels of detail (LOD).

Thanks for your support and attention to this topic!

Creating custom CGLs for terrain mesh: experience and difficulties