BGLDEC - A resample BGL DECompressor

#63
Hmm, looks like it is running out of memory. How big is your file? 3769 subsections is a lot.

I'll look into making how I read the file in more memory efficient, there are optimizations that can be performed.
 

rhumbaflappy

Moderator
Staff member
Resource contributor
#65
Most users of your decompiler will just want the highest QMID level of the BGL. A "-t" switch would be nice to specify the top level decompiled and stitched only, or perhaps this could be the default setting. That should save some memory.
 
#66
Thanks Dick, noted. I am planning on revamping the CLI for a 1.0 release and have added this to the list. This won't save memory the way things are structured, but would run much faster.

The memory management is getting messy due to the size of some of these files, I'm having to come up with some creative solutions as I'm sure the MSFS authors had to do as well. Examining TMFViewer, it cheats a little bit. It scans the file for the TRQ1 pointers, and just caches these QMIDs in memory along with the locations in the bgl file. When a paint call is made to GDI, it then reopens the input file, seeks, and decompresses each chunk on the fly based on the target QMID for the screen space. The "cheat" is that it can examine the screen space error for the next highest LOD, and if it is imperceptible it can render at a lower LOD. When zoomed in very far it can render at full resolution, but only needing to decompress a small subset of the tiles. If GDI forces a repaint, it will repeat the full draw, holding no data in application memory. You can observe this by panning a large distance with the mouse on a high LOD and watching the screen "blip" as it repaints. This is very similar to how the FSX terrain engine works and the programs definitely share some code. When writing the full bitmaps out to single files we don't get this luxury.

For now my algorithm reads all of the compressed data into memory, and one row at a time, decompresses, writes to an in-memory output buffer for the whole file. When it is done with each row it throws away the uncompressed. But the whole raw output remains in memory before it is saved to disk. I could pretty easily change this to where it streams one row to disk at a time which would be slower to write but should make for much better memory usage.

For example, when decompressing the default kedw_photo.bgl, the highest QMID level would result in a 2GB bmp file, requiring 4GB+ of memory the way things are currently structured, resulting in a crash. Implementing the above change would knock this to under 100MB.
 
Last edited:

WebSimConnect

Resource contributor
#70
if you could output one band (height in meters) in geotiff format, your tool would be great and if just could specify geo rectangle (north, west, east south) that would go to all relevant files and produce one output file, that tool would be simply perfect !!!
 

WebSimConnect

Resource contributor
#72
staring point : https://trac.osgeo.org/geotiff

and I guess open source libtiff library shall be just for that task. I've never used it though, as GDAL tools and python are normally sufficient for me, anyway it does not look like rocket science:

Code:
/*
 * makegeo.c -- example client code for LIBGEO geographic
 *     TIFF tag support.
 *
 *  Author: Niles D. Ritter
 *
 * Revision History:
 *   31 October, 1995    Fixed reversed lat-long coordinates   NDR
 *
 */

#include "geotiffio.h"
#include "xtiffio.h"
#include <stdlib.h>
#include <string.h>

void SetUpTIFFDirectory(TIFF *tif);
void SetUpGeoKeys(GTIF *gtif);
void WriteImage(TIFF *tif);

#define WIDTH 20L
#define HEIGHT 20L

int main()
{
    char *fname = "newgeo.tif";
    TIFF *tif;  /* TIFF-level descriptor */
    GTIF *gtif; /* GeoKey-level descriptor */
  
    tif=XTIFFOpen(fname,"w");
    if (!tif) goto failure;
  
    gtif = GTIFNew(tif);
    if (!gtif)
    {
        printf("failed in GTIFNew\n");
        goto failure;
    }
  
    SetUpTIFFDirectory(tif);
    SetUpGeoKeys(gtif);
    WriteImage(tif);
  
    GTIFWriteKeys(gtif);
    GTIFFree(gtif);
    XTIFFClose(tif);
    return 0;
  
failure:
    printf("failure in makegeo\n");
    if (tif) TIFFClose(tif);
    if (gtif) GTIFFree(gtif);
    return -1;
}


void SetUpTIFFDirectory(TIFF *tif)
{
    double tiepoints[6]={0,0,0,130.0,32.0,0.0};
    double pixscale[3]={1,1,0};
  
    TIFFSetField(tif,TIFFTAG_IMAGEWIDTH,    WIDTH);
    TIFFSetField(tif,TIFFTAG_IMAGELENGTH,   HEIGHT);
    TIFFSetField(tif,TIFFTAG_COMPRESSION,   COMPRESSION_NONE);
    TIFFSetField(tif,TIFFTAG_PHOTOMETRIC,   PHOTOMETRIC_MINISBLACK);
    TIFFSetField(tif,TIFFTAG_PLANARCONFIG,  PLANARCONFIG_CONTIG);
    TIFFSetField(tif,TIFFTAG_BITSPERSAMPLE, 8);
    TIFFSetField(tif,TIFFTAG_ROWSPERSTRIP,  20L);
  
    TIFFSetField(tif,TIFFTAG_GEOTIEPOINTS, 6,tiepoints);
    TIFFSetField(tif,TIFFTAG_GEOPIXELSCALE, 3,pixscale);
}

void SetUpGeoKeys(GTIF *gtif)
{
    GTIFKeySet(gtif, GTModelTypeGeoKey, TYPE_SHORT, 1, ModelGeographic);
    GTIFKeySet(gtif, GTRasterTypeGeoKey, TYPE_SHORT, 1, RasterPixelIsArea);
    GTIFKeySet(gtif, GTCitationGeoKey, TYPE_ASCII, 0, "Just An Example");
    GTIFKeySet(gtif, GeographicTypeGeoKey, TYPE_SHORT,  1, KvUserDefined);
    GTIFKeySet(gtif, GeogCitationGeoKey, TYPE_ASCII, 0, "Everest Ellipsoid Used.");
    GTIFKeySet(gtif, GeogAngularUnitsGeoKey, TYPE_SHORT,  1, Angular_Degree);
    GTIFKeySet(gtif, GeogLinearUnitsGeoKey, TYPE_SHORT,  1, Linear_Meter);
    GTIFKeySet(gtif, GeogGeodeticDatumGeoKey, TYPE_SHORT,     1, KvUserDefined);
    GTIFKeySet(gtif, GeogEllipsoidGeoKey, TYPE_SHORT,     1, Ellipse_Everest_1830_1967_Definition);
    GTIFKeySet(gtif, GeogSemiMajorAxisGeoKey, TYPE_DOUBLE, 1, (double)6377298.556);
    GTIFKeySet(gtif, GeogInvFlatteningGeoKey, TYPE_DOUBLE, 1, (double)300.8017);
}

void WriteImage(TIFF *tif)
{
    int i;
    char buffer[WIDTH];
  
    memset(buffer,0,(size_t)WIDTH);
    for (i=0;i<HEIGHT;i++)
        if (!TIFFWriteScanline(tif, buffer, i, 0))
            TIFFError("WriteImage","failure in WriteScanline\n");
}
 
Last edited:
#76
It's a command line program, like resample. Open a command prompt (Windows key + r, type cmd, press enter). In that window type cd "foldername", where foldername is the folder that holds bgldec. For me that is "C:\Users\sean\Desktop\BGL".

From there you would type the command in the post above, for example:
Code:
bgldec AA_Jordan_066.bgl BMPMASK-STITCH jordan
Hope that helps
 
#77
Version 0.9.2 is up! This is a minor release.

I rewrote the STITCH code, it now outputs the rows in place using the algorithm I previously described. This reduces memory usage by a factor of up to 20x on very large files, so it won't run out of memory. If it surpasses the 2GB limit for a BMP file it will write it as a raw .bin instead.

Also there is a sizing / overlap issue that I fixed with stitch and PixelIsPoint types (257 pixel types here: http://www.fsdeveloper.com/wiki/index.php?title=BGL_File_Format#TRQ1_Record). The easy way to explain this is to open up dem4km.bgl in TMFViewer -> View LOD 3 and set QMID Grid to QMID 13. Note how the pixels are centered on the QMID grid instead of filling it in like photoscenery, that is PixelIsPoint. For these types of tiles the 257th row and column overlaps any touching tiles in the sim. If there is a neighbor north or west it wins and overlays that row, same algorithm as in the sim.
 

rhumbaflappy

Moderator
Staff member
Resource contributor
#79
Hi Sean. New version works good for me. I see for FSW, ORBX has changed something for the worldlc.bgl. The rest of FSW seems to be just FSX files. Don't know if you are interested in FSW...
 
#80
@See Michel - could you PM me with a Dropbox/Onedrive(or equivalent) link to the BGL that is giving you trouble, so I can debug on my end? I have no known issues at the moment, would like to patch it up if I can find the issue.

@rhumbaflappy - That is great to hear Dick! I do want to keep compatibility with all FSX-derived sims, and I was hoping that would be the case. Flight seems to be the only system that has changed the format (32bpp for photoscenery, and additions to worldlc). EDIT - will investigate the differences.

Would folks be interested in adding lclookup format support to this? It is a simple format, I decoded it a long time ago. It can be a little confusing to those that haven't worked with Landclass authoring before. @HolgerSandmann has a spreadsheet out there that does a wonderful job of explaining it.
 
Last edited:
Top