• Which the release of FS2020 we see an explosition of activity on the forun and of course we are very happy to see this. But having all questions about FS2020 in one forum becomes a bit messy. So therefore we would like to ask you all to use the following guidelines when posting your questions:

    • Tag FS2020 specific questions with the MSFS2020 tag.
    • Questions about making 3D assets can be posted in the 3D asset design forum. Either post them in the subforum of the modelling tool you use or in the general forum if they are general.
    • Questions about aircraft design can be posted in the Aircraft design forum
    • Questions about airport design can be posted in the FS2020 airport design forum. Once airport development tools have been updated for FS2020 you can post tool speciifc questions in the subforums of those tools as well of course.
    • Questions about terrain design can be posted in the FS2020 terrain design forum.
    • Questions about SimConnect can be posted in the SimConnect forum.

    Any other question that is not specific to an aspect of development or tool can be posted in the General chat forum.

    By following these guidelines we make sure that the forums remain easy to read for everybody and also that the right people can find your post to answer it.

We NEEEED a multithreaded resample!!! ARRRGHHH!!!

RicherSims

Resource contributor
Messages
566
Country
dominica
Complete shot in the dark... but how hard it would it be to create a tool to somehow magically enable multiple resample instances to run simultaneously and compile into 1 BGL?

Or an even longer shot in the dark, to somehow get the source code for resample from Lockheed developers and rewrite it so that is multithreaded?

Or even fuuurther... start from scratch, figure out how resample does what it does and create an entire new and improved multithreaded resample 2.0?

I'm currently sitting at my PC watching 3 instances of resample run on an 8 core processor and only 39% of the total avaialble processing power is being used. One of them sits at 5% after 2 and a half hours and has a further 46 hours to go... **pulls hair out**
I'm running 3 of them just so that I can experiment with different terrain nightmaps, and, in the event that the one which finishes first produces unsatisfactory results, I can cancel the other 2 and start over. There has GOT to be a way to speed things up for large scale outputs. (Yes I know I can work on a smaller area first and then go full scale later, but i'd still have to wait for 2 straight days eventually!)
 
Messages
7,450
Country
us-illinois
Hi MOUSY:

I agree that it would be helpful to have a faster-working version of FS / P3D SDK Resample.


Until we have one, it merits repeating a caveat regarding avoidance of LZW compression in Resample TIF source files:

http://www.fsdeveloper.com/forum/threads/blend-water-masks.435509/#post-726137


PS: I edited today another of my posts regarding Resample (one from 2011) that did not yet mention this caveat. :pushpin:


Hope this info helps anyone still using LZW Compression in aerial imagery and/or elevation data Resample TIF source files. :)

GaryGB
 

RicherSims

Resource contributor
Messages
566
Country
dominica
Wow. Gary I was TOTALLY unaware that LZW compression TRIPLES (TRIPLES!!!) compile time! Thanks for that huge tip!
(I must also remember to work off my SSD, but its already so cramped!)

Now back to the original topic. Does anyone have any ideas on how we can go forward with this. I would put all my programming know how into this if I learnt there was a way.
 
Messages
1,518
Country
unitedstates
I don't have an answer for you but what could you possibly be compiling that would take 46 hours!? An entire country into one bgl :p. I usually perform a batch run of 4-5 resamples running at once on my I-7 CPU at 90%. I can create 25 bgls in around 20 minutes covering around 50 square miles (about 130 square kilometers).
 

RicherSims

Resource contributor
Messages
566
Country
dominica
An entire country into 1 BGL is correct.
4 TIFS for 3 different seasons and a night map, each covering about 1200 square kilometers at 0.5m/px, each LZW compressed to 3.4GB (would otherwise be 11GB BigTiffs and thus unusable by resample).
The resulting BGL is 400MBs at 85% compression.

I'm considering using multiple BGLs instead of a single large one, however I have not yet fully investigated the effects on VAS usage in SIM.
 

GHD

Messages
12,243
Country
england
Hope this info helps anyone still using LZW Compression in aerial imagery and/or elevation data Resample TIF source files. :)
I use LZW compression for all water and blend masks and have no problems.
 
Messages
7,450
Country
us-illinois
Thanks for sharing that practical insight, George; apparently 1-Bit B+W and/or 8-Bit Gray Scale TIF Terrain Mask files using LZW compression do not take as much time to process as 16-Bit, 24-Bit, or 32-Bit "color" source files. :)

[EDITED]

NOTE: Final output source file format for TIFF Masks must actually be 8-Bit gray-scale; see:

http://www.fsdeveloper.com/forum/th...-scenery-achievable.440912/page-4#post-789447

[END_EDIT]


Indeed, I have edited my suggestions on "preferred" settings to use with TIF source files for aerial imagery and terrain mesh on the assumption, IMHO, that most would-be users may want to forego LZW compression in favor of saving processing time.

However, those with special requirements involving very large data sets wishing to conserve USERVA in a FS / P3D SDK Resample command mode Windows task session, and/or those having a goal of minimizing size or number of output BGLs, certainly might want to choose the (non-lossy) LZW compression option to preserve data fidelity, rather than utilizing the (always lossy) compression options offered by Resample itself ...with the understanding that Resample processing of all source files may take upwards of 3 times longer. :idea:


One might also wonder whether another option might be faster processing by Resample of source files located within folder chains compressed by Windows NTFS folder compression (...which at one time was promoted by MS due to reportedly minimal impact on system throughput speed due to its background use of 'unused' CPU clock cycles on 1 GHz and faster CPUs) ? :scratchch


Regardless of the processing scenario, one may also speed things up by setting the FS / P3D SDK Resample command mode CPU task priority in Windows Task Manager to "Realtime". ;)


GaryGB
 
Last edited:

arno

Administrator
Staff member
FSDevConf team
Resource contributor
Messages
32,858
Country
netherlands
Hi,

I doubt we can make a new resample. The file format is not fully known.

What I do is run multiple resample instances in parallel. And I'm planning to stretch that to control multiple instances on multiple machines controlled from one place.

I'm actually running a job now, I'll post the performance here when it's done.
 

arno

Administrator
Staff member
FSDevConf team
Resource contributor
Messages
32,858
Country
netherlands
Hi,

Just for your information, here are the stats of my last run. It's an area of 2500 kmsq covered by LOD17 photo scenery (25 cm input imagery). It took 12 hours and 30 minutes to generate the whole area. The BGL files are divided by LOD11 (typically around 300 MB in size).

I was running 4 resamples in parallel for this, on my build machine with 8 GB of RAM and an i5 processor. Most of the time is running resample, but generating the watermasks from vector data is part of the process as well.
 

RicherSims

Resource contributor
Messages
566
Country
dominica
Good info Arno.
Any particular reason for splitting the BGLs into LOD11 areas?
And were your TIFFs uncompressed?
 

arno

Administrator
Staff member
FSDevConf team
Resource contributor
Messages
32,858
Country
netherlands
Yes, the input files are uncompressed GeoTIFF files.

I guess I might get away with LOD10 for the BGL size as well, but else I go over the 2 GB limit. That's something I would still have to experiment with a bit.
 

RicherSims

Resource contributor
Messages
566
Country
dominica
Ok. I thought that splitting the BGLs into LOD 11 or 10 coverage areas was to improve in sim perforamce, example, decreased VAS usage.
 

arno

Administrator
Staff member
FSDevConf team
Resource contributor
Messages
32,858
Country
netherlands
I'm sure it helps performance to organize the files by LOD, but I only do it when the area doesn't fit in one file due to the 2 GB limit. This 2500 sqkm area results in 40 GB of BGL files right now.
 

RicherSims

Resource contributor
Messages
566
Country
dominica
Hey Arno, thanks for the tip. I used global mapper to split my large color matched TIFFs into tiles each 26754x26754 px. This gives me uncompressed files coming in at 1.999GB. I can compile 8 or 9 at once and the time is now down to an amazing 2 hours.
 
Top