We NEEEED a multithreaded resample!!! ARRRGHHH!!!

MOUSY

Resource contributor
#1
Complete shot in the dark... but how hard it would it be to create a tool to somehow magically enable multiple resample instances to run simultaneously and compile into 1 BGL?

Or an even longer shot in the dark, to somehow get the source code for resample from Lockheed developers and rewrite it so that is multithreaded?

Or even fuuurther... start from scratch, figure out how resample does what it does and create an entire new and improved multithreaded resample 2.0?

I'm currently sitting at my PC watching 3 instances of resample run on an 8 core processor and only 39% of the total avaialble processing power is being used. One of them sits at 5% after 2 and a half hours and has a further 46 hours to go... **pulls hair out**
I'm running 3 of them just so that I can experiment with different terrain nightmaps, and, in the event that the one which finishes first produces unsatisfactory results, I can cancel the other 2 and start over. There has GOT to be a way to speed things up for large scale outputs. (Yes I know I can work on a smaller area first and then go full scale later, but i'd still have to wait for 2 straight days eventually!)
 
#2
Hi MOUSY:

I agree that it would be helpful to have a faster-working version of FS / P3D SDK Resample.


Until we have one, it merits repeating a caveat regarding avoidance of LZW compression in Resample TIF source files:

http://www.fsdeveloper.com/forum/threads/blend-water-masks.435509/#post-726137


PS: I edited today another of my posts regarding Resample (one from 2011) that did not yet mention this caveat. :pushpin:


Hope this info helps anyone still using LZW Compression in aerial imagery and/or elevation data Resample TIF source files. :)

GaryGB
 

MOUSY

Resource contributor
#3
Wow. Gary I was TOTALLY unaware that LZW compression TRIPLES (TRIPLES!!!) compile time! Thanks for that huge tip!
(I must also remember to work off my SSD, but its already so cramped!)

Now back to the original topic. Does anyone have any ideas on how we can go forward with this. I would put all my programming know how into this if I learnt there was a way.
 
#4
I don't have an answer for you but what could you possibly be compiling that would take 46 hours!? An entire country into one bgl :p. I usually perform a batch run of 4-5 resamples running at once on my I-7 CPU at 90%. I can create 25 bgls in around 20 minutes covering around 50 square miles (about 130 square kilometers).
 

MOUSY

Resource contributor
#5
An entire country into 1 BGL is correct.
4 TIFS for 3 different seasons and a night map, each covering about 1200 square kilometers at 0.5m/px, each LZW compressed to 3.4GB (would otherwise be 11GB BigTiffs and thus unusable by resample).
The resulting BGL is 400MBs at 85% compression.

I'm considering using multiple BGLs instead of a single large one, however I have not yet fully investigated the effects on VAS usage in SIM.
 
#7
Thanks for sharing that practical insight, George; apparently 1-Bit B+W and/or 8-Bit Gray Scale TIF Terrain Mask files using LZW compression do not take as much time to process as 16-Bit, 24-Bit, or 32-Bit "color" source files. :)

[EDITED]

NOTE: Final output source file format for TIFF Masks must actually be 8-Bit gray-scale; see:

http://www.fsdeveloper.com/forum/th...-scenery-achievable.440912/page-4#post-789447

[END_EDIT]


Indeed, I have edited my suggestions on "preferred" settings to use with TIF source files for aerial imagery and terrain mesh on the assumption, IMHO, that most would-be users may want to forego LZW compression in favor of saving processing time.

However, those with special requirements involving very large data sets wishing to conserve USERVA in a FS / P3D SDK Resample command mode Windows task session, and/or those having a goal of minimizing size or number of output BGLs, certainly might want to choose the (non-lossy) LZW compression option to preserve data fidelity, rather than utilizing the (always lossy) compression options offered by Resample itself ...with the understanding that Resample processing of all source files may take upwards of 3 times longer. :idea:


One might also wonder whether another option might be faster processing by Resample of source files located within folder chains compressed by Windows NTFS folder compression (...which at one time was promoted by MS due to reportedly minimal impact on system throughput speed due to its background use of 'unused' CPU clock cycles on 1 GHz and faster CPUs) ? :scratchch


Regardless of the processing scenario, one may also speed things up by setting the FS / P3D SDK Resample command mode CPU task priority in Windows Task Manager to "Realtime". ;)


GaryGB
 
Last edited:

arno

Administrator
Staff member
FSDevConf team
Resource contributor
#8
Hi,

I doubt we can make a new resample. The file format is not fully known.

What I do is run multiple resample instances in parallel. And I'm planning to stretch that to control multiple instances on multiple machines controlled from one place.

I'm actually running a job now, I'll post the performance here when it's done.
 

arno

Administrator
Staff member
FSDevConf team
Resource contributor
#9
Hi,

Just for your information, here are the stats of my last run. It's an area of 2500 kmsq covered by LOD17 photo scenery (25 cm input imagery). It took 12 hours and 30 minutes to generate the whole area. The BGL files are divided by LOD11 (typically around 300 MB in size).

I was running 4 resamples in parallel for this, on my build machine with 8 GB of RAM and an i5 processor. Most of the time is running resample, but generating the watermasks from vector data is part of the process as well.
 

MOUSY

Resource contributor
#10
Good info Arno.
Any particular reason for splitting the BGLs into LOD11 areas?
And were your TIFFs uncompressed?
 

arno

Administrator
Staff member
FSDevConf team
Resource contributor
#11
Yes, the input files are uncompressed GeoTIFF files.

I guess I might get away with LOD10 for the BGL size as well, but else I go over the 2 GB limit. That's something I would still have to experiment with a bit.
 

MOUSY

Resource contributor
#12
Ok. I thought that splitting the BGLs into LOD 11 or 10 coverage areas was to improve in sim perforamce, example, decreased VAS usage.
 

arno

Administrator
Staff member
FSDevConf team
Resource contributor
#13
I'm sure it helps performance to organize the files by LOD, but I only do it when the area doesn't fit in one file due to the 2 GB limit. This 2500 sqkm area results in 40 GB of BGL files right now.
 

MOUSY

Resource contributor
#14
Hey Arno, thanks for the tip. I used global mapper to split my large color matched TIFFs into tiles each 26754x26754 px. This gives me uncompressed files coming in at 1.999GB. I can compile 8 or 9 at once and the time is now down to an amazing 2 hours.
 
Top