• Which the release of FS2020 we see an explosition of activity on the forun and of course we are very happy to see this. But having all questions about FS2020 in one forum becomes a bit messy. So therefore we would like to ask you all to use the following guidelines when posting your questions:

    • Tag FS2020 specific questions with the MSFS2020 tag.
    • Questions about making 3D assets can be posted in the 3D asset design forum. Either post them in the subforum of the modelling tool you use or in the general forum if they are general.
    • Questions about aircraft design can be posted in the Aircraft design forum
    • Questions about airport design can be posted in the FS2020 airport design forum. Once airport development tools have been updated for FS2020 you can post tool speciifc questions in the subforums of those tools as well of course.
    • Questions about terrain design can be posted in the FS2020 terrain design forum.
    • Questions about SimConnect can be posted in the SimConnect forum.

    Any other question that is not specific to an aspect of development or tool can be posted in the General chat forum.

    By following these guidelines we make sure that the forums remain easy to read for everybody and also that the right people can find your post to answer it.

Profiling the loading of files in P3D

  • Thread starter Thread starter Deleted member 12505
  • Start date Start date
D

Deleted member 12505

Guest
I'm trying to discover why P3D V2.5 12945 seems to want to read my entire scenery library (including bathymetry data even when I have it disabled) during mid-flight.

I'm posting here for any possible insight in what appears to me to be very abnormal file access.

I ran test flight from KDAB (Daytona) with (default airport with FTX Global, OpenLC, etc.) ... I logged File IO to a .CSV file. I was surprised to see so many BGLs being accessed for such a sort flight and not only that but aircraft.cfg were .

You can download the zip file here containing the entire log: https://drive.google.com/file/d/0Bw0Q-fAfEZwyaFVxZ0t1dTBkX2s/view?usp=sharing

Reading my entire scenery library during mid flight?
Rows 469219 - 472437

Reading Nagasaki Intl and Omura BGLs?
Rows 483287 - 483306
http://robainscough.com/images/85a8464db077f4f3a86fabb50ae382db.jpg

Reading Orbx NewZeal scenery (FTX_NZSI*.bgl)
Rows 483308 - 483319

Reading Aircraft.cfg for All installed 3rd party aircraft during mid flight??
Rows 499490 - 500156

Bathymetry is disable so why is it being referenced?
Rows 505527 - 505547
Rows 505553 - 505589
http://robainscough.com/images/89d02a951aec0dfa0105b99499db0809.jpg

Aerosoft EGLL_Docking.bgl
Rows 483271 - 483282
Rows 506334 - 506345
http://robainscough.com/images/2b13be46c960fabd818d0288d9f228fc.jpg

As you can see from the log file there are cached reads FASTIO_READ and there are also considerable IRP_MJ_READ which are much slower. There's obviously some less efficient allocations vs. end of files (actual) but that's fairly normal and I doubt many scenery designers check their BGLs to make sure they fit "standard allocations".

Maybe this is "normal" but it just seems very odd why I'd be loading Aerosoft EGLL_Docking.BGL when I'm flying around KDAB default airport (Aerosoft KDAB is NOT enabled) ... I do experience rapid VAS consumption almost exactly at I pass over from land to water which as you can see P3D loads Bathymetry data even though I have Bathymetry disabled.

Any insight welcome on the mysteries of file access in P3D.

Cheers, Rob.
 
Last edited by a moderator:
Hi Rob:

I don't yet have P3D to test, but perhaps it might be of interest to compare a parallel scenario with FSX. :idea:

Any scenery listed within the Scenery.Cfg file whether set "active" or "not active" will always be 'inventoried' by MSFS (any version AFAIK) during initial start up, or when re-scanning / re-indexing Scenery.Cfg after re-saving the Scenery Library (either from FSX Main Menu GUI or pull-down menu mid-flight).

Thus any objects or other content mapped via BGL, MDL, AGN, SPD, XML files etc. will be subject to verification that the mapped objects and all attached / sub-mapped meta-objects and textures are physically present, otherwise an error message would be triggered internally.

By default, such error messages are not displayed ...unless the following FSX.Cfg file parameters are manually added and enabled:

Code: 
[SCENERY]
// To show alerts or areas, set to = 1
// To hide alerts or areas, set to = 0

// Alert for missing textures
ShowMissingTextureAlert=1

// Alert for missing objects
MissingLibraryAlert=1
http://www.fsdeveloper.com/wiki/index.php?title=Missing_textures


Although I have tested FSX with SysInternals FileMon or ProcMon in the above scenery Library-related scenarios I described above, I have not tested FSX during a live flight session ...to see if it is doing any of the things you reported above with P3D.

Possibly you are already aware of the functionality and configuration options that follow; I'll mention them for readers un-familiar with these ideas. ;)


I found that my start-up time for FSX was slower when default and add-on scenery for areas of the world Geographically un-related to my present flight session were set "active" in the scenery.Cfg file.

I tried to "disable" anything Geographically un-related to my flying area by un-checking them in Scenery Library GUI; this make FS startup 'somewhat' faster.

When I physically removed the "disabled" Area / layer entries from Scenery.Cfg, my start-up time for FSX was substantially faster.


SysInternals FileMon or ProcMon no longer showed attempts to read certain scenery folders ...when the 'links' to them were physically removed from Scenery.Cfg (by removing their Area / layer entries entirely from Scenery.Cfg). :pushpin:


FYI: To physically remove save / restore Areas / layers from Scenery.Cfg, I use Hans Hartmann's FS2004 Scenery Config Manager (aka "SCM2004") with a couple of batch files and REG files to fool it into allowing use with FSX as well.

SCM2004 Scenery Manager

http://library.avsim.net/search.php?SearchTerm=Hans Hartmann&CatID=root&Go=Search


Also note-worthy, is Michael Heise's utilities:

MH's FS Scenery Configurator V1.2

http://library.avsim.net/esearch.php?CatID=fsxutil&DLID=114149


I have yet to try out his newer version:

FS Scenery Configurator V2.0

http://library.avsim.net/esearch.php?CatID=fsxutil&DLID=123080


You may find some of his other utilities interesting too:

http://library.avsim.net/search.php?CatID=root&SearchTerm=michael heise&Sort=Added&ScanMode=1&Go=Change View


In addition to using SCM 2004 to disable / remove scenery area entries in Scenery.cfg that are not reasonably close to one's flight path which have the new FSX "numeric" area names, one can do the same with disabling / removing various legacy named "city" folder area entries in Scenery.cfg still used by FSX for content storage (often containing numerous photorealistic texture tiles!). :wizard:

< ex: [FSX install path]\Flight Simulator X\Scenery\Cities\Istanbul\ etc. >


This program can be used in conjunction with the FSX SDK Base File diagram to selectively edit the Scenery.cfg file and thereby reduce the burden of currently unused files that FSX is "aware" of during its load process, so that one can minmize file read operations even further than appears to be achieved via the above "DisablePreload=1" tweak.


The FSX SDK Base File Information diagram is shown and discussed in detail in this interesting thread:

http://www.fsdeveloper.com/forum/showthread.php?t=5864

...and is documented here:

https://msdn.microsoft.com/en-us/library/cc707102.aspx#Base_File_Information

http://www.prepar3d.com/SDKv2/LearningCenter/environment/terrain_and_scenery.html#Base File Information


Another related and very interesting thread on optimizing the FSX scenery engine to only load data needed for a given flight (with excellent graphical illustration by Luis Feliz Tirado now lost due to AVSIM's RIDICULOUS 7-day content purge policy!
icon_evil.gif
) appeared in the AVSIM forums discussing the number of scenery area quads (QMIDs) which load under and outside one's flight path depending on the slider settings for "Level Of Detail (LOD) Radius":

"The LOD radius slider (LOD is the same grid, now named QMID) simply determines how many geographically limited files are loaded:

Low = 2.5 quadrants
Medium = 3.5 quadrants
High = 4.5 quadrants"

http://forum.avsim.net/topic/65490-strange-fsx-file-accesses/page-2



NOTE: The thread immediately above actually starts on the bottom of Page-4 and must be scrolled upwards in reverse order, IIUC, due to AVSIM's forum maintenance and archiving utility having "mangled" many older threads. :banghead:

But, here's the OP "teaser" that sounds somewhat similar to your P3D scenario, and which also brings to mind your own highly-informative prior thread here:

http://forum.avsim.net/topic/423417-lod-radius15-but-its-not-about-distance/

...and boez' subsequent thread here:

http://forum.avsim.net/topic/452357-terrain-lod-slider-fsx-v-p3d/

========================================================
"#60 Gypsy Baron

  • Member
  • photo-thumb-126956.jpg
  • Members
  • 04.png
  • 875 posts
Posted 14 February 2008 - 01:50 AM

I began a flight from Baltimore-Washinton International to Toronto this evening and was noticing an unusual amount of disc activity coupled with 'hang ups' of up to 3-4 seconds, so I opened FileMon to see if I could get idea of what was going on.

It seemed FSX was continuously opening, reading and closing files in the global scenery folders and it appeared it was the same groups of files, over and over.

But what was strange was some files that were accessed continually throughout the entire flight that were related to airports and scenery THOUSANDS of miles from my flight path. Tens of thousands of file accesses to scenery far removed from the flight path.

One particular group of files for Khrabrovo, Kalingrad, Russia were accessed more than any of the other 'out of area' files. There must have been 15-20 different files from this addon airport that were being opened, read, and closed throughout the flight."

http://forum.avsim.net/topic/65490-strange-fsx-file-accesses/page-4#entry464815

========================================================



Expanding on my findings above with Scenery.Cfg, I experimented with disabling all aircraft (besides the default C-172 kept as a 'fail-safe fall-back' for use of 3rd party saved flight files), and the one I was actually flying in a flight ...by re-naming the Aircraft.Cfg to ex: 'Aircraft.OFF' for aircraft I was not currently flying.

My start-up time for FSX was substantially faster.

BTW: I later read that those who use AI traffic and/or who fly in multi-player mode may wish to instead re-name Panel.Cfg (rather than Aircraft.Cfg) to ex: 'Panel.OFF' ...so that the 'visual model' of the aircraft can still be displayed by FS when needed.

However, this results in the longer FS "start-up" inventory process still taking place for those aircraft having a normally-named Aircraft.Cfg.



3rd party add-ons from authors such as OrbX FTX, Aerosoft etc. 'may' customize / substitute FSX default land class / water class / autogen files and their definitions / configurations via substitution of various FSX "system" files such as Terrain.Cfg, LCLookup.BGL, WorldWC.BGL, WorldLC.BGL, Regions.BGL, Seasons.BGL, the Autogen XML / SPD / *an.AGN files etc..

Subsequently, we may find that in an effort to expand the options for use of land class texture tiles (and their autogen annotations and/or other mapped content such as 'attached' meta-objects etc.), such authors are burdening the USERVAS and content loading sub-system of MSFS with file I/O during the course of a live flight session.

This may occur as mapped content is 'called' to be loaded when the user aircraft enters the Geographic area mapped by ex: BGL and Autogen files (and/or scenery conditional display modules such as "Living World / "FTX Flow" or SODE / other XML SimConnect conditional content display modules), causing Geographic position-dependent triggering of dynamically loaded content which was not yet 'known' to be required when the scenery was 'pre-loaded' for the location of the user aircraft when the flight was started.



It might be helpful to the analysis process to try a test scenario which uses only a purely default installation with no 3rd party add-ons installed.

This might then be compared with a test scenario which uses only (1) ex: Aerosoft 3rd party scenery add-on installed.

Subsequent test scenarios with ex: OrbX FTX may prove even more complex and informative about "repetitive use" and dependencies upon availability of land class texture tiles and/or Autogen / objects / meta-objects etc. from other Geographic regions of the world ...otherwise "Geographically un-related" to ones present flight session. :yikes:


BTW: IIUC, modules / gauges which run in the MSFS address space as child processes of FSX may cause FSX.exe itself to "appear" to be reading scenery files ...when logged in FileMon / ProcMon.


Hope this "abundance" of info might prove helpful in sorting out the scenario with P3D that you described above in your OP. :)

GaryGB
 
Last edited:
Hi Gary,

Many thanks for the response. The triggering of 'inventoried' scenery from the scenery.cfg is actually happening without any user interaction ... I'm not accessing any menu and not attempting to change a view, staying in the VC, that's what has me puzzled. This entire "inventory" process continues thru-out the flight duration.

In a no add-on installation, the File IO thrashing is considerably less and I have not found a single instance where the entire scenery library is "inventoried" during flight ... a clue perhaps.

Based on what you've suggested, the Enable/Disable of scenery elements seems to imply it's evaluated only for rendering but is read regardless of enable/disable. Which according to Adam (LM) is how it works for Bathymetry also ... I wish I knew why from a design perspective? For no add-on environments there isn't much of a performance hit, but start to fill the world up and it does start to become a performance issue.

In another scenario, I demonstrated how accessing the scenery library via UI would eventually result in an OOM if accessed many times even though my location is static (not flying just sitting on a runway). Here is a video demonstrating that issue:
and this is with a NO Add-on P3D ... VAS was being consumed at a rate of 44MB per scenery library access. In situations where I had many add-on installed, I'd would be consuming VAS at an alarming rate even though I'm not actually flying anywhere just sitting on the runway.

I have brought this up to LM's attention several times but scenery processing seems to be one of those spaghetti code areas (partly from years of compatibility processing) that induce developer "fear" when walking that code path.

Thank you for your insight and links. You've given me some ideas on how to manage my environment better.

Cheers, Rob.
 
Hi Rob:

Thanks for the heads-up on that 44-MB VAS consumption per mid-flight access of the Scenery Library GUI. :yikes:

I had never looked at the USERVAS stats for what occurs mid-flight as a result of doing what FS scenery developers do repeatedly in an average work session; that's excellent info to keep in mind as one works, to more accurately test run time VAS footprint and performance impact of ones scenery. :idea:

GaryGB
 
The same erosion of VAS resources occurs for modelers and gauge/system programmers every time "Reload User Aircraft" is invoked to test new changes to XML scripts. :confused:
 
Indeed, Bill, I was wondering about whether that might also be an issue for aircraft / gauge development. :scratchch


BTW: Bill, may I digress briefly and "invite your insight" to offer comment on some 3D modeling issues in another thread ? :oops:

I would value your insight on the reported FSX SDK removal of a 4MM MDL vertex 'welding' requirement, and the seemingly incongruous reported ability to achieve reduced vertex expense when "smoothing" is used:


My post on the above issues: http://www.fsdeveloper.com/forum/threads/misaligned-textures.434275/#post-712756

The thread URL: http://www.fsdeveloper.com/forum/threads/misaligned-textures.434275/


Thanks in advance for any input you might share with us. :)

GaryGB
 
Last edited:
Gary,

Just to clarify the Scenery Library access VAS consumption was not "mid-flight" ... it was parked on a RWY, no actual motion thru the virtual world. The 90MB for menu access appears to be one time only. The 44MB per scenery library access is cumulative.

Bill,

Is LM aware - I didn't see any mention in the Beta forum, done private? LM are doing a lot of cleanup for V3 so definitely worth reminding them before it hits Beta.

Cheers, Rob.
 
Yeah, I better go remind them. Who knows, they might actually read my post and/or listen... :duck:
 
They listen ... more so than I listen to our customers ... doooh! ;)

Cheers, Rob.
 
UPDATE:

I did a lot more testing removing scenery, airports, aircraft, etc. to see if the continuous re-scan of my entire scenery library actually had any impact on VAS ... tested with FTX Global, PILOT's Mesh, PMDG 777, Aerosoft KDAB. VAS was a little better (100MB) but not A LOT better ... not worth the effort.

Even though a "loaded" Scenery library with many installed aircraft and airports does generate considerably more File IO, it doesn't seem to cause that much of a VAS issue.

I can replicate the large VAS usage (>250MB) with a coastline transition (from land to water) in EVERY case ... it's exactly at the point of transition where I get sudden consumption of VAS. I've reported to LM and provided the list of add-ons to duplicate.

Cheers, Rob.
 
Thanks for the updates, Rob... good to know you're finding these issues and reporting them to LM ! :)

GaryGB
 
Yeah, Adam has me regression testing prior V2.5 hotfixes to see if I can replicate the same issue in those ... ugh ... sleep's overrated I guess.

Bill ... you sure you want LM to listen to you ... it might cost you a LOT more work ;)
 
UPDATE:

More testing ... it appears it "might" be Region related (Region.bgl). The area's I tested that had the sudden large VAS consumption KDAB and KLAX, were right on the border of North American South region and Tropics American region. Here is a sample of VAS usage as this region becomes "active":

646546 Monitor IPC:024C (S32) = 470308
656608 Monitor IPC:024C (S32) = 385772
666686 **** WARNING! Free memory is very low!
666686 Monitor IPC:024C (S32) = 253032
676717 **** WARNING! Free memory is very low!
676717 Monitor IPC:024C (S32) = 253812

10 second increments - 230MB consumed ...

Now the real issue, how to prevent this without "core changes" or more importantly (or maybe not) without breaking considerable compatibility with existing 3rd party products -- I think the two may go hand in hand (damned if you do and damned if you don't).

Cheers, Rob.
 
Hi Rob,

it actually makes sense that crossing a region boundary would require more memory (and file access activity) because the sim will need to load a second set of landclass ground textures and autogen and maintain both sets (or even more, as in some locations more than just two regions meet). It's the equivalent of dawn or dusk flights when both day and night textures are loaded at the same time and blended together.

While there may be ways to improve the efficiency of these transition zones -- loading new content and unloading files no longer needed -- I doubt that the issue can be fundamentally changed. As a fully global simulator there needs to be some mechanism that loads new content periodically as the user's aircraft moves forward. Moreover, unloading cannot happen at the same time as loading given the long view distances required in a flight simulator.

Cheers, Holger
 
Hi Holger,

It does make sense ... couple of ideas:

1. Divide the regions into small sections (break compatibility)
2. Do away with regions completely and operate on a "view" radius and/or even incorporate weather visibility at altitude to reduce overhead -- a smarter cone

I think #2 could be done all on the code side without breaking existing compatibility.

It does explain why some areas, like Europe are considerably higher on VAS usage. I know LM are working on optimizations, hopefully they'll take a close look as I suspect there is a better VAS friendly approach, not so sure about performance friendly.

The other option is to go the 64bit route, but that will most likely break compatibility also ... don't want JV to go on another rant ;)

Cheers, Rob.
 
Hi Rob,
I just came upon this thread and thought I would ask you about using Premocache v2 that just came out. As I understand the product, you automatically cache scenery files to an external (or internal) SSD or an external flash drive and as you fly along the sim will then look in that drive for the scenery files instead of your normal hard drive at a much faster speed. This should in theory do away with the little pauses and also maybe increase FPS. You can download the product and use it foe a free evaluation, then if you don't like it, you don't have to pay for it, but it will no longer work. Might be worth a try.
Thanks.
Howard
 
Hi Guys

No promotion here, but I'd like to invite you to try a free tool that I did develop myself, which goal is to build some scenery "profiles" allowing you to enable/disable sceneries that you may need, or not, for a flight.
Given the fact that you don't need all your sceneries to be active anytime, this free tool will just allow you to enable sceneries for a certain type of flight.

Let's say, as an example, that you wanna flight from Paris to Lisbon. Are you sure you need your photoreal sceneries of austria or USA ?
For sure, you don't need them.

So, building profiles, will let you assemble sceneries needed for achieving a specific flight.
You can easilly build as many "profiles" you need for your different fligths.

It's a totally free tools that you can download here:
http://sceneryprofiler.com/
Or here:
https://www.fspilotshop.com/advanced_search_result.php?keywords=Scenery+Profiler
Or here:
https://secure.simmarket.com/scenery-profiler-fsx-p3d-(fr_10954).phtml

Have great flights !!!
 
You must log in or register to reply here.
Back
Top