• Which the release of FS2020 we see an explosition of activity on the forun and of course we are very happy to see this. But having all questions about FS2020 in one forum becomes a bit messy. So therefore we would like to ask you all to use the following guidelines when posting your questions:

    • Tag FS2020 specific questions with the MSFS2020 tag.
    • Questions about making 3D assets can be posted in the 3D asset design forum. Either post them in the subforum of the modelling tool you use or in the general forum if they are general.
    • Questions about aircraft design can be posted in the Aircraft design forum
    • Questions about airport design can be posted in the FS2020 airport design forum. Once airport development tools have been updated for FS2020 you can post tool speciifc questions in the subforums of those tools as well of course.
    • Questions about terrain design can be posted in the FS2020 terrain design forum.
    • Questions about SimConnect can be posted in the SimConnect forum.

    Any other question that is not specific to an aspect of development or tool can be posted in the General chat forum.

    By following these guidelines we make sure that the forums remain easy to read for everybody and also that the right people can find your post to answer it.

Bug in SPLITGRID Step ?

Hi,

I have just added a change to the development release which adds support for holes to the import shp step. So now holes should be imported correctly. Even islands inside holes should work.

Let me know if there are any issues. There will be some performance hit for processing the holes. But on the other hand I have optimized my Boolean code more now, so that should make splitgrid a bit faster.
 
Hi,
I have a little ask:
Could Scenproc only do this ? importshp -> correct holes -> exportshp with attributes ?

Because, I would correct Holes problem on big shapes files before split it in many parts to work on thoses with Scenproc... and I would avoid to lose time to work on big shape file each time...

edit:

and a second one: could I activate or desactivate this "correct hole" function ?

thanks Arno
 
Last edited:
Hi,

Yes, if you import a SHP and then export it directly the holes will be removed. So that would work.

Why would you want an option for this? Disabling the hole removal will result in the incorrect results as before. scenProc would not export the objects with the holes again.
 
Thank you Arno.... Again.
Perfect :)

For activate/unactivate it was just to gain Time if you work on a file that you know There is no hole in... It was just because you said that this option were Time eater :)

But the option is already perfect like that... I go to test it now ! Thank you again Arno
 
First test on big shapefile coming from Corrine2006 Forest Data... Importshp step fail... I send you a link to the shape file by PM (50Mo)
 
Ok, I'll have a look.

Send from my phone using Tapatalk, so excuse the short sentences and possible typos
 
SPLITGRID seems to be the main performance bottleneck atm with big jobs
there might be a way to speed things up

when scenproc runs, i will not touch the machine until the job completes
while processing SPLITGRID step; my machine only uses 25% of CPU and available memory
i was thinking; maybe we can utilize idle resources to speed thing up

would it be possible for scenProc to divide and multithread SPLITGRID step?
possibly 2-4 smaller, parallel jobs utilizing more idle resources?


TIA


Chris
 
Last edited:
Hi Chris,

scenProc is already 64 bit application and the SPLITGRID step already runs things in parallel (maximum 64 threads if I remember correctly). So what I see here is that all my 8 cores are on full load, giving me almost 100% load.

It seems the parallel job code for .NET I use works so that it schedules up to 64 tasks and then runs them. But before new tasks are started all 64 have to finish, so while there are only one or two that still need to finish you might see a lower CPU usage. Not much I can do about that.
 
Thank you Arno

i have noticed on W7 more resources are utilized; at least 50% at any given time
with XP i see only 25% utilization at any given time

interestingly enough, i run same job on both machines; starting at the same time
both completed about the same time frame

Best regards

Chris,
 
More threads is not always better. Sometimes it is more efficient to focus all resources on finishing one thing quickly :).

But when I added the multi-thread support to scenProc it make the SPLITGRID step quite a lot faster.
 
its prob just me; being impatient :D
im looking at 3 blocks filled in SPLITGRID step, its been two and half hours now with no movement
on task manager - all looks positive,
about 600mb mem usage with slight fluctuation and constant 25% CPU usage (scenProc thread)

i wish there was a way to confirm scenProc is processing and not stuck in a loop



Chris,
 
Hi,

The progress bar should keep moving (slowly), that's the sign everything is still alive. Or is your window completely frozen?

It might be an idea to try to process a slightly smaller area if things take so long. For example split your data in two and process it separately.
 
Thanks Arno

actually everything is working smoothly
window is not frozen or non responsive, looks ok on task manger as well
i just saved the log file as its working

i just realized this isn't such a big job
i run much bigger before and completed very quickly

im only running tree's and buildings; no lights or roads
im also not using my previously edited waterway file (no Mediterranean coastline poly), just the plain one from geofabric; and im not even pulling anything from it


current log

Code: 
9:08 AM	IMPORTSHP	Information	Reading SHP file D:\OSMDATA\natural.shp
9:08 AM	IMPORTSHP	Information	Read 7485 features from file
9:08 AM	IMPORTSHP	Information	Reading SHP file D:\OSMDATA\landuse.shp
9:08 AM	IMPORTSHP	Information	Read 7039 features from file
9:08 AM	IMPORTSHP	Information	Reading SHP file D:\OSMDATA\waterways.shp
9:08 AM	IMPORTSHP	Information	Read 1243 features from file
9:08 AM	IMPORTSHP	Information	Reading SHP file D:\OSMDATA\buildings.shp
9:08 AM	IMPORTSHP	Information	Read 71009 features from file
9:08 AM	SPLITGRID	Information	Splitting features in 0.0146484375 x 0.010986328125 grid...

its 12:15 PM PST



Chris
 
Last edited:
Hi,

I would say that is taking longer than I would expect. It seems you have around 100000 features, I would expect that to take less time. But of course it depends on the area they cover as well. If it covers a big area, it will take longer.
 
Thank you Arno

what do you consider a large area?

these are the coordinates with 25% Mediterranean water front
NorthWestCornerLatitude = 33deg 27min 12.13sec north
NorthWestCornerLongitude = 34deg 12min 14.77sec east
SouthEastLatitude = 29deg 23min 18.34sec north
SouthEastLongitude = 35deg 57min 42.89sec east

roughly 700km length, with varying width 200-40km

Chris,
 
Last edited:
Hi,

I would see big is anything bigger than roughly 100x100 km in size. So that is also roughly 1x1 degree.

Of course it depends on how dense your data is, but for the Netherlands test data I used bigger areas slow things down too much or use too much memory.
 
Thanks Arno

interesting test i just run
i killed the job i was running on my XP
moved the same code with the same files over to my W7 station
run the same exact job, completed in 11 minutes

simultaneously i run the job again on my XP station
only this time i commented out my waterways.shp
it start SPLITGRID; than gets stuck after three blocks

Chris,


W7 Log

Code: 
1:16 PM	IMPORTSHP	Information	Reading SHP file D:\OSMDATA\natural.shp
1:16 PM	IMPORTSHP	Information	Read 7485 features from file
1:16 PM	IMPORTSHP	Information	Reading SHP file D:\OSMDATA\landuse.shp
1:16 PM	IMPORTSHP	Information	Read 7039 features from file
1:16 PM	IMPORTSHP	Information	Reading SHP file D:\OSMDATA\waterways.shp
1:16 PM	IMPORTSHP	Information	Read 1243 features from file
1:16 PM	IMPORTSHP	Information	Reading SHP file D:\OSMDATA\buildings.shp
1:16 PM	IMPORTSHP	Information	Read 71009 features from file
1:16 PM	SPLITGRID	Information	Splitting features in 0.0146484375 x 0.010986328125 grid...
1:27 PM	SPLITGRID	Information	Features divided in 49914 cells
1:27 PM	SETAGNBUILDINGHEIGHT	Information	Setting autogen building heigth...
1:27 PM	SETAGNBUILDINGHEIGHT	Information	Set autogen building heigth for 49914 cells
1:27 PM	SETAGNBUILDINGTEXTURE	Information	Setting autogen building texture to BLD_IS1
1:27 PM	SETAGNBUILDINGTEXTURE	Information	Set autogen texture for 49914 cells
1:27 PM	SETAGNBUILDINGTEXTURE	Information	Setting autogen building texture to BLD_IS2
1:27 PM	SETAGNBUILDINGTEXTURE	Information	Set autogen texture for 49914 cells
1:27 PM	SETAGNBUILDINGTEXTURE	Information	Setting autogen building texture to BLD_IS3
1:27 PM	SETAGNBUILDINGTEXTURE	Information	Set autogen texture for 49913 cells
1:27 PM	CREATEAGNGENBUILD	Information	Creating autogen generic buildings with filter FTYPE=POLYGON;FROMFILE=buildings.shp;type!hotel;FWIDTH<15...
1:27 PM	CREATEAGNGENBUILD	Information	Created 63123 autogen objects
1:27 PM	CREATEAGNGENBUILD	Information	Creating autogen generic buildings with filter FTYPE=POLYGON;FROMFILE=buildings.shp;type!hotel;FWIDTH>20...
1:27 PM	CREATEAGNGENBUILD	Information	Created 14195 autogen objects
1:27 PM	CREATEAGNROWHOUSE	Information	Creating autogen row houses with filter FTYPE=POLYGON;FROMFILE=buildings.shp;type!hotel;FWIDTH<20;FWIDTH>15...
1:27 PM	CREATEAGNROWHOUSE	Information	Created 16252 autogen objects
1:27 PM	CREATEAGNPOLYVEG	Information	Creating autogen polygon vegetation with filter FTYPE=POLYGON;FROMFILE=natural.shp;type=forest...
1:27 PM	CREATEAGNPOLYVEG	Information	Created 4816 autogen objects
1:27 PM	CREATEAGNPOLYVEG	Information	Creating autogen polygon vegetation with filter FTYPE=POLYGON;FROMFILE=natural.shp;type=park...
1:27 PM	CREATEAGNPOLYVEG	Information	Created 4238 autogen objects
1:27 PM	CREATEAGNPOLYVEG	Information	Creating autogen polygon vegetation with filter FTYPE=POLYGON;FROMFILE=landuse.shp;type=residential...
1:27 PM	CREATEAGNPOLYVEG	Information	Created 5696 autogen objects
1:27 PM	WRITEAGNFILES	Information	Writing AGN files to folder D:\OSMDATA\AGN\All
1:27 PM	WRITEAGNFILES	Information	Written 5156 autogen files
1:27 PM	SceneryProcessor	Information	scenProc finished processing
1:27 PM	SceneryProcessor	Information	    8.07 seconds for IMPORTSHP
1:27 PM	SceneryProcessor	Information	  643.94 seconds for SPLITGRID
1:27 PM	SceneryProcessor	Information	    0.58 seconds for SETAGNBUILDINGHEIGHT
1:27 PM	SceneryProcessor	Information	    3.46 seconds for SETAGNBUILDINGTEXTURE
1:27 PM	SceneryProcessor	Information	    8.24 seconds for CREATEAGNGENBUILD
1:27 PM	SceneryProcessor	Information	    5.77 seconds for CREATEAGNROWHOUSE
1:27 PM	SceneryProcessor	Information	    4.66 seconds for CREATEAGNPOLYVEG
1:27 PM	SceneryProcessor	Information	    2.36 seconds for WRITEAGNFILES
1:27 PM	SceneryProcessor	Information	  677.07 seconds total
 
Hi,

just to warn Chris about something:

I see that you use this GUID {00000000-0000-0000-0000-000000000000} for building generation. Be warn that if you use AGN Merge Tool, thoses building would be ignored cause of null GUID... if you want to conserve those, use {00000000-0000-0000-0000-000000000100} or any other GUID but not null GUID.
 
You must log in or register to reply here.
Back
Top