• Which the release of FS2020 we see an explosition of activity on the forun and of course we are very happy to see this. But having all questions about FS2020 in one forum becomes a bit messy. So therefore we would like to ask you all to use the following guidelines when posting your questions:

    • Tag FS2020 specific questions with the MSFS2020 tag.
    • Questions about making 3D assets can be posted in the 3D asset design forum. Either post them in the subforum of the modelling tool you use or in the general forum if they are general.
    • Questions about aircraft design can be posted in the Aircraft design forum
    • Questions about airport design can be posted in the FS2020 airport design forum. Once airport development tools have been updated for FS2020 you can post tool speciifc questions in the subforums of those tools as well of course.
    • Questions about terrain design can be posted in the FS2020 terrain design forum.
    • Questions about SimConnect can be posted in the SimConnect forum.

    Any other question that is not specific to an aspect of development or tool can be posted in the General chat forum.

    By following these guidelines we make sure that the forums remain easy to read for everybody and also that the right people can find your post to answer it.

P3D v5 Splitting large Geojson file into Shapefile

Messages
446
Country
us-missouri
There seems to be a 2 GB file size restriction in QGIS as I normally can turn Microsoft building footprint files into shapefiles quickly and easily. Florida comes in at just over 2GB unfortunately and I can't turn it into autogen buildings. Looks like there is an "ogr2ogr" OSGeo4W command that could do the job, but I am not familiar with all the required functions that would convert the file, or split it into smaller files. Any help would be greatly appreciated.
 

arno

Administrator
Staff member
FSDevConf team
Resource contributor
Messages
31,532
Country
netherlands
What I once did is split the geojson files into multiple files, that way I could process them easier. They are text based files, so splitting them is not too hard. I wrote a little tool for that.
 
Messages
446
Country
us-missouri
What I once did is split the geojson files into multiple files, that way I could process them easier. They are text based files, so splitting them is not too hard. I wrote a little tool for that.
Too big to be opened in Notepad++
Do you still have that tool?
 
Last edited:

arno

Administrator
Staff member
FSDevConf team
Resource contributor
Messages
31,532
Country
netherlands
Here you go, it is a very simple command line tool. It takes to arguments, the JSON file you want to split and the folder where the output files are written.
 

Attachments

  • JSONSplitter.zip
    2.6 KB · Views: 54
Messages
1,278
Country
unitedstates
Arno - wanted to try your JSON splitter. Every time I run it the command window opens up for about a second and closes. Can't get a chance to input arguments. From what I can read before it closes is "unhandled exception" System out of range exception: Index was outside the bounds of the array. ??

Do you create a bat file to run it? So I was able to capture this before it closed again: JSONSplitter.Program.Main(String[] args)
 
Last edited:

arno

Administrator
Staff member
FSDevConf team
Resource contributor
Messages
31,532
Country
netherlands
You need to run it from the command prompt. So first you open the command prompt, navigate to the right folder and then start the tool with the arguments typed behind it.
 
Messages
1,278
Country
unitedstates
Got it working which lead me to a new question. It broke down California into 11 .json files (roughly 300MB each). I take these are split not by geographic area but by attribute? The reason I ask is out of the 11 files only file 11 would open in QGIS, and it showed all foot prints for the entire state. I looked at some of the other ten files in Notepad++ and they all look the same like file 11 but will not open? Just curious.
 

arno

Administrator
Staff member
FSDevConf team
Resource contributor
Messages
31,532
Country
netherlands
They should all open (it's a while ago I used the tool myself). Each has about 1/11th of the data.
 
Messages
1,278
Country
unitedstates
Arno, I came back to explore JSONSplitter again which I now do need. Using the above posted example of California, it splits the data but I am not sure based on what criteria. For example, If I look at a small town before splitting all the footprints are their. After splitting, I see some of the footprints in one file and then the building next to it is in a different file. Is there a way to split the files geographically like North to South so footprints next to each other are found in the same file? Otherwise, I would have to load up all the files which defeats the purpose of splitting. o_O Maybe add a user input parameter? FYI, I am using the MS geoJSON data if that makes any difference.
 
Last edited:

arno

Administrator
Staff member
FSDevConf team
Resource contributor
Messages
31,532
Country
netherlands
Hi,

At the moment it just splits the features in the order that they are stored. So if the buildings of a town are spread over the entire file, they might indeed end up in different output files.

When I made the tool that was no issue for me, as I processed the data further in another tool before using it. I would have to check if I can add some kind of sorting to the data. Now it's really a quick and dirty tool to split things, so it does not really analyze the data further.
 
Messages
1,278
Country
unitedstates
If you have time to take a 2nd look at it that would be MOST APPRECIATED :p. Running the entire MS 3.5GB geojson brought my PC to its knees, even with 32GB of RAM. On a 2nd note, do you know if the MS data has footprint attributes such as the type of building it represents? Just curious if you have looked at their data. I'm still exploring. Thx.
 

arno

Administrator
Staff member
FSDevConf team
Resource contributor
Messages
31,532
Country
netherlands
The last time I look at the MS building footprints, they were just footprints with no attributes at all.
 
Messages
1,278
Country
unitedstates
"I would have to check if I can add some kind of sorting to the data."

That would be great Arno if you could take a look. Nothing fancy... maybe just sort by starting at the northern most point (adding some sort of flag/attribute?), working its way South until it has its 11 parts. Maybe a 2nd flaf where one could sort West to East. That would be quite manageable. :p

Happy Easter
Marcus
 

arno

Administrator
Staff member
FSDevConf team
Resource contributor
Messages
31,532
Country
netherlands
It would require that all data has to be read and stored to be able to sort things. So I think it will slow it down quite a bit (and use more memory).
 
Messages
1,278
Country
unitedstates
I see... and that could be very well be true. However, if we break it down it only has to be done one time and I can use it easily after that. If left as it is, we have to re-run the entire 3.5GB 138 times (on our current project). Hence, the fix could be a huge time saver. If I have to double my RAM from 32GB to 64GB we can/will easily do this if required for a new version. Besides California, we have to do this for a slated 10 more US states... a multi-year project.

If time is short for this, I can easily understand as it seems everybody want a part of you, ha! What's it written in? Would it be possible to get the code?
 

arno

Administrator
Staff member
FSDevConf team
Resource contributor
Messages
31,532
Country
netherlands
Hi,

I'll see if I can find some time, it would be an interesting exercise. The tool is written in C#.
 
Messages
1,278
Country
unitedstates
Well, if ya wanna have fun, I would say add a flag (is that the correct lingo?), if you want North-to-South, a 2nd flag if you want West-to-East, and a third flag as to how many parts you want to break it into.
 

arno

Administrator
Staff member
FSDevConf team
Resource contributor
Messages
31,532
Country
netherlands
I haven't forgotten this thread. I'm almost done implementing the sorting. Just need to fix a few issues and test performance.
 
Top