P3D v5 Splitting large Geojson file into Shapefile

[crushnik]

There seems to be a 2 GB file size restriction in QGIS as I normally can turn Microsoft building footprint files into shapefiles quickly and easily. Florida comes in at just over 2GB unfortunately and I can't turn it into autogen buildings. Looks like there is an "ogr2ogr" OSGeo4W command that could do the job, but I am not familiar with all the required functions that would convert the file, or split it into smaller files. Any help would be greatly appreciated.

 

[arno]

What I once did is split the geojson files into multiple files, that way I could process them easier. They are text based files, so splitting them is not too hard. I wrote a little tool for that.

 

[crushnik]

What I once did is split the geojson files into multiple files, that way I could process them easier. They are text based files, so splitting them is not too hard. I wrote a little tool for that.

Too big to be opened in Notepad++
Do you still have that tool?

 

[arno]

Here you go, it is a very simple command line tool. It takes to arguments, the JSON file you want to split and the folder where the output files are written.

 

[crushnik]

Thank you Arno, you're the best!

 

[Clutch Cargo]

Arno - wanted to try your JSON splitter. Every time I run it the command window opens up for about a second and closes. Can't get a chance to input arguments. From what I can read before it closes is "unhandled exception" System out of range exception: Index was outside the bounds of the array. ??

Do you create a bat file to run it? So I was able to capture this before it closed again: JSONSplitter.Program.Main(String[] args)

 

[arno]

You need to run it from the command prompt. So first you open the command prompt, navigate to the right folder and then start the tool with the arguments typed behind it.

 

[Clutch Cargo]

ah... makes sense. We'll give that a try. thx.

 

[Clutch Cargo]

Got it working which lead me to a new question. It broke down California into 11 .json files (roughly 300MB each). I take these are split not by geographic area but by attribute? The reason I ask is out of the 11 files only file 11 would open in QGIS, and it showed all foot prints for the entire state. I looked at some of the other ten files in Notepad++ and they all look the same like file 11 but will not open? Just curious.

 

[arno]

They should all open (it's a while ago I used the tool myself). Each has about 1/11th of the data.

 

[Clutch Cargo]

Arno, I came back to explore JSONSplitter again which I now do need. Using the above posted example of California, it splits the data but I am not sure based on what criteria. For example, If I look at a small town before splitting all the footprints are their. After splitting, I see some of the footprints in one file and then the building next to it is in a different file. Is there a way to split the files geographically like North to South so footprints next to each other are found in the same file? Otherwise, I would have to load up all the files which defeats the purpose of splitting. Maybe add a user input parameter? FYI, I am using the MS geoJSON data if that makes any difference.

 

[arno]

Hi,

At the moment it just splits the features in the order that they are stored. So if the buildings of a town are spread over the entire file, they might indeed end up in different output files.

When I made the tool that was no issue for me, as I processed the data further in another tool before using it. I would have to check if I can add some kind of sorting to the data. Now it's really a quick and dirty tool to split things, so it does not really analyze the data further.

 

[Clutch Cargo]

If you have time to take a 2nd look at it that would be MOST APPRECIATED . Running the entire MS 3.5GB geojson brought my PC to its knees, even with 32GB of RAM. On a 2nd note, do you know if the MS data has footprint attributes such as the type of building it represents? Just curious if you have looked at their data. I'm still exploring. Thx.

 

[arno]

The last time I look at the MS building footprints, they were just footprints with no attributes at all.

 

[Clutch Cargo]

"I would have to check if I can add some kind of sorting to the data."

That would be great Arno if you could take a look. Nothing fancy... maybe just sort by starting at the northern most point (adding some sort of flag/attribute?), working its way South until it has its 11 parts. Maybe a 2nd flaf where one could sort West to East. That would be quite manageable.

Happy Easter
Marcus

 

[arno]

It would require that all data has to be read and stored to be able to sort things. So I think it will slow it down quite a bit (and use more memory).

 

[Clutch Cargo]

I see... and that could be very well be true. However, if we break it down it only has to be done one time and I can use it easily after that. If left as it is, we have to re-run the entire 3.5GB 138 times (on our current project). Hence, the fix could be a huge time saver. If I have to double my RAM from 32GB to 64GB we can/will easily do this if required for a new version. Besides California, we have to do this for a slated 10 more US states... a multi-year project.

If time is short for this, I can easily understand as it seems everybody want a part of you, ha! What's it written in? Would it be possible to get the code?

 

[arno]

Hi,

I'll see if I can find some time, it would be an interesting exercise. The tool is written in C#.

 

[Clutch Cargo]

Well, if ya wanna have fun, I would say add a flag (is that the correct lingo?), if you want North-to-South, a 2nd flag if you want West-to-East, and a third flag as to how many parts you want to break it into.

 

[arno]

I haven't forgotten this thread. I'm almost done implementing the sorting. Just need to fix a few issues and test performance.