- Messages
- 34,535
- Country
Hi all,
In this thread I want to describe some of my experiences with processing big datasets with scenProc. I have tried a few different approaches in the last days and they didn't all work that well.
Let me first give a little more details about the dataset I used. It is a dataset that covers the entire country of the Netherlands. It's the vector data that is used to produce the official maps, etc. This dataset is organised in map sheets (each file is basically the data for one part of the printed map). In total it is about 20 GB of data.
So which approaches did I try?
Do everything at once
The first idea was to just load everything in scenProc at once and see how that goes. But I think you can imagine that this is not the best approach. It will take quite a lot of memory and doing the split grid step on so much data will also take long. And as soon as you run out of physical memory and start to do swapping it will become very slow. So this is not a good approach...
Limit area of reading
My second idea was to still load all the files every time, but apply a geographical filter to limit which area is actually read. This works better, since the memory usage is not so high. And by calling the configuration in batch mode I could every time process a different area. But in the end it was not that quite. Even for files that where outside of the area of interest it would sometimes still take 20 seconds or so to determine that nothing has to be read. When trying to do a lot of batch runs, that means quite some time is wasted.
It could be this experience only applies to the GML format that I used and that other formats take less time to determine there are no objects. But at least in my case it didn't work as fast as I wanted.
One file at a time
So I ended up just processing one GML file at a time. And in my batch process I went over all files in my source folder. Since my files cover map sheets, they will not align with the FSX autogen LOD grid. So this means that some autogen tiles will get features from multiple GML files. To prevent merging issues I simply used the IMPORTAGN step to load already existing autogen and then the new objects from the current files were added to it.
In the end this process went quicker than I expected. It took me less than 2 hours to process the entire country.
In this thread I want to describe some of my experiences with processing big datasets with scenProc. I have tried a few different approaches in the last days and they didn't all work that well.
Let me first give a little more details about the dataset I used. It is a dataset that covers the entire country of the Netherlands. It's the vector data that is used to produce the official maps, etc. This dataset is organised in map sheets (each file is basically the data for one part of the printed map). In total it is about 20 GB of data.
So which approaches did I try?
Do everything at once
The first idea was to just load everything in scenProc at once and see how that goes. But I think you can imagine that this is not the best approach. It will take quite a lot of memory and doing the split grid step on so much data will also take long. And as soon as you run out of physical memory and start to do swapping it will become very slow. So this is not a good approach...
Limit area of reading
My second idea was to still load all the files every time, but apply a geographical filter to limit which area is actually read. This works better, since the memory usage is not so high. And by calling the configuration in batch mode I could every time process a different area. But in the end it was not that quite. Even for files that where outside of the area of interest it would sometimes still take 20 seconds or so to determine that nothing has to be read. When trying to do a lot of batch runs, that means quite some time is wasted.
It could be this experience only applies to the GML format that I used and that other formats take less time to determine there are no objects. But at least in my case it didn't work as fast as I wanted.
One file at a time
So I ended up just processing one GML file at a time. And in my batch process I went over all files in my source folder. Since my files cover map sheets, they will not align with the FSX autogen LOD grid. So this means that some autogen tiles will get features from multiple GML files. To prevent merging issues I simply used the IMPORTAGN step to load already existing autogen and then the new objects from the current files were added to it.
In the end this process went quicker than I expected. It took me less than 2 hours to process the entire country.
