scenProc big area tips (1)

[arno]

When you are trying to process a big area in scenProc getting all vector data loaded can be hard. Either you will have some big shape files that take long to load or you need to do some pre-processing to slice up the data before. But did you know there is an easier way?

scenProc uses the OGR library to read the vector data. This library does not only support many formats, like shape files or GML, but it can also connect to geographical databases. So if you put all your geo data in such a database scenProc can read from it using the ImportOGR step. And only the data for the bounding coordinates that you specified will be read. Instead of a file name you need to specify a database connection string like this:

PG:”dbname=’databasename’ host=’addr’ port=’5432′ user=’x’ password=’y'”

Of course initially you need to setup the database and put your data in it. The connection string above is for a PostgreSQL database with the PostGIS extension installed. There is an easy installer to get all this installed quickly. And with ogr2ogr it’s also not so hard to put your data into the database.

So the next time you start a project where you will use scenProc to process a big areas consider setting up a geo database first to make your life a bit easier.

Continue reading...

 

[apollon01]

Hello,

I am aware that the original post from Arno is a bit dated but my question relates directly to it so I do not want to create a new thread.

I am indeed in the situation described in the original post: I have a big area with rather huge and complicated polygons. To process, I am using batch processing on LOD10 but even though to load some features takes eternity

Therefore, I have two options:
1) split the source vector files to match the LOD setting of my batch process - but this proves too complex and time consuming even though on a test area this approach works well
2) set up a database

I want to implement the 2) (also to learn something new).

I have saved my test shapefile to SQLite database (via qgis) but I have no idea about the syntax of the string to put into ImportOGR step to call the features from the SQLite database.

I would appreciate any help on this as though I am not completely new to working with gis data, I am completely new to the databases and related staff.

Thanks!

Milan

 

[arno]

Hi,

Although it doesn't help you much now, I have some updates in development that speed up reading complex polygons with holes. Somewhere in the next months that should make it to the development release.

I never used SQLLite databases myself, I would have to check the ogr manual on them first.

 

[arno]

I did a quick scan of the docs and I think for SQLite you can just enter the filename and that should be sufficient.

 

[apollon01]

Thanks for the info Arno.

I have already tried to read the SQLite database directly as you suggest. However, in the morning I found it in the same stage as I left it in the evening The shapefile took about 1h30 to load, just FYI.

I probably just let it work a couple of days (weeks ) more et voilà.

Milan

 

[arno]

Hi,

I'll see if I can do a little test here, need to make some SQLite file with some data then.

 

[apollon01]

Arno, do not bother just because of me. I am working on a work-around. It seems that no-one else is either
a) having issues / working with huge files, or
b) using SQLite files.

Regards,

Milan

 

[arno]

Hi,

I'm sure more people struggle with big areas.

I did a little test with converting a shp to SQLite, but it loaded much longer than the shp file. Maybe I did something wrong.

Like I wrote in the blog post I'm using a PostgreSQL database with satisfaction.