OSM

[WarpD]

Can ScenProc now directly read in OSM XML data?

If so... then the stupid question is... can I get OSM to give me XML for a specific U.S. state? I'm finding the SHP file data from geofabrik.de to be inaccurate when compared against the active OSM image itself.

 

[arno]

Yes, you can read the osm files directly. Be aware that the attributes are often different from the shp extracts.

I would not expect positional differences though.

 

[MyPC8MyBrain]

it is altered
geofab show's you exactly the OSM mapping to new features names and coulomb's
i was in OSM nightmare for three days trying to read directly from OSM XML; good luck with that
500 million records later; is where i gave up on it and killed the process

 

[arno]

I find the original osm file has more attributes in general. The shapefiles sometimes skip attributes you still want to use.

 

[MyPC8MyBrain]

how do you sort through it though?
my file decompressed to over 60GB of data

i may used the wrong tool/approach for this (used their wiki instructions)

i wanted to load the file to see what exist and whats the features names etc
but the process was so long; i killed it on day third after 500 million records were read
i thought it will stop at 50mil, than 100 mil, at 250mil i was drawing blanks

 

[arno]

Often I use another tool to first clip to the area I want to read. So I don't try to load so many gb.

Else specifying the bounding box in the import step should also work to limit things.

 

[WarpD]

Obviously it's possible to extract regions, such as the state of Virginia in the U.S. or Switzerland in Europe, etc. I wish I knew how to do that.

Can you still use QGIS Desktop with an OSM file?

 

[arno]

I use osmosis to filter out regions, but recent versions of gdal also work (ogr2ogr tool). My starting point is usually a downloaded osm file of a country or state.

 

[MyPC8MyBrain]

Thank you Arno
they must have monster clusters working these files over at geofabric
they generate these filtered OSM files on a daily basis

i am aware of the bounding box option
the problem i was facing is; i didn't know what sort of features are described there (couldn't find any documentation either)
i thought loading the file will allow me to study whats is possible with the features that it holds

i couldn't find much information on how to efficiently work and manipulate raw OSM files

@WarpD
QGIS claim they can work with raw OSM
from my experience; they can barely work with standard data files
joking aside; its a convoluted process i wasn't able to execute with QGIS

 

[arno]

This is an excellent source of information for which attributes are there:

http://wiki.openstreetmap.org/wiki/Map_Features

 

[arno]

From my experience QGIS doesn't read osm files that well. But since they also use gdal/OGR it might have improved now that OGR supports osm quite good.

 

[MyPC8MyBrain]

Thank you Arno
this is perfect and is what i was looking for

with QGIS it appears that one needs to build the DB over GRASS layer to work with raw osm

 

[arno]

GRASS is a very cool piece of software, but unfortunately it's not really easy to use

 

[WarpD]

I must confess that at this point I really don't know how to use an .osm file to create autogen with scenProc. Is it the 'same' as how I use the shp files or is it completely different?

I ask because I got this message:
"3:33 PM IMPORTOGR Warning Skipping feature: Too many tags in relation 148838"

What feature? LOL Here's what I ran:

# Ohio autogen
IMPORTOGR|E:\Documents\OpenStreetMap SHP Files\Ohio\ohio-latest.osm|*|*|NOREPROJ
#
SPLITGRID|AGN
#
CREATEAGNPOLYVEG|FTYPE=POLYGON;type=forest|{e8b937fd-a1f2-4bd5-8548-2c80d30102af}
CREATEAGNPOLYVEG|FTYPE=POLYGON;type=park|{e04669c0-9f7b-42e8-a2c7-eee870c59d8e}
CREATEAGNPOLYVEG|FTYPE=POLYGON;type=riverbank|{69ef7159-099e-46ba-ba86-07f5ef1e990e}
#
CREATEAGNGENBUILD|FTYPE=POLYGON;FROMFILE=buildings.shp;FWIDTH<20|{5ae04eb6-934c-4f63-bb48-5e7dee601212}|MAXRATIO=2
CREATEAGNGENBUILD|FTYPE=POLYGON;FROMFILE=buildings.shp;FWIDTH>20|{6089A0BD-CED1-4c47-9A9E-64CDD0E16983}
#
EXPORTAGN|FSX|D:\MegaSceneryEarth\Ohio\Texture

 

[arno]

Hi,

That warning is not too bad. I have seen it more often with OSM files. It's basically the OGR library telling you it can't read one feature. But since there are thousands you probably won't miss it.

The OSM file uses slightly different attributes, so you need to adjust your filters a bit. Try it like this:

# Ohio autogen
IMPORTOGR|E:\Documents\OpenStreetMap SHP Files\Ohio\ohio-latest.osm|*|*|NOREPROJ
#
SPLITGRID|AGN
#
CREATEAGNPOLYVEG|FTYPE=POLYGON;landuse=forest|{e8b937fd-a1f2-4bd5-8548-2c80d30102af}
CREATEAGNPOLYVEG|FTYPE=POLYGON;leisure=park|{e04669c0-9f7b-42e8-a2c7-eee870c59d8e}
CREATEAGNPOLYVEG|FTYPE=POLYGON;waterway=riverbank|{69ef7159-099e-46ba-ba86-07f5ef1e990e}
#
CREATEAGNGENBUILD|FTYPE=POLYGON;building=*;FWIDTH<20|{5ae04eb6-934c-4f63-bb48-5e7dee601212}|MAXRATIO=2
CREATEAGNGENBUILD|FTYPE=POLYGON;building=*;FWIDTH>20|{6089A0BD-CED1-4c47-9A9E-64CDD0E16983}
#
EXPORTAGN|FSX|D:\MegaSceneryEarth\Ohio\Texture

 

[WarpD]

Ok, will give that a try. I find the 'documentation' on OSM and it's XML content to be difficult to locate exactly what one seeks. Guess I just don't 'think' as they do or something. LOL

 

[WarpD]

This time it says "too many tags in relation 148838"

Is there a way for me to determine what 148838 was?

 

[arno]

That's the id of the feature in the osm dataset. Each feature has a unique number.

But you can ignore the warning. Scenproc will just skip that one feature and continue.

I find the osm documentation very good and useful. So I guess I think differently

 

[WarpD]

Ok... if I want to reduce what is read in from the osm file, that's what the 'attributes' field is in the IMPORTOGR line is for?

I was thinking it might run faster if I could restrict what in the OSM file is actually even looked at to begin with.

 

[arno]

Yes, if you enter landuse;leisure;building there it should only load what you use later in the script.