Problems using UK Ordnance Survey shapefiles with sceproc - can you help?

[radiofreeeurope]

Hi guys

First, a thousand thanks to Arno for his magic scenproc tool. It really is great.

I've had some success experimenting with the OSM shapefiles from Geofabrik. Next I wanted to try something more systematic: a complete set of autogen tiles for the whole of the UK.

For this purpose, I wanted to use the GIS data produced by Ordnance Survey, the UK's national mapping agency. The 'OS VectorMap District' product provides coverage for the whole of the UK at a scale of 1:15000. It is available in ESRI format, and includes separate layers with accurate polygons for every building and woodland in the country. Obviously this data is much more complete than the OSM equivalent at this time, and since 2010 it has been available free of charge for personal and commercial use as part of the UK government's OS OpenData initiative:

http://www.ordnancesurvey.co.uk/oswebsite/products/os-vectormap-district/index.html

The product is distributed in 100km-square tile zip-files (based on the OS UK grid system). Individual files are available on request from OS, but there are also cached copies to download here:

http://parlvid.mysociety.org:81/os/

As a test, I downloaded the 'TQ' file (London and the South-East) and imported the Woodland layer into QGIS. I cropped and saved a small test selection as a new UTF-8 WGS84 shapefile. I ran this shapefile through scenproc and although it loads correctly, scenproc seems to get stuck at the splitgrid stage, with no visible progress after several hours. I initially suspected this may be due to the higher definition of the polygons, but when I ran a similar test with only a single, simple polygon selection in the shapefile, the result is the same. I think therefore there must be a compatability issue which escapes me?

Would someone more knowledgeable than myself be kind enough to spare the time to have a look?

Test file: http://parlvid.mysociety.org:81/os/vmd-vector-2011-03/vmdvec_tq.zip
Tech spec: http://www.ordnancesurvey.co.uk/oswebsite/docs/user-guides/os-vectormap-district-user-guide.pdf

Many thanks!
Jeff Pollard

http://www.youtube.com/user/FreeTheRadio

 

[arno]

Hi Jeff,

I think the problem is that the data is not using latitude longitude coordinates, but the UK national grid. This coordinate system is in meters. As a result of that the splitting into the small AGN size tiles will take very long indeed, because these are calculated in degrees.

If you tell scenProc that the data is in the UK national grid, it should work fine however. I think this is the right coordinate system (although I found a few called UK national grid):

http://spatialreference.org/ref/sr-org/6832/

If I enter the Proj4 string I found on the link above in my SPC file and then run it, it takes about 10 minutes to do the work. Given the quite big area covered by the SHP file that is quite normal. Here is the test SPC I used:

IMPORTSHP|TQ_Woodland.shp|*|+proj=tmerc +lat_0=49 +lon_0=-2 +k=0.9996012717 +x_0=400000 +y_0=-100000 +ellps=airy +units=m +no_defs
#
SPLITGRID|AGN

 

[radiofreeeurope]

Voortreffelijk!

I was convinced that I should convert the original shapefile to WGS84 beforehand, it never occurred to me to try to run it straight into scenproc as-is with the original string. Working perfectly now. I better get cracking on my batch script then...

Thanks Arno!

For anyone else with similar coordinate-related issues, you can also look up your exact projection string in QGIS, under Layer - Properties - Metadata - Layer Spatial Reference System.

 

[BASys]

Hi Folks

Jeff -
Excellent to see someone working on this.

Back during FSX's various betas
I looked at generating UK tree coverage.
i.e. pre OpenData initiative.

In case you've not already investigated this aspect,
IIRC, NERC/JNCC, or somesuch similar,
have datasets giving tree types/densities/etc.
which would enhance any outputs
over OS's basic coniferous/non-coniferous/mixed classifications.

If required, info's on an old PC, which I can dig out this weekend.

From a quick look,
the Forestry Commission might also have useful data.

Look forward to anything you produce.

HTH
ATB
Paul

 

[radiofreeeurope]

Thanks Paul.

Yes, in fact the OS 'Woodland' shapefiles have no attribute detail at all, just the polygons themselves. I did find a GB National Forest Inventory shapefile on the Forestry Commission website you linked to, which does have some tree-type attribute info (e.g. broadleaved, conifer, shrubland, etc). It doesn't provide as much coverage or polygon definition as the OS shapefiles, but it should be relatively straight-forward for me to 'intersect' the two sources together in QGIS and then assign some relevant object IDs in ScenProc. For the time being, I'm just using generic building and tree IDs. If things go well then I'll look to add in some further detail (e.g. tree-types, building height, library objects, night lighting, etc).

Unfortunately I've found that the OS 'Building' shapefiles do generalise the built-up areas quite a lot. This can result in ScenProc generating one big building where many smaller ones should be. I initially tried using Post Office Address point data and Voronoi processing to split up the larger polygons beforehand, but I've found the most effective method is actually just to use the built-in filters in ScenProc (e.g. REPLACEPOLYGONBUILDINGRECTANGLES, FAREA, etc). I'm just running some tests now to determine which settings give the most convincing results.

Cheers
Jeff

 

[BASys]

Revolution-X

Hi Folks

Congratulations Jeff.

Wishing you every success with Revolution-X

Knock the spots of Fantasy Terrain eXperience Gobble X
at least for the UK, (ATM).

ATB
Paul

 

[=Hollywood=]

Just saw the video of Revolution-X today, nice work!