ScenProc script suggestions

[crushnik]

I finally figured out a script that creates autogen from the millions of Microsoft building footprints into P3D! I am wondering if anyone has a good script that would work to incorporate variables into the autogen, similar to the impressive results people get from OSM data.

Is it possible to tell ScenProc to use zoning data from OSM or perhaps P3D's population density file and assign the results to my shapefile from the Microsoft data to create taller buildings in urban areas for example?

The script I am using (see below and don't laugh - it's just a trial run) is extremely primitive and looks decent enough, apart for some interesting rows of storage buildings that are two stories high and have very high-pitched roofs. It would be nice to figure out a script that makes large commercial buildings have flat roofs, industrial buildings look like factories and warehouses, etc.

Also, is there any way to take a large waterbody shapefile (from USGS) and use it to create watermasks for the 3.75-minute ortho tiles I am using?

#
ImportOGR|C:\Users\Prepar3D v4 Add-ons\Wild Weasel Missouri\Data\Missouri Buildings.shp|-90;-89;37;38|*|AUTODETECT
#
SplitGrid|AGN|*|*
#
SETAGNBUILDINGHEIGHT|*|0.6;0.3;0.1;0.0
#
CreateAGNGenBuild|FWIDTH<20|{5ae04eb6-934c-4f63-bb48-5e7dee601212}
CreateAGNGenBuild|FWIDTH>20|{f5d3c9b7-59af-4df1-be66-b06933ebd3ba}
#
ExportAGN|P3D v2|C:\Users\Prepar3D v4 Add-ons\Wild Weasel Missouri\Texture

 

[JRobinson]

Also, is there any way to take a large waterbody shapefile (from USGS) and use it to create watermasks for the 3.75-minute ortho tiles I am using?

That one I can answer (but I don't know how to do it using sceneproc).
You need to know the exact corner coordinates and pixel sizes of your source imagery. After you've reprojected it to WGS84 run gdalinfo on it (I'm talking about using the command line console "OSGeo4W Shell" BTW) to obtain that information:

gdalinfo "C:\path\to\file\t2307_1719_WGS84.tif" > "C:\path\to\file\t2307_1719_WGS84_gdalinfo.txt"

In the resulting "t2307_1719_WGS84_gdalinfo.txt" you'll find the corner coordinates as well as a line that looks like this:

Pixel Size = (0.000001147037836,-0.000001147037836)


Then use gdal_rasterize to burn your shape onto a .tif:

                                                                                              West        South        East       North
gdal_rasterize -burn 255 -burn 255 -burn 255 -tr 0.000001147037836 -0.000001147037836 -te -116.1014557 48.6993455 -116.0870363 48.7084082 -ot BYTE "C:\path\to\file\waterbody.shp" "C:\path\to\file\t2307_1719_WGS84_blendmask.tif"

The two numbers after -tr (target resolution) are your pixel sizes, just copy them from t2307_1719_WGS84_gdalinfo.txt (backspace the comma out and put a space between them). "-burn 255" is intentionally repeated, once for band 0, band 1 and band 2 (RGB). You'll end up with a black image with white where your water bodies are so you'll need to invert it in PhotoShop. Add it to your .inf using the same ulxMap, ulyMap, xDim, and yDim you used for your main imagery (or copy & paste it to "alpha1" and omit adding it to your .inf entirely).

 

[arno]

Hi,

The short and simple answer is yes, you can do all of this

Now the slightly longer answer.

If you look at section 5.4 of the manual you see a script where different buildings types are identified. In this script industrial and commercial buildings are classified by checking if there are inside polygons of these zones. You can use this approach with OSM data as well. The script also shows some strategies to vary roof types.

If you have raster data with population density you can use the ImportGDAL step to load the data and then the AddCellAttributeSampleRaster step to add an attribute to a cell based on the population density. The script in section 5.7 of the manual then shows how you can use such an attribute to vary the building height.

Reading population density data directly from the BGL in the sim is not supported at this moment.

As for the water polygons, scenProc can do that as well, but since I''m busy with a new ModelConverterX release I haven't finished the work yet to put that feature in the publicly available release. But I have made a complete photoreal scenery with water and blend masks with only scenProc.

 

[crushnik]

That one I can answer (but I don't know how to do it using sceneproc).
You need to know the exact corner coordinates and pixel sizes of your source imagery. After you've reprojected it to WGS84 run gdalinfo on it (I'm talking about using the command line console "OSGeo4W Shell" BTW) to obtain that information:

gdalinfo "C:\path\to\file\t2307_1719_WGS84.tif" > "C:\path\to\file\t2307_1719_WGS84_gdalinfo.txt"

In the resulting "t2307_1719_WGS84_gdalinfo.txt" you'll find the corner coordinates as well as a line that looks like this:

Pixel Size = (0.000001147037836,-0.000001147037836)


Then use gdal_rasterize to burn your shape onto a .tif:

                                                                                              West        South        East       North
gdal_rasterize -burn 255 -burn 255 -burn 255 -tr 0.000001147037836 -0.000001147037836 -te -116.1014557 48.6993455 -116.0870363 48.7084082 -ot BYTE "C:\path\to\file\waterbody.shp" "C:\path\to\file\t2307_1719_WGS84_blendmask.tif"

The two numbers after -tr (target resolution) are your pixel sizes, just copy them from t2307_1719_WGS84_gdalinfo.txt (backspace the comma out and put a space between them). "-burn 255" is intentionally repeated, once for band 0, band 1 and band 2 (RGB). You'll end up with a black image with white where your water bodies are so you'll need to invert it in PhotoShop. Add it to your .inf using the same ulxMap, ulyMap, xDim, and yDim you used for your main imagery (or copy & paste it to "alpha1" and omit adding it to your .inf entirely).

Once I entered all the code into Notepad, I'll basically just have to change the filenames, copy, then paste into the command line. Works great so far, thanks!

 

[crushnik]

Apart from the harsh edges, which I imagine there is a way to feather into the scenery, my beloved Mississippi River is looking much better!

Is there a way to have one large-scale watermask (covering an entire state, for example) and tell Resample to only use a particular section of it for my smaller-scale ortho tiles?

 

[arno]

I never tried to use a watermask with different extends from the imagery. But if they are both geotiff it might just work

 

[JRobinson]

I've never tried it either but my guess is you'd exceed the 2Gb limit for .tifs anyway making a Mississippi-sized .tif with any usable resolution.

 

[arno]

A tiff over 2 gb is no issue, as long as your bgl stays under that limit.

 

[JRobinson]

It is actually because resample keeps returning an error like this indefinitely if the .tif is over 2Gb. That's been my experience anyway.

E:\1S1 Porthill\PR\USGS2009_CAJ3_Rykerts_60cm\Rykerts_60cm_pr
ocessed.tif: 2572386680: Invalid strip byte count, strip 0.
GeoTIFF: read error.
E:\1S1 Porthill\PR\USGS2009_CAJ3_Rykerts_60cm\Rykerts_60cm_pr
ocessed.tif: 2572386680: Invalid strip byte count, strip 0.
GeoTIFF: read error.
E:\1S1 Porthill\PR\USGS2009_CAJ3_Rykerts_60cm\Rykerts_60cm_pr
ocessed.tif: 2572386680: Invalid strip byte count, strip 0.
GeoTIFF: read error.
E:\1S1 Porthill\PR\USGS2009_CAJ3_Rykerts_60cm\Rykerts_60cm_pr
ocessed.tif: 2572386680: Invalid strip byte count, strip 0.
GeoTIFF: read error.
E:\1S1 Porthill\PR\USGS2009_CAJ3_Rykerts_60cm\Rykerts_60cm_pr
ocessed.tif: 2572386680: Invalid strip byte count, strip 0.
GeoTIFF: read error.
E:\1S1 Porthill\PR\USGS2009_CAJ3_Rykerts_60cm\Rykerts_60cm_pr
ocessed.tif: 2572386680: Invalid strip byte count, strip 0.

 

[crushnik]

What I am thinking is that instead of clipping a 3.75-minute chunk out of a larger hydro shapefile to burn a watermask, would I be able to tell the program to pull just what I need and not have to do this process countless times?

Example: in ScenProc I can tell the program to load the shapefile I made for the entire state of Missouri which has millions of buildings. Instead of working with that massive file, I tell ScenProcFrom to load the shapefile, but only the specific 1 degree by 1 degree tile that I need and quickly turn it into autogen (originally I was using QGIS and making a separate shapefile for each grid -- much like what I am hoping to avoid in the above paragraph). A couple of minutes later, I can just change the lattitude and longitude extents and have another grid of autogen.

I would love to be able to do something that simple with watermasks if at all possible.

 

[arno]

Hi,

That's what I do when I make the watermasks with scenProc. I get imagery for a specific tile, make the watermask for it and compile with resample.

 

[GaryGB]

Hi Chris:

IIUC, you want to:

* Use a lesser number of "regional" SDK Resample Land-Water and/or Blend 'Mask' 8-Bit gray-scale *.TIF files for larger areas

...and:

* 'sample' only certain Geographic areas of those data sources for SDK Resample "local" output BGLs of smaller areas ?


If, so, AFAIK this can be done using the SDK Resample [Destination] Georeferencing Parameters to set 'limit of sampling region in degrees' values in the "local" *.INF file:

http://www.prepar3d.com/SDKv4/sdk/world/terrain/terrain_overview.html#The Resample Tool

GaryGB

 

[crushnik]

Hopefully that is the answer I've been looking for. If not, to clarify (what I admit is a confusing message): I would like to make one 1x1 lat/long watermask TIF. One square degree of scenery takes 64 3.75-minute ortho images. Instead of clipping the same 'master' watermask file 64 times into 64 separate greyscale TIF files, I'd prefer to have some code that tells Resample to just use these extents from the master watermask instead, saving me somewhere around 63 very tedious steps.

Thanks for giving me hope Gary!

 

[crushnik]

I just realized JRobinson's suggestion works great on individual tiles, but not for my large-scale application: Each tile will have very slightly different pixel sizes, so a massive master watermask won't work.

From two neighboring 3.75-minute tiles (37N089W 44-northeast and 37N089W 44-southeast):
Pixel Size = (0.000008983152808,-0.000007147640000)
and
Pixel Size = (0.000008983152808,-0.000007141700126)

 

[JRobinson]

What's the file size of these 3.75-min .tifs? What is the px/meter resolution? What sort of pixel sizes are you coming up with? Can you warp 4, 8, or 16 of them together and make one giant .tif? If you could do that you could run gdalinfo on that and make a considerably larger watermask, maybe reduce the number from 64 per 1 deg x 1 deg section to 4, 8, or 16?

 

[crushnik]

Four Missouri and Grand Canyon TIF files are about 1 gig when combined -- California files are roughly three times larger. Pixel size for what I'm working right now is 6963 x 8750 and that varies slightly with lattitude. I struggled with merging the GeoTIF raster images in QGIS before, but made some headway just a few moments ago on merging vectors when I combined two shapefiles for this project. I'll try merging the ortho images again (now that I know more about settings) because if I can get anywhere, that's DEFINITELY the way to go!

Maybe it just takes a really long time to merge rasters.

EDIT: Four 26-megabyte JP2 tiles took almost ten minutes to merge (seems longer when the progress bar never leaves 0%)... And the result looked like I converted the image to something you'd see on a Super Nintendo game -- maybe all the stretching and warping to fit on a curved surface?

 

[JRobinson]

Sounds like a PITA, I'd try the OSGeo4W Shell:

gdalwarp -of GTiff -co "INTERLEAVE=PIXEL" -t_srs "+proj=latlong +datum=WGS84" -r cubic "C:\path\to\file\source1.jp2" "C:\path\to\file\source2.jp2" "C:\path\to\file\source3.jp2" "C:\path\to\file\source4.jp2" "C:\path\to\file\output_WGS84.tif"

If you get "ERROR 6: No translation for Mercator_Auxiliary_Sphere to PROJ.4 format is known." try this:

gdalwarp -s_srs EPSG:3857 -t_srs EPSG:4326 -of gtiff "C:\path\to\file\source1.jp2" "C:\path\to\file\source2.jp2" "C:\path\to\file\source3.jp2" "C:\path\to\file\source4.jp2" "C:\path\to\file\output_WGS84.tif"

You can add as many sources as you want, I'd keep adding .jp2s until the output is as close to 2 Gb as you can get without going over. They need to be adjoining obviously and the output shouldn't have big black voids (3 sources across the top and two across the bottom will leave a huge black void on the bottom right for example). It does take a while and the WGS84 output will be distorted, that's normal. You may also see black wedges around the edges, they won't hurt anything either.

 

[crushnik]

Keeping my fingers crossed. I'll owe you a lot of beer if this works!

 

[crushnik]

Sounds like a PITA, I'd try the OSGeo4W Shell:

EDIT: HOLD ON, I SEE WHAT I DID WRONG... processing now.

Here's what went down:

C:\>gdalwarp -of GTiff -co "INTERLEAVE=PIXEL" -t_srs "+proj=latlong +datum=WGS84" -r cubic "C:\Users\Mary Beth\Downloads\m_3809063_ne_15_1_20150817_20151021.jp2" "C:\Users\Mary Beth\Downloads\m_3809063_nw_15_1_20150824_20151021.jp2" "C:\Users\Mary Beth\Downloads\m_3809063_se_15_1_20150824_20151021.jp2" "C:\Users\Mary Beth\Downloads\m_3809063_sw_15_1_20140721_20141009.jp2" "C:\Users\Mary Beth\Downloads\m_3809064_ne_15_1_20150817_20151021.jp2" "C:\Users\Mary Beth\Downloads\m_3809064_nw_15_1_20150813_20151021.jp2" "C:\Users\Mary Beth\Downloads\m_3809064_se_15_1_20150817_20151021.jp2" "C:\Users\Mary Beth\Downloads\m_3809064_sw_15_1_20150817_20151021.jp2"

ERROR 1: Output dataset C:\Users\Mary Beth\Downloads\m_3809064_sw_15_1_20150817_20151021.jp2 exists,
but some command line options were provided indicating a new dataset
should be created.  Please delete existing dataset and run again.

 

[JRobinson]

It assumed the last .jp2 in the list was intended to be the output file. You need to specify an output file last, ie "C:\Users\Mary Beth\Downloads\output_WGS84.tif".