P3D v4 Elevation Data including "missing data"

[ScKevin]

Hi,

currently I am restarting an old project to make it compatible for P3D V4. For the DEM, I got a new (finally) higher resolution. However I have a problem and I am asking myself the whole time if that is possible and when yes, how.

My mesh data does not fill a full square GeoTiff file. So after exporting my mesh from Global Mapper, I get the mesh of my area of interest and the whole area around is filled with "zero". So after resampling my files, I see in tmfviewer my project area with hills and depressions and the area around is just flat land (which is absolutely not true).
My solutions were these until now, but none of them worked:
- Adding a fade out in the GeoTiff to avoid hard "lines" in the surface structure of P3D.
- Using a blendmask to show resample.exe the area to compile.

Now I have another idea, but I am unsure how to make it work:
I want to make sure that Global Mapper writes in my exported GeoTiff file an inexistant value for the DEM (for example -30000) for the whole area where I do not have a high resolution. Afterwards in the resample.exe-ini-file I want to add that NullValue, so that resample.exe does leave everything out of the project area apart. Anyone knows how to make this idea work? Anybody knows if this is even possible?

thank you for your help!
Kevin

 

[=rk=]

Resample.exe can use a grayscale image to synthesize elevation and with an image editor you could probably merge your quality data with default or fictional elevation data to fill the area of coverage. Terrabuilder is a shell application that uses this concept, you may want to look into it.

 

[rhumbaflappy]

A common practice is to make the null value in the DEM as -32768. Then set the Null Value in the INF as -32768. The following also appears to work:

http://www.fsdeveloper.com/forum/threads/fsx-set-nullvalue-for-terrain-mesh.3827/

 

[ScKevin]

Hi both of you!

After reading several threads I also have another idea. I wanted to import more DEM with a different resolution to get rid of the cliffs from the .tiff-squared-image. I wanted to solve this like this. My area of interest has a high resolution. Then the more we are getting to the border of the square the resolution of the raster will get lower, so that the lowest resolution in the raster can be found on the 4 sides of my squared raster .tif-image. Is that possible and does this really a better job?

@=rk=:
Thanks for your replies. As I do not want to buy another application I stick to what I have, so the option to use TerraBuilder will not be a solution

@rhumbaflappy:
I tried now the two options stated in your suggested thread and it did not worked as expected, so here my little workflow, in order that we can find the faulty part.
To save the DEM-file into a format resample.exe can use, I use GlobalMapper.
First I import my original raster file into GlobalMapper. Then I change the projection to WGS84. Afterwards I go in the Options dialog to change under Alter Elevation Values the parameter Replace Invalid with and there I set -3257. After doing this, I export the raster file to GeoTiff-->24-bit RGB (Full Color, May Create Large Files). I also check the option to Generate a TFW (World) file and the option to Generate a PRJ file. Then I click OK to export the raster.
My .ini-file for the DEM looks like this:

[Source]
Type               = GeoTIFF
Layer               = Elevation
SourceDir          = "SourceData"
SourceFile         = "DEM.tif"
NullCellValue      = -3257

[Destination]
DestDir             = "Output"
DestBaseFileName    = "DEM"
DestFileType        = BGL
LOD                 = Auto

After executing the .ini-file via resample.exe I get the DEM.bgl file. However in that file, all parts where I do not have mesh are set to an altitude of "0", so they are basically just flat.

thank you for your help
Kevin

 

[=rk=]

Terrabuilder is free and I found no need for the "advanced" functionality in order to perform the procedures discussed.

I wanted to import more DEM with a different resolution to get rid of the cliffs from the .tiff-squared-image. I wanted to solve this like this. My area of interest has a high resolution. Then the more we are getting to the border of the square the resolution of the raster will get lower, so that the lowest resolution in the raster can be found on the 4 sides of my squared raster .tif-image. Is that possible and does this really a better job?

Kevin, the secret trick of Dick's null value is that it incorporates the functionality of a blend mask, which normally affects the data geometrically, directly into the data filter. So, essentially, you can do exactly as you describe. I am pretty sure you would have to complete each pass as a distinct .bgl, but they could all be in the same folder/layer, because they would be complementary.
Reading through your procedure, it would seem to me that you and the software have different opinions about whether or not regions with 0' elevation should be considered invalid data. I don't know the software, but would it not allow you to edit specific elevations? If not, simply rewrite your .ini to conform to the format in the linked thread:

[Source]
Type               = GeoTIFF
Layer               = Elevation
SourceDir          = "SourceData"
SourceFile         = "DEM.tif"
NullCellValue      = 0

[Destination]
DestDir             = "Output"
DestBaseFileName    = "DEM"
DestFileType        = BGL
LOD                 = Auto

 

[crushnik]

I have a similar issue as the original poster. Creating 10m mesh around San Diego, and the USGS data does not include any of Mexico or beyond our territorial waters. The actual ground looks great, as does the ocean which has a "0" elevation value. But where the TIF has no data, I get elevation data of -575 and there are massive cone-shaped spikes of ground along the edge of the data (which I think comes from resampling).

After fiddling with every possible tweak in the INF, I am now wondering if there is a flight sim-friendly way to tell OSGeo4W to plug in -32767 into the no data holes when I'm reprojecting to WGS84? I've completed the entire United States to release as an uncompressed 10m mesh freeware package, but the Mexican/Canadian border tiles are all that is holding me back. See https://www.fsdeveloper.com/forum/threads/terrain-mesh-on-us-mexico-border.450385/#post-866367

 

[GaryGB]

Hi Chris:

One must "back-fill" data voids causing spikes / pits where there is an abrupt change in elevation data sets (or missing DEM data).

It is not sufficient to use a NO_DATA attribute for QMID grid vertex points in terrain mesh, as FS 'may' show spikes / pits at run time.

Use this data set for back fills: SRTM V3+ data set

LP DAAC - NASA Shuttle Radar Topography Mission (SRTM) Version 3.0 (SRTM Plus) Product Release

LP DAAC - Land Processes Distributed Active Archive Center - Sioux Falls, SD

lpdaac.usgs.gov

EarthExplorer

Query and order satellite images, aerial photographs, and cartographic products through the U.S. Geological Survey

earthexplorer.usgs.gov

These threads discuss how to back-fill a given higher resolution DEM with the latest extensively 'fixed' SRTM V3+ data set:

https://www.fsdeveloper.com/forum/threads/having-troubles-with-airport-mesh.445990/post-829259

https://www.fsdeveloper.com/forum/t...atten-scenery-object-moved.446355/post-832795


FYI: The load sequence / layering for the 2 DEM source data sets may be different in GIS applications other than Global Mapper

PS:

A common practice is to make the null value in the DEM as -32768. Then set the Null Value in the INF as -32768. The following also appears to work:

http://www.fsdeveloper.com/forum/threads/fsx-set-nullvalue-for-terrain-mesh.3827/

AFAIK, the value used in GIS and in FS SDK Resample is actually based on the 'binary counting' value of -32767

GaryGB

 

[crushnik]

Thanks @GaryGB ! As I study the material you sent, I think you've given me enough help this year to qualify for a Christmas card at minimum!

I found what might be something along the lines of the file you referenced above in Earth Explorer, under NASA LPDAAC Collections --> NASA SRTM (SRTM 3) Collections --> NASA SRTM3 SRTMGL1, but I noticed in the first link you sent that it was SRTM Plus data, and what I found was SRTMGL1 https://lpdaac.usgs.gov/products/srtmgl1v003/

It gives me options to download metadata and a HGT file, which I've never messed with or heard of. Given that I've got to download and re-do many dozens of scenery tiles to fill in the holes in Canada and Mexico, I really hope this is something I can do somewhat quickly with OSGeo4W commands or else my months-long nationwide scenery project is probably going to be canceled, after thinking I already crossed the finish line.

 

[GaryGB]

Hi Chris:

Global Mapper's built-in download features the above cited data set under the name:

"SRTM Worldwide Elevation Data (1-arc-second Resolution, SRTM Plus V3)"


USGS / NASA / JPL have newer names, but an overview of current SRTM derived data sets is discussed here:

Shuttle Radar Topography Mission

SRTM - OpenStreetMap Wiki

wiki.openstreetmap.org

https://www.arcgis.com/home/item.html?id=cadb028a356046479fcda5207a235560

"SRTM v3.0 (SRTM Plus) is the newest version, published in 2015 by NASA as a part of NASA Making Earth System Data Records for Use in Research Environments (MEaSUREs) project, which incorporates topographic data to fill the gaps or voids in earlier versions of SRTM data. For the void filling with the Delta Surface Fill algorithm, ASTER DEMs have been used as auxiliary data source, or interpolations have been applied. Many variants of DEM are available in SRTM v3.0, with SRTMGL1 being one of the key products from SRTM v3.0. ‘GL1’ on its name stands for “Global 1-arc second”. It provides regularly spaced DEM grids of 1 arc-second (approximately 30 meters) and covering 80% of Earth’s landmass, between 60° North and 56° South. This product is divided into 1° x 1° latitude and longitude tiles in “geographic” projection, as shown here."

EarthExplorer has this data under NASA LPDAAC Collections > NASA SRTM (SRTM3) Collections

AFAIK, these "should" all be the void-filled data sets you may wish to use.

[EDITED]

Basically one would fill the LODs via use of both this newer 30 Meter (1-Arc Second) SRTM set and the 10 Meter DEM set your project already uses, with a (merged) 2-layer source for SDK Resample along the CONUS borders, regardless of whether there are 10 Meter data voids to be back-filled.

This is important to do, because otherwise FS may render spikes / pits at run time when a mesh BGL does not have all contiguous Quad Grid mesh levels derived from 'more proximate' resolutions of elevation posting intervals in source data that are 'practical' for SDK Resample to create via over / under -sampling, when output of a span of LODs for a defined Geographic coverage extent ...is 'forced' via the *.INF file.

So, effectively, one is "over-sampling" 30 Meter / 1-Arc second data to create new elevation data points at 10 Meter intervals when filling voids.

It also results in smoothing of transitions from 10 Meter points onto 30 Meter points, and eliminates spikes / pits at voids and edges of data sets.

This merged 2-layer source approach 'may' be necessary with a fall-back to FS' default DEM4KM BGL file for a smooth FS LOD mesh display without also using SRTM 90 Meter data, since ACES reportedly utilized earlier revisions of SRTM data sources for areas along USA - Mexico borders ...which may be less compatible with a newer / more detailed 'composite' elevation data set used for your 10 Meter terrain mesh project.

One may instead enable FS default terrain mesh display in areas "South of the border" to 'ease a transition' from your 10 Meter mesh into the DEM4KM mesh, but IMHO, only if one does not still see any spikes / pits at data voids "North of the border"- and edges of data sets- at run time.

[END_EDIT]

NOTE: Global Mapper has an option to interpolate layers into a single output file derived from 2 layers of DEM data at different resolutions.

I assume QGIS / OSGEO4W may be able to do the same, so you can avoid the complexity of creating a multi-source *.INF file for SDK Resample using multiple layers of DEMs at differing resolutions within that INF ?

GaryGB

 

[crushnik]

Thanks Gary! With an Earthdata.NASA.gov login I can go to this site https://e4ftl01.cr.usgs.gov/MEASURES/SRTMGL1.003/ and entering the desired area, such as "N33W118" in the search bar. Easy enough.

Writing multisource INF files isn't too much extra work, so if I can figure out a gdal_translate command to convert the HGT file to a GeoTiff, I'd be back in business! I'm coming up empty so far... do you have any suggestions, @JRobinson ?

 

[crushnik]

Just to see what it would do, I tried

gdalwarp -of GTiff -co "INTERLEAVE=PIXEL" -t_srs "+proj=latlong +datum=WGS84" -r cubic "C:\Users\crush\Downloads\N33W118.hgt" "C:\Users\crush\Downloads\N33W118.tif"
Creating output file that is 3601P x 3601L.
Processing C:\Users\crush\Downloads\N33W118.hgt [1/1] : 0Using internal nodata values (e.g. -32768) for image C:\Users\crush\Downloads\N33W118.hgt.
Copying nodata values from source C:\Users\crush\Downloads\N33W118.hgt to destination C:\Users\crush\Downloads\N33W118.tif.

Looks like USGS and NASA name their tiles differently: when I loaded my 1/3 arc-second GeoTiff (with no Mexico data) and the new 1 arc-second GeoTiff, the new file appeared above my old file (both named N33W118). But this just might work... keeping my fingers crossed while I figure out how to prioritize the layers in my Multisource INF!

 

[crushnik]

[Source]
Type= MultiSource
NumberOfSources = 2
  
[Source1]
Type=GeoTIFF
Layer=Elevation
SourceDir="."
NullValue=-32768
SourceFile="n33w118.tif"
PixelIsPoint=0
ulxMap=-118.000555555555
ulyMap=33.0005555555552
xDim=9.25925925258432E-05
yDim=9.25925925258432E-05

[Source2]
Type=GeoTIFF 
Layer=Elevation
SourceDir="."
NullValue=-32768
SourceFile="N32W118.tif"
PixelIsPoint=0
ulxMap=-118
ulyMap=33
xDim=0.000277777777777779
yDim=0.000277777777777779

[Destination]
DestDir="."
DestBaseFileName="n33w118"
DestFileType=BGL
UseSourceDimensions=1
FractionBits=3
LOD=4,12

I tried NullValue=0,0,0 and NullValue=-32768 and this still gets me the same exact results as I had in this post: https://www.fsdeveloper.com/forum/threads/terrain-mesh-on-us-mexico-border.450385/#post-866367

 

[GaryGB]

Looks like USGS and NASA name their tiles differently: when I loaded my 1/3 arc-second GeoTiff (with no Mexico data) and the new 1 arc-second GeoTiff, the new file appeared above my old file (both named N33W118). But this just might work... keeping my fingers crossed while I figure out how to prioritize the layers in my Multi-source INF !

Hi Chris:

It may be less complex to first merge multiple resolutions of DEM data into (1) back-filled output elevation-type GeoTIFF via a GIS application.

[EDITED]

SRTM 'global' HGT files provide missing data used to back-fill, so one gets a full set of elevation data points for tiles along the CONUS borders where the USGS 10 Meter DEM clips off at the borders of USA data sets.

[END_EDIT]

For info on how to prioritize layers (aka "sub-sections") by resolution in a SDK Resample multi-source *.INF, you may wish to review this thread:

Different LODs within one BGL using Resample.

Hi guys. I've been trying to optimize my photoscenery and I'm wondering if it is possible to place multiple LODs within one BGL, instead of using different BGLs, each with a higher resolution. I know that resample will automatically choose the best LOD according to the resolution of your...

www.fsdeveloper.com

GaryGB

 

[GaryGB]

Hi Chris:

Considering that most of the global GIS community apparently is still being 'a bunch of Wi(nA)mps' when it comes to converting HGT files to GeoTIFFs, I looked for more recent DEM data available that you can download, and found this 'new' composite set via ESRI which offers GeoTIFFs:

Global Terrain DEM (Esri)

A multi-resolution layer providing access to elevation values for use in analysis with functions for slope, aspect, and hillshade.

arc-gis-hub-home-arcgishub.hub.arcgis.com

The 'dynamic' web portal is able to extract only 'filtered' Map view areas specified by end users, so output extents 'may' not be 1x1 degree tiles.

NOTE: Map zoom level dynamically determines output resolution of ex: GeoTIFF DEMs, so consistent extraction extents may require 'API' inputs.



Attached is Global Mapper Metadata and a screenshot of a ('Filtered Dataset') 100 MB GeoTIFF download from that portal at USA - Mexico border at:

Tijuana International Airport (IATA: TIJ, ICAO: MMTJ), sometimes referred to as General Abelardo L. Rodríguez International Airport, in Tijuana, Baja California, Mexico,

FILENAME=[path]\x_____xGZs29vmZ8sfPmyH8MeBC3Q..x_____x_ags_ce42276e_f9a8_428e_ba03_fb775e4ed9e2.tif
DESCRIPTION=x_____xGZs29vmZ8sfPmyH8MeBC3Q..x_____x_ags_ce42276e_f9a8_428e_ba03_fb775e4ed9e2.tif
UPPER LEFT X=-13029388.286
UPPER LEFT Y=3841840.911
LOWER RIGHT X=-13014251.863
LOWER RIGHT Y=3826704.488
WEST LONGITUDE=117° 02' 41.9510" W
NORTH LATITUDE=32° 35' 46.6689" N
EAST LONGITUDE=116° 54' 32.4490" W
SOUTH LATITUDE=32° 28' 54.0062" N
UL CORNER LONGITUDE=117° 02' 41.9510" W
UL CORNER LATITUDE=32° 35' 46.6689" N
UR CORNER LONGITUDE=116° 54' 32.4490" W
UR CORNER LATITUDE=32° 35' 46.6689" N
LR CORNER LONGITUDE=116° 54' 32.4490" W
LR CORNER LATITUDE=32° 28' 54.0062" N
LL CORNER LONGITUDE=117° 02' 41.9510" W
LL CORNER LATITUDE=32° 28' 54.0062" N
PROJ_DESC=Mercator / GOOGLE / meters
PROJ_DATUM=GOOGLE MAPS (SPHERE RADIUS 6378137)
PROJ_UNITS=meters
EPSG_CODE=EPSG:3857
COVERED AREA=162.83 sq km
NUM COLUMNS=5000
NUM ROWS=5000
NUM BANDS=1
PIXEL WIDTH=3.028 meters
PIXEL HEIGHT=3.028 meters
MIN ELEVATION=13.302 meters
MAX ELEVATION=399.256 meters
ELEVATION UNITS=meters
BIT DEPTH=32
SAMPLE TYPE=32-bit Floating Point
PCS_CITATION=ESRI PE String = PROJCS["WGS_1984_Web_Mercator_Auxiliary_Sphere",GEOGCS["GCS_WGS_1984",DATUM["D_WGS_1984",SPHEROID["WGS_1984",6378137.0,298.257223563]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Mercator_Auxiliary_Sphere"],PARAMETER["False_Easting",0.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",0.0],PARAMETER["Standard_Parallel_1",0.0],PARAMETER["Auxiliary_Sphere_Type",0.0],UNIT["Meter",1.0]]
GT_CITATION=PCS Name = WGS_1984_Web_Mercator_Auxiliary_Sphere
GEOG_CITATION=GCS Name = GCS_WGS_1984|Datum = D_WGS_1984|Ellipsoid = WGS_1984|Primem = Greenwich|
PHOTOMETRIC=Greyscale (Min is Black)
BIT_DEPTH=32
SAMPLE_FORMAT=Floating Point
TILE_WIDTH=128
TILE_HEIGHT=128
COMPRESSION=None
PIXEL_SCALE=( 3.02789, 3.02789, 1 )
TIEPOINTS=( 0.00, 0.00, 0.00 ) --> ( -13029389.800, 3841842.425, 0.000 )
MODEL_TYPE=User-defined system
RASTER_TYPE=Pixel is Area
VERT_DATUM=None Specified

CAVEAT: The downloaded extract is a single 'classic' GeoTIFF *.TIF without a *.TFW.

Thus, one must either retain an intact Geo-referencing Metadata GeoTIFF file header, or use a GIS application to separately create a *.TFW for it.


Note this example PIXEL HEIGHT / WIDTH = 3.028 meters, which can be under-sampled by specified LOD output in the SDK Resample *.INF file.

I hope this proves useful for your current and future project tasks.

GaryGB

 

[crushnik]

As I explore new options you've presented, I'm wondering... what is telling Resample to convert "no data" pixels to -575 elevation data?

If I could put a stop to converting Mexican "no data" areas into 500-foot holes instead of default scenery showing through, that would be spectacular. Converting NASA's HGT file to a GeoTiff was pretty easy, and I tried merging the high and low resolution GeoTiff files, but the pixel scale in my new INF file is 0.00027, so it looks like my high resolution tile became a low resolution tile with extra steps. Also I noticed a black strip of pixels on the eastern edge of the low-res data, but it didn't show up after Resample.

I'll continue studying potential solutions.
Sincere thanks as always Gary,

 

[GaryGB]

Hi Chris:

If you post URLs here to directly download the (2) original USGS source files you are trying to merge above, perhaps we may better assist you.

GaryGB

 

[crushnik]

https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/13/TIFF/n33w118/USGS_13_n33w118.tif (1/3 arc-second data GeoTiff)

HGT file requires a NASA login, so I attached it below. I converted it to a GeoTiff using the code above (https://www.fsdeveloper.com/forum/threads/elevation-data-including-missing-data.440670/post-867195)

Also, it appears that USGS names its tiles on the top-left coordinate while NASA names theirs based on the bottom-left coordinate, which explains the two different names for the files.

Thanks!

 

[GaryGB]

https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/13/TIFF/n33w118/USGS_13_n33w118.tif (1/3 arc-second data GeoTiff)

HGT file requires a NASA login, so I attached it below. I converted it to a GeoTiff using the code above (https://www.fsdeveloper.com/forum/threads/elevation-data-including-missing-data.440670/post-867195)

Also, it appears that USGS names its tiles on the top-left coordinate while NASA names theirs based on the bottom-left coordinate, which explains the two different names for the files.

Thanks!

Hi Chris:

AFAIK, the same EarthExplorer login can simply be repeated for the second NASA / JPL login if one has already registered / used USGS login credentials.

Full Display of N32W117.SRTMGL1​

EarthExplorer

Query and order satellite images, aerial photographs, and cartographic products through the U.S. Geological Survey

earthexplorer.usgs.gov

https://e4ftl01.cr.usgs.gov/MODV6_Dal_D/SRTM/SRTMGL1.003/2000.02.11/N32W117.SRTMGL1.hgt.zip

Full Display of N32W118.SRTMGL1​

EarthExplorer

Query and order satellite images, aerial photographs, and cartographic products through the U.S. Geological Survey

earthexplorer.usgs.gov

https://e4ftl01.cr.usgs.gov/MODV6_Dal_D/SRTM/SRTMGL1.003/2000.02.11/N32W118.SRTMGL1.hgt.zip


I have downloaded, and will be examining / testing the above data set today; I'll post a reply ASAP.

GaryGB

 

[GaryGB]

Hi Chris:

Are these the type of anomalies you see along the USA - Mexico Border when the above cited DEMs are merged to make a LOD-12 terrain mesh?

Default FSX:

GaryGB

 

[JRobinson]

I downloaded the 1/3 arc-second USGS_13_n33w118.tif , reprojected it to WGS84, did gdalinfo on that and determined the NoData Value to be -999999. I put NullValue = -999999 in the .inf and resampled it. I see no anomalies like that and there's nothing showing -575m in TMFViewer (or the sim). There is if I don't use NullValue = -999999 however. I see variations in elevation along the Mexico border but I believe that's to be expected where the LOD12 mesh joins the default LOD10 mesh.

Jim