Islands disappear after multiple BGL conpilation

[MEPaquette]

Hi folks
I am having problem with SBuilderX compilation
When I first started my project of revising UTX water polygons and my lakes and rivers, everything seemed to be going great.
I started working on Lake Nipissing.
1) Used CvxExtract on file HP12615D found in C...FSX/Scenery/UTCanWater/Scenery/
2) Open up SBuilderX, and starte new project
3) Appended file HPXHP12615D.shp into new project
4) Then saves as 2 files. one names Nipissing_Exc and second as Nipissing
5) In "Nipissing_Exc", I would change the property of all required polygon to "Exclude_All_Water_Polygons"
6) In "Nipissing", I woul change the property of all required polygon to "Hydro_Generic_River_Perennial" and revise the altitude to appropriate real world altitude (or close to it)
7) I would then choose all polygons in "Nippising_Exc" and do the BGL Compile, and then move the "cvx_Nipissing_Exc.bgl to my FSX/Addon Scenery/MEP/Scenery/ directory
8) I then did the same with my "Nipissing" file.

This procedure was done many times so that I might verify my progress as I was going.
After completion of the whole lake, I found a couple of islands that were missing. I had the shorelines, but not the land. I figured I would get back to this later on as I wanted to proceed with the rest of my water system,
Everything was going goof until recently. I atrted finding a few more islands going missing. At this point, I would go back into my SBuilderX file, go the island polygon, change its property to "Set as outer", the go into "polygon" mode, re-select the island polygon and "Set as Hole" and select the main polygon. this procedure was working fine for a while.
On a few occasion when there were lots of island, this procedure would not work. I ended up re-drawing the main polygon and tie all island polygon to the new main polygon.

I then went back to my Nipissing file and attempted to fix up the few missing island. Now it seems the more time I do the BGL Complile the more islands go missing.

Does my multiple "bgl compile" be causing this problem? Or could it be that I am trying to do too big of an area in one file? Or could it actually be that my computer can't handle those big files and "dropping" the island information during the BGL compile " process?

Michel

 

[GaryGB]

Hi folks
I am having problem with SBuilderX compilation
When I first started my project of revising UTX water polygons and my lakes and rivers, everything seemed to be going great.
I started working on Lake Nipissing.

1) Used CvxExtractor on file HP12615D found in [FSX_install path]\Scenery\UTCanWater\Scenery\ <-- Please use 'back-slash' characters in Windows paths !
2) Open up SBuilderX, and start a new project
3) Appended file HPXHP12615D.shp into new project
4) Then saves as 2 files. one names Nipissing_Exc and second as Nipissing
5) In "Nipissing_Exc", I would change the property of all required polygon to "Exclude_All_Water_Polygons"
6) In "Nipissing", I would change the property of all required polygon to "Hydro_Generic_River_Perennial" and revise the altitude to appropriate real world altitude (or close to it)
7) I would then choose all polygons in "Nipissing_Exc" and do the BGL Compile, and then move the "cvx_Nipissing_Exc.bgl to my [FSX_install path]\Addon Scenery\MEP\Scenery\ directory
8) I then did the same with my "Nipissing" file.

This procedure was done many times so that I might verify my progress as I was going.
After completion of the whole lake, I found a couple of islands that were missing. I had the shorelines, but not the land. I figured I would get back to this later on as I wanted to proceed with the rest of my water system.

Everything was going good until recently. I started finding a few more islands going missing. At this point, I would go back into my SBuilderX file, go the island polygon, change its property to "Set as outer", the go into "polygon" mode, re-select the island polygon and "Set as Hole" and select the main polygon. this procedure was working fine for a while.

On a few occasions when there were lots of islands, this procedure would not work. I ended up re-drawing the main polygon and tie all island polygon to the new main polygon.

I then went back to my Nipissing file and attempted to fix up the few missing island. Now it seems the more time I do the BGL Compile, the more islands go missing.

(Could) my multiple "bgl compile" be causing this problem? Or could it be that I am trying to do too big of an area in one file? Or could it actually be that my computer can't handle those big files and is "dropping" the island information during the BGL compile " process?

Michel

Hi Michel:

First let us clarify the way in which your CVX vector Exclude BGLs and replacement BGLs must be utilized by FS.


You must place the SBuilderX-created "Nipissing_Exc.bgl" in a separate sub-folder chain, and link this to its own Scenery Library GUI Area layer.

Your CvxExtractor-derived / edited "Nipissing.bgl" must be kept in a separate sub-folder chain linked to its own Scenery Library GUI Area layer.


* Note that your "Nipissing.bgl" Scenery Library GUI Area layer must be positioned ABOVE the "Nipissing_Exc.bgl" Scenery Library GUI Area layer.

* Both your "Nipissing.bgl" Scenery Library Area and "Nipissing_Exc.bgl" Scenery Library Area must be positioned and kept ABOVE ALL UTX layers.

This sequence allows us to first Exclude existing specified content- then Replace it with new / edited / custom scenery content.


Please attach (2) separate screenshots using FS SDK TMFViewer, (1) for each of your "Nipissing.bgl" and "Nipissing_Exc.bgl" files.


PS: I am unaware of a limit to allowed Point / Vertex, Poly-line, or Polygon count in either SBuilderX or FS SDK SHP2VEC causing omission of content.

GaryGB

 

[MEPaquette]

Gary
On the TMF viewer file, do you want polygon fill on or off?
Do you also want QMID grids on or off, and iff on, what level?

 

[GaryGB]

Gary
On the TMF viewer file, do you want polygon fill on or off?
Do you also want QMID grids on or off, and if on, what level?

Hi Michel:

While custom water polygons are most commonly going to be subject to clipping at LOD-11 boundaries due to the default SDK SHP2VEC compiler CVX vector XML parameter values used, the custom content in FSX and UTXCan may have been generated using larger quad sizes, so you must first inspect them in SDK TMFViewer to see what mix of quad grid sizes actually apply to water polygons ...before CvxExtractor derivation / being Appended to SBuilderX.

Generally speaking, in large water bodies, a mix of larger LOD sized quads are used, and only when more detailed modifications of water polygons are required along shoreline areas of terrain, a mix of increasingly smaller quads will be used, down to a minimum of LOD-9 / QMID-11 in size.


So, IIUC, results may vary with excluding polygons having the same targeted GUID, over the extent of a quad larger than LOD-9 / QMID-11 in size ...because it was originally authored with a larger assigned "clip" attribute.


For example, you may recall I pointed out that UTXCan utilized a LOD-6 Area quad clip level rather than the default LOD-5 Area quad clip level:

https://www.fsdeveloper.com/forum/threads/fsx-and-ultimate-terrain-canada-and-us.456742/post-912448


To see what the actual quad sizes are for Hydro polygons in a custom CVX vector BGL file, load it into TMFViewer without any grid displayed.

Next, in TMFViewer Menu > View > Vector Data > Polygons > un-check "Fill"; note the 'mix' of actual quad 'clip' sizes used for Hydro polygons.

Screenshots of your CVX vector BGLs cited above, should match this example screenshot using what, IIUC, is the same 'local' CVX BGL from FSX default:

GaryGB

 

[MEPaquette]

Gary
First screen shot is the UTX file HP12615D.bgl zoomed into Lake nippissing. The QMID vary from level 13 to Level 15

This next screen shot is my Nipissing_Exc.bgl

This next one is my Nipissing.bgl

now here is the top view of the west arm area of lake Nipissing

And this is after I have added a folder for the excuded files and added to my scenerie file:

BTW, I have double checked the locations of the FTX files and they are for the West coast of Canada and US.

Michel

 

[GaryGB]

Hi Michel:

I believe it may be helpful to get you some more room at the top of the Scenery Library GUI to work in.

Since you now have disclosed that you use multiple OrbX add-ons, this will be more challenging due to how OrbX add-ons work, even though you have indicated those add-ons are intended for local display in the West coast of Canada and US.

Please try a basic procedure required, IMHO, if attempting to modify scenery on a FS computer with a OrbX add-on installed:


Let us remember we must first create a FTX Central Scenery Library Insertion Point to leave room at the top for our files.

Otherwise, FTX Central keeps automatically positioning OrbX add-ons at the top layer of the Scenery Library ... above your Mod.


Here's an excerpt from one of my posts on FTX Central "Insertion Point" option for OrbX scenery layers in Scenery Library:


By default, FTX Central always re-inserts FTX layers to a position at- / near the top of- P3D Scenery Library GUI stack of layers.


By default, when FTX Central utility runs automatically following installation / configuration of OrbX FTX scenery packages:

* Custom entries are added to the 'active' Terrain.Cfg if they are not already present.

* Several OrbX FTX scenery Area layers are added to the 'top' position of the 'active' FS Scenery Library GUI stack of layers.


Because by default, OrbX FTX scenery Area layers are added to the 'top' position of the P3D Scenery Library GUI, custom scenery may not render properly (if at all), so one should allow for non-OrbX add-on or personally-created custom scenery to be loaded later by placing them above the OrbX FTX scenery Area layers ...at- or nearer to- the 'top' position of the FS Scenery Library GUI.


Each time OrbX FTX Central utility is run OrbX FTX scenery Area layers are again automatically added to the 'top' position of the 'active' FS Scenery Library GUI stack of layers by default.


One can change that default behavior of OrbX FTX Central utility by configuring the OrbX FTX scenery Area layers to be automatically placed at a lower position in the 'active' FS Scenery Library GUI stack of layers by:


OrbX FTX Central Menu > Tools > Scenery Library Configuration > FTX Scenery Library Insertion Points


NOTE: Newer versions of OrbX FTX Central utility may be subject to a different menu / dialog configurations


This results in an assigned position, above which OrbX FTX scenery Area layers will NOT be inserted ...until a new Insertion Point is assigned within that FTX Central utility dialog.


* Please move- and keep- your own custom add-on content to an Area layer position above any / all OrbX FTX entries


Personally, I prefer moving any / all OrbX FTX Area layer entries down below UTX entries wherever possible.


PS: Your edited version of Lake Nipissing looks rather nicely detailed.

I shall comment further on your latter post above, after the procedure I recommended here ...has been completed.

GaryGB

 

[MEPaquette]

Gary
That procedure was easy to do. FTX is now below UT Exclusion.
But I am also thinking that since I am not doing any flying in the West at the moment, I should just turn FTX off
Might be less taxing on my scenery
But by moving FTX off and even going as far as turning it off, I go back to my West Arn and islands are still missing.

And the "nicely detailed" is not by me but by UTX. All I did was change the property on their polygons. But I agree that UTX did a very good job.

Michel

 

[GaryGB]

Hi Michel:

Because as stated, I do not have UTXCan installed, I do not have the same file sets available to examine.

I shall instead attempt to illustrate the method to be used with UTXCan by showing how UTX excludes CVX vector content in UTX USA.


Generally speaking, prior to replacement of original scenery content, CVX vector Excludes must first be made and linked to an Area layer that is:

* Above the content to be excluded

* Below the content that is intended to replace the excluded content


Since I do not have the same BGLs available for UTXCan as you do, I shall instead illustrate this by use of a nearby Area to the West.

Take for example:

{FSX_install_path]\Scenery\0301\scenery\cvx2515.bgl (...a FSX default BGL file)

UTX USA provides a (custom) Exclude for this Area in this BGL:

{FSX_install_path]\Scenery\UtExcl\Scenery\HPU2515U.BGL

AFAIK, this is the required method for excluding entire quads of CVX vector objects via specific Exclude GUIDs with quad sized polys.

This process is distinct from use of a small triangle poly to overlap / intersect (1) specific Poly-line and/or Polygon to exclude it (only !) up to the 'clip' extent of the quad it is located within.


I do not have your files to test, but IIUC, by using UTXCan CVX Hydro line / poly objects inside quad objects when all are assigned an exclude GUID, they may exclude each other.


In SBuilderX:

I recommend instead using quad-sized / triangle-based exclude methods as otherwise shown above, to exclude CVX vector objects

To do this, draw new polygons over quad areas to be excluded, assign a Exclude GUID, and "slice" them to quad (QMID-11 ?) borders.


NOTE: Export a SBuilder Exchange file (*.SBX) before compiling a BGL, as 'some' CVX data may otherwise not be stored in *.SBP files.

Then delete the Appended UTXCan CVX Hydro lines / polys from the 'Exclude' project before compiling to a CVX vector BGL.

Leave only the quad-sized / shaped polygon objects with the assigned exclude GUID in the Exclude project ...when it is to be compiled.


Here is some info copied from a reply I made to a prior thread, on a way to simplify this work-flow for making the quad exclude polys.

Of particular note will be the method of SBuilderX author Luis de Sa' to "Slice to QMID-11" ...in his video: normans.swf


BTW: If you also later decide to further customize the replacement CVX vector content otherwise provided by FSX default and/or UTXCan, performing a "Exclude and Replace" for ALL CVX vector scenery content within entire QMID-11 sized quads is the most commonly utilized approach to customizing shorelines and water bodies ...due to the greater likelihood of successful display in place of default or other 3rd party scenery content.

However, that approach does sometimes require a greater amount of work.

http://www.fsdeveloper.com/forum/threads/sbuilderx-exclusion-question.432328/

http://www.fsdeveloper.com/forum/threads/adjusting-only-part-of-island-coastline.434472/


Alternatively, one can use specific excludes in the form of small triangles that intersect CVX vector scenery objects, and replace default CVX vector scenery objects with ex: copies of that original (excluded) scenery content derived from 'Appended' SHP files exported from CVXExtractor (but in a substantially-modified form) ...to create ones custom scenery that is aligned to the real world aerial imagery.


However, one may be compelled to still utilize the GUID for "Exclude and Replace" ALL CVX vector scenery content within entire QMID-11 sized quads approach.

This is because there may be GPS Hydro Polygons that cannot otherwise be excluded by local specific excludes in the form of small triangles that intersect CVX vector scenery objects; reportedly that is due to the GPS Hydro Polygon 'exclude' option not having been included in the FSX SDK SHP2VEC compiler GUID processing feature set by MS Game Studios / ACES.

http://www.fsdeveloper.com/forum/threads/how-to-find-out-and-exclude-default-polygons-used.277450/

http://www.fsdeveloper.com/forum/threads/replacing-an-island.16237/


To verify the GUID for CVX vector objects seen in TMFViewer:

* Right-click on the polygon border or poly-line, then select "Identify Vector Features"

* Correlate the GUID with SDK CVX Vector references:

https://www.prepar3d.com/SDKv5/sdk/world/terrain/terrain_overview.html#The Shp2Vec Tool

https://www.prepar3d.com/SDKv5/sdk/world/terrain/terrain_overview.html#Vector Attributes

https://www.prepar3d.com/SDKv5/sdk/world/terrain/vector_shape_properties_guids.html


In some scenarios, one may have to use a QMID-11 quad-sized "Park Polygon" method to perform a successful "Exclude and Replace" of ALL CVX vector scenery content within your project area where GPS Hydro polygon vector GUID attributes are seen in FS SDK TMFViewer ...as described above in the linked thread.


But, chances are, your initial efforts will focus solely on excluding / replacing Hydro Poly / shoreline content and fixing Altitudes of same.

So, to further reply to your threads about replacement of default CVX vector objects with ones custom CVX vector content:

If one already has a large 1-piece shoreline poly-line and hydro polygon, one can use the tools in SBuilderX to 'slice' and modify it so that it can still be used, but as 2 separate pairs of objects: (1) shoreline and (1) hydro polygon each ...with each pair in 2 adjacent QMID-11 quads:

http://www.ptsim.com/forum/viewtopic.php?p=5785&sid=1f45de417f712cfa137705b2ca803464

http://www.ptsim.com/sbuilderx/videos/normans.swf


NOTE: The above linked "ShockWave Flash" video, when double-clicked, should play automatically via the Adobe Flash plugin for most modern web browsers; if it does not, it can be played by Windows' own default 'Media Player'.

MS Edge and Chrome reportedly can play *.SWF ShockWave Flash files, but this may be disabled by security apps.

How to play a *.SWF ('Adobe ShockWave Flash') file in Windows Media Player

Change the default program for opening a *.SWF file to Windows Media Player by:

1.) Right-click the *.SWF file > Properties > General > Change > Windows Media Player.

2.) After this 'Windows file association' is done, double-click *.SWF files to play them in Windows Media Player.

GaryGB

 

[MEPaquette]

Gary
Let me try to summarize what I can understand from your last post.
To do my exclusion file for Lake Nipissing, I should draw a large polygon around the lake and surrounding area directly connected to the lake which will have the same elevation, then "clip" this large polygon to QMID11. Then I delete all UTX polygons, after which I assign the proper GUID to exclude just the water polygons.
After this I can compile the exclude file.

Or are you saying I should do an exclude that matches all the quads that contains any parts of my area I want to work with? In this case I have a total of 12 quadrants that I would need to do, which includes a whole lot more small lakes and rivers that I don't want to change.
Here is a clip from TMF Viewer showing the extend of small lakes and rivers that would be affected by a "quadrant" exclude (QMID 11 level).

On the other hand, I could do a series of small triangles that intersects all the UTX water polygons I want to change, then delete all UTX polygons. In this case, I would end up with hundreds of triangle, and I am likely to miss the very small polygons which might be as small as as 21m x 21m ( and sometimes even smaller).

My bottom line questions is: will the exclude method have an effect on my "missing" island on my new rebuild cvx file?

PS: This is the UTX exclude file for the area I am working on:

It covers an area from latitude 47.00 D to 47.84 D and longitude 78.75 D to 82.49 D. (QMID Level 7, and LOD 5)

And this is the original cvx2615 from FSX

Michel

 

[GaryGB]

Gary
Let me try to summarize what I can understand from your last post.
To do my exclusion file for Lake Nipissing, I should draw a large polygon around the lake and surrounding area directly connected to the lake which will have the same elevation, then "clip" this large polygon to QMID11.

Then I delete all UTX polygons, after which I assign the proper GUID to exclude just the water polygons.

After this I can compile the exclude file.

No, I am not recommending that you draw a (1) large polygon around the lake and surrounding area directly connected to the lake, which will have the same elevation, then "clip" this large polygon to (1) QMID-11 quad polygon.

As my above example using the UTX-USA file shows in SDK TMFViewer, the method used is to create multiple exclude polygons matching each quad border;
this is done in SBuilderX as shown in the author's *.SWF video via the "Slice To QMID-11" feature.

Or are you saying I should do an exclude that matches all the quads that contains any parts of my area I want to work with? In this case I have a total of 12 quadrants that I would need to do, which includes a whole lot more small lakes and rivers that I don't want to change.

I recommend that you instead do an exclude that matches borders of each QMID-11 quad that contains any parts of the area you want to work with.

On the other hand, I could do a series of small triangles that intersects all the UTX water polygons I want to change, then delete all UTX polygons. In this case, I would end up with hundreds of triangle, and I am likely to miss the very small polygons which might be as small as as 21m x 21m ( and sometimes even smaller).

The individual CVX vector object excludes within individual quads would be much too cumbersome to do.

And there is no need to delete all polygon objects via triangles., when a single QMID-11 exclude polygon sliced to the borders of each quad can be used.

My bottom line questions is: will the exclude method have an effect on my "missing" island on my new rebuild cvx file?

My current hypothesis is: Yes, eliminating the possibility of "excludes ...excluding ...excludes" ...'may' eliminate the "missing" island anomaly at run time.

Because your prior described method (IIUC) assigned the exclude attribute GUID to all CVX vector objects in a copy of the original UTXCan BGL, rather than simply excluding either:

* An entire QMID-11 quad using a precisely "sliced" QMID-11-sized / shaped rectangular polygon

* Less than an entire QMID-11 quad using a less-than-QMID-11-sized / shaped rectangular polygon

* Less than an entire QMID-11 quad using multiple less-than-QMID-11-sized / shaped rectangular polygons and/or triangles


PS: Have you downloaded / installed the FreeMesh X 30 Meter freeware (or other 3rd party) terrain mesh for Canada yet ?

GaryGB

 

[MEPaquette]

Gary
I will do a couple of test.
Working with only the West Arm area, I will delete the quadrants to QMID 11 and and then redo another updated file with the revised elevations.

I will also go back to my original exclude file and break out the West Arm area use that in conjugation with the new updated file for West Arm and see if there is any difference

I have a feeling that I manipulated the original Nipissing update file too often and it might have gone corrupt

Report back later

Michel

 

[GaryGB]

Gary
I will do a couple of test.
Working with only the West Arm area, I will delete the quadrants to QMID 11 and and then redo another updated file with the revised elevations.

I will also go back to my original exclude file and break out the West Arm area use that in conjugation with the new updated file for West Arm and see if there is any difference

I have a feeling that I manipulated the original Nipissing update file too often and it might have gone corrupt

Report back later

Michel

Well... in a FS Development-adapted version of a popular stage performer's well-wishing expression "Break A Leg", Ill just say "Break An Arm" (carefully).


PS: Remember that UTX uses "layers" of CVX vector content in multiple BGLs, and if the same GUID is used in a different "layer" BGL, AFAIK, the same FSX rendering engine rules may apply, wherein CVX vector objects with matching GUIDs that overlap / intersect each other within a assigned QMID quad clip level, may exclude each other.

Bear in mind that one may not need to have used a specified GUID for an exclude, as like ...excludes ...like ...if non-exclude objects overlap / intersect.

So, due to the multi-dimensional "layered" methods UTX uses, we must think in 3D layers of 2D content.

But, as we see in screenshots of FSX default CVX vector BGLs, objects with different GUIDs, can- and do- overlap / intersect without excluding each other.


I am not certain if CVX vector objects with matching GUIDs, but different assigned clip levels (with different quad sizes) ...can still exclude each other.


Unfortunately SBuilderX never implemented an easy-to-use "layer" feature set, so if one wishes to attempt something similar to "layers", one must Append all UTXCan *.SHP files 1-at-a-time, while also assigning each a different color ...to 'try' distinguishing them from each other in SBuilderX' workspace.

Otherwise, one must work with source *.SHP files in a GIS application that has a "layer" feature set, and which can assign colors to imported *.SHP files.

Once vector objects are edited, they can be Appended as *.SHP files to SBuilderX to assign the CVX vector GUIDs required for use by FSX SDK SHP2VEC.

GaryGB

 

[MEPaquette]

Gary

I was finaly able to complete my “experimentation” with different “exclude” styles.
First, here is the initial SBX files with all 3 layers of water polygons appended for the 4 QNID11 size squares that covers only the West Arm area of lake Nipising:

Pink is HP12615D
Green is H22615D
Turquoise is HP32615D

First test was doing an exclusion file to the QMID 11 size squares as you instructed. Here is the TMViewer file of this exclusion file:

Then I proceede to create a new SBX file for the West Arm area starting with a fresh SBX file with HP12615C shape file.
Here is my new file in TMViewer

Now here is the “top view” in FSX

You will notice that there are 2 groups of island missing even though they are part of the .bgl file. Also, all the lakes in around the area fixed up are also missing. I would need to go to ALL of the surrounding lake and assign the water property to them.

Using my style of exclude as shown below in TMViewer and using another brand new updated file:

This is what I get in top view of FSX

Same problem as with quadrant exclude except I am not losing all the lakes outside of my work area except for maybe a few nearby

From this, my conclusion is that doing excludes by QMID 11 “quadrant” on by changing polygons of existing file yields the same error in my final scenery.

Afterwards, I did further experiments on these 2 areas and found 3 thinks I can do to get my islands back.
Sometimes just offsetting one of the outer polygon points that intersects the next polygon will bring back my islands
Sometimes simply go to each island polygon, switch them to “outer” and back to “hole”
But as last resort, I redraw the outer polygon, delete original one and “tie” back all of the island polygon to this new main one,

After this, I redid my whole Nipissing (this also gave me the opportunity to change my original elevation of 196m to 195m which is the value shown on the paper cope that I received).
After redoing all of Nipissing, including West Arm and the upper section of the French River up to the dam, I only had 6 polygons in witch islands were not showing up.

My conclusion is that the “errors” or coming from the UTX shape files. No programming can be perfect all of the time.

My reasoning for this is that while doing the Ottawa River, the section in file HP12615C had maybe 1 error in the few islands there was. Then when I got to file HP12615D, in the first section I did, only the polygons on the Ontario sides were missing all of the islands while on the Quebec side none wee missing. I still have 3/4 of the Ottawa River in this file to do, and the way I see it, all I can do is deal with the missing islands as they come up.

Michel