Shading and smoothing issues

[Skoker]

http://imgur.com/a/v6k0W

I've been making a model and have been having some shading and smoothing issues in the test exports. I've tried with both the P3D 3DS MAX and FSX SP2 GMAX SDKs . I'm not happy at all with how it has been turning out, but both produced similar results, so I can only assume it has to do with the actual model. Any help greatly appreciated.

 

[tgibson]

For the nose section vs the rest of the fuselage, select both parts and then add an Edit Mesh modifier. Then go to Polygon submode and do your smoothing on both at the same time.

Even better, Attach the nose to the rest of the fuselage, weld the common vertices, and then smooth. That should eliminate the shading errors if the normals of adjoining polygons of the two parts do not have the same normals (i.e. do not point the same direction).

 

[Skoker]

For the nose section vs the rest of the fuselage, select both parts and then add an Edit Mesh modifier. Then go to Polygon submode and do your smoothing on both at the same time.

Even better, Attach the nose to the rest of the fuselage, weld the common vertices, and then smooth. That should eliminate the shading errors if the normals of adjoining polygons of the two parts do not have the same normals (i.e. do not point the same direction).

Did that and it is still having the issues.

 

[Milton_Shupe]

It seems that if you have done the suggested things, and have checked that edges do not need to be turned, that maybe someone else's eyes may help.
Could you share the source of the nose and the front part of the fuselage for others to take a look?

I am curious why the fuselage seems to have specular lighting showing and the nose does not. Do you have a material assigned with those qualities to the fuselage and not the nose?

EDIT: Can you post a screenshot of the mesh with edges turned on in wireframe mode.

 

[Skoker]

I found a solution/work around. I just needed to apply the reset XForm modifier and collapse the modifier tree. I don't think the exporter likes the way I built those surfaces.

http://i.imgur.com/zCkBoJ2.jpg

Different issue/something I could use a little help on is that the smoothing in my window area isn't right despite all being in the same smoothing group. Suggestions on how to fix?

http://i.imgur.com/RQ6sm2f.jpg

http://i.imgur.com/YS8PbEU.jpg

 

[fsintegral]

Haha ! No wonder why no answer so far. Shadowing issue around (boolean) windows are one of the most time consuming tweaks I've encountered so far...

... the more since what I'm about to share is probably the worst way you could imagine how I would deal with them. If someone knows a better way which would work in most cases, I'm eager to learn aswell.

So, context :
I work in GMax which runs well on my laptop. So I'm sorry, I can't give you the equivalent in Max or whatever tool you're using (obviously not GMax)

To replicate your issue, I started with a simple fuselage, and I created three Windows "blocks" to cut into the fuselage using the Boolean Compound Object.

This is what I've got As you can see, GMax is shitty sometimes : it wasn't able to compute the third window. Then I deleted the polygon cuts. Oh ! I'm working exclusively in Polygon Mode. Don't know what you're using, but in GMax Poly mode, you know for sure how triangles edges are laid on or wether some are missing. This will be very important below...

As you can see, polygons are not yet triangles here. So I convert my Object to type Patch, in order to force GMax calculate each actual triangles, then convert back the object to type Polygon.

This is the result ! How UGLY it is, isn't it ? Same smoothing group, but it doesn't render well because all diagonals (edges) are messed up.
This is somewhat similar to your issue I guess. So I think you just cut holes in your fuselage.

Look carefully in Vertex mode. On the left, you have green arrows showing the 6 vertices I used to create the corner of a Window. However, upon the boolean cut, only the first corner to be processed were correctly cut, but the 7 remaining were not that well done (red arrows showing extra vertices)
I've shown you this process in GMax to highlight the fact that boolean cuts are prone to failure (in Gmax that is) so I don't recommend to use it for more complex things than cutting a control surface (flaps or ailerons)
=> Check again in your 3D modeling tool. If you have tons of vertices dangling around, restart your windows from scratch...

 

[fsintegral]

How to get window cuts the other way, with the least amount of polygons, while keeping details and precision high ?
=> Do it vertex per vertex ! Seriously ? YES !

So I start by placing the vertices of the top left corner of ONE window in a vertical plan (ie, I'll get a flat window to start with) I use the initial boolean cut to get the vertices aligned. Use the side view to move your vertices around.

I do the same on the bottom left side to have nearly half a flat window. Then I create a polygon, which I duplicate then mirror. Be carefull : In Gmax, you can't mirror an object through the usual mirroring tool. You must/should detach the polygon(s), clone it manually, then use the parametric mirroring to mirror the entire object. Backface shading issues in FS after export are the result of the wrong use of the mirror button.

Then move the mirrored polygon to match the edges of your actual window. Complete the missing polygon to get a perfectly vertical window.
Next, draw horizontal edges linking each pair of aligned vertex (delete the vertical ones) and draw diagonal edges to get triangles. Be sure to have all diagonals going the same direction. Cyan arrows in the picture below. Same for top and bottom sides. This process is very important if you want a smooth shading when you're done with this windows chapter.

However, leave the center square without any diagonal as below (red arrow) That's because we're going to add extra vertices to reflect the curvature of the fuselage.

And here are the "extra" vertices :
I use Edge mode/Divide, then just add 4 vertex on each side of center section of the window. It doesn't matter they are not aligned (yet) because they are all in the same plan. That's why the window has to be flat and vertical to start with. Of course, I'm going to align them to get horizontal edges linking them by pairs...

And here are them, aligned, linked, and diagonals added !

Oh ! Of course, select each pair of vertices, and move them in the vertical axis to have them nearly at the same interval. You don't want to have too obvious sizes difference between the five rectangles.
Go to front/back view mode, and start selecting each vertex, move them along the horizontal axis to conform with the fuselage curvature.

From this :

you should get this :

Don't hesitate to go on with the zoom. You must have a nice curve, without any sharp turn or contra turn. Keep in mind the windows has to be merged back with the fuselage, so, you should leave room for the top and bottom edges to maintain a nice curvature.

And here is your Window, curved. Apply shading to have an early look.

Clone the polygons to get a secund window, and align it with the next one in side view. You get this :

Now, leave your Windows object, and select your fuselage containing the initial (wrong) windows. Select the vertices of one window and those representing the left side of the one next to it :

Delete those. You get part of the fuselage missing, and your two windows floating on top.

Select you windows object again, and go in polygon creation mode to create the fuselage section separating two windows. You'll start by creating flat polygons involving four vertices per polygon in a manner to get horizontal edges between them.

And when you're done with that piece of fuselage, draw diagonal edges to get actual triangles. Did you notice ? Each diagonal has the same orientation.

Early look on the fuselage panel :

 

[fsintegral]

Now that you have your two windows placed, go back on the fuselage, then delete all extra vertices to have a clear view of the windows editing.

Merge your windows with the fuselage.

Now, select two vertices of your actual fuselage from one end of the cylinder, those that would contain the curvature of the windows, and clone them in the longitudinal axis to meet the (vertical) axe of your first window.

Do the same for your second window (use vertices coordinates to get precise alignment)
Create 6 polygons in order to complete the void between your windows and the fuselage. Bear in mind that the 4 vertices you created are not actually linked to fuselage polygons. They are "floating" around.

Here is a front view of the top polygons linking the windows to the fuselage. Notice the care took on the curvature.

As you can see on the following picture, the nasty shading issue has disappeared around the windows (green arrows) However, you get two obvious shading artifacts along the top and bottom sides of the windows edges.

I really hope you now understand what was the causes of the shading issues :
1) Too much vertices in a tiny area. => Use the least amount of vertices.
2) Wrong orientation of diagonals/edges. => Keep the diagonals parallels; use in conjunction with perfectly horizontal edges.
3) Too pronounced curvature for a group of polygons in a small area. => Try to flatten smoothed polygons faces as much as possible while keeping a slight curvature.


How to get rid of that horizontal shading artifact ?
1) Use the least amount of vertices. As you can't delete those of the windows, your have no option here.
2) Keep the diagonals parallels in conjunction with horizontal edges. Diagonals and edges are OK here.
3) Try to flatten smoothed polygons faces as much as possible. Oh ! We can do that here ! But in order to keep a nice curve, we'll have to add at least one more face to the fuselage cylinder. The curvature will then look less sharp. Let's try this on the top side of the windows...

First, select the two vertices shown in the picture below, and move them a little away from the fuselage.

They should land here (front view - keep in mind the curvature) Moving the vertices there will require the creation of an extra face on the fuselage, but that will decrease the curvature between the top side of the windows and the fuselage, and that's what we want.

Now, just like you've done with the two windows, clone one vertice from one end of the fuselage, and place it just halfway between the two vertices you just moved.

Create a polygon (triangle) to get back the feeling the windows are linked to the remaining part of the fuselage.

And here you are. Top side windows shading issue gone !

That's one way to do it. Let me show you another one, while not really unrelated, is a little different.

Bottom side : select all vertices of the lower part of the windows/fuselage panel. The selection should include all vertices - but only those - that are the cause of the shading issue.

Then use the non-linear scaling to significantly reduce the curvature in the lateral axis (Red arrow constrain in picture above) This is possible and easy to do because this lower part of the windows is actually at a location where the fuselage is nearly vertical. I didn't make a screenshot to show the result because there is no difference you would notice. However, the shading is improved, while not enough to get rid of the horizontal artifact.
=> So, after combining this scaling method with the extra added face above (notice the added central vertice and the pair moved upwards in picture below) this is what we get :
Bottom shading issue gone !

 

[fsintegral]

Okay. We've got two windows and a fuselage panel between them, but what about other windows ?
Haha. Didn't though of that..
Just kidding.

1) Delete one window (the window, not the panel)

2) Delete the extra polygons (and vertices) on your fuselage cylinder to have a clear view of the windows + panel (again)

3) Clone your single window + panel, then align the cloned part perfectly with the former.

4) Repeat the process as many times as required.

After getting all your windows, select the vertices. All the vertices.

Then apply a welding. Anything as advanced as Gmax has this feature. I use a threshold of 0.005m.

And this is what you get :

In order to get the windows/panels merged with the fuselage cylinder, you'll have to create polygons. Quick screens...

Then, select each window polygon, and detach them from the main fuselage.

Now, I hope you'll make good use of this technique. I've chosen to show you the hardest way to have a clear and detailed overview of the whole process. Reading this won't help you much, it's by doing it you can master it. And only when you clearly understand the cavecats, you'll be able to find better ways to do it without the hassle of moving each vertex by hand. Actually, there are at least 4 easier methods, but that's something you'll have to discover by yourself. I just can give you hints that have one thing in common : Follow procedures and don't neglect steps.

Hint 1 : Because you're doing something rather detailed, make sure you're using perfect references right from the begining. (3 views, decals, cross sections...) You're going to spend a lot of time, and through the process, there will be many "no going back" available. Always compare the result with actual photos of the real thing.

Hint 2 : Safe booleans involve simple shapes. Instead of creating a flat window and drawing vertex per vertex, polygon per polygon, you could use boolean, one window at a time, then fix each vertice as required.

Hint 3 : Before thinking of windows, make sure your fuselage sides/edges perfectly align with top and bottom side of your windows (doors, etc.) This, to avoid doing fixes on the windows, but rather on the fuselage itself, only, which is much simplier in shape/polygons. The same advice applies to any cut you could do anywhere on your model : Cargo doors, flaps, spoilers, ailerons, etc. Failing to place vertices right where cuts will occur will lengthen your work due to the required refinements here and there.

Hint 4 : Some tools have scripting capabilities. Instead of creating vertices and polygons by hand, you can use scripts to generate them. But you'll have to code the scripts, most of the time... By the way, Gmax seems to have some bugs in its own scripting engine (or it doesn't support all features of the tool) When using scripts, you should know what the tool is capable of.

Hint 5 : In Mesh mode (at least in Gmax) you can play with diagonals, rotate them. Why I like Poly mode is because I quickly lose track of what I have to do... There's no room for errors in Poly mode, but most users feel better in mesh mode. Do most of the editing in the mode you're used to, but from time to times, clone your objects and have a look in other modes. Bet you will discover new techniques that way...

Have a nice flight.

 

[hairyspin]

The above is proof, were any needed, that there's at least six possible ways to do a job in Gmax. Wow.

 

[Paul Domingue]

Wow is right. Thanks for taking the time to post your method.

 

[hairyspin]

fsintegral, to cut down on the extra vertex count make sure both the fuselage and window cutters are editable poly. Your Boolean result gets progressively more messy as you cut with the second and then third cutter object, so simplify things for Gmax by combining the three window cutters into one object with Attach: one cutter, one operation, less mess. Also cap the cutter objects - yours are open ended and Gmax doesn't always work well with these in Boolean ops. 3ds Max has the ProBoolean tool which can use a list of cutters in a single operation.

Get rid of one source of long thin triangles (one cause of the shading problem) by cutting a loop either side of where the window is to go. A second loop closer to the window may also help. Once you have done the Boolean and converted to e-poly, unwanted edges can be simply deleted: in 3ds Max you use the Remove button. To get rid of unwanted vertices, Target Weld them, or install MeshTools 2.5 and use the Erase Vertex Clean tool (MeshTools are indispensable!): in 3ds Max use the Remove button again. Booleans attract most flak over the extra vertices generated but we don't have to keep them, they should always be tidied up. Then create edges where and how you want them.

I understand you're introducing extra curvature to the windows to reduce or eliminate the shading problem, but might it not be an idea to just make the mesh more detailed? A 24-segment cylinder is going to be smoother than a 16-segment one and today's PCs - hang it all, yesterday's PCs too - can handle poly counts larger than a few thousand. Just my tuppence ha'penny.

 

[fsintegral]

fsintegral, to cut down on the extra vertex count make sure both the fuselage and window cutters are editable poly. Your Boolean result gets progressively more messy as you cut with the second and then third cutter object, so simplify things for Gmax by combining the three window cutters into one object with Attach: one cutter, one operation, less mess. Also cap the cutter objects - yours are open ended and Gmax doesn't always work well with these in Boolean ops. 3ds Max has the ProBoolean tool which can use a list of cutters in a single operation.

^^ Great ! Although they were both editabe poly (and the three windows were one single object) I think it's the fact I didn't filled both ends that produces the complex vertex issue. This is a good demonstration of how important is doing it the correct way from the start. But to avoid errors, you have to know what causes them I just thought GMax has limited capabilities (assuming it's a scaled down version of Max) but I learned something new today. Thank you. Will make "solid" objects cutters from now on...

Get rid of one source of long thin triangles (one cause of the shading problem) by cutting a loop either side of where the window is to go. A second loop closer to the window may also help. Once you have done the Boolean and converted to e-poly, unwanted edges can be simply deleted: in 3ds Max you use the Remove button. To get rid of unwanted vertices, Target Weld them, or install MeshTools 2.5 and use the Erase Vertex Clean tool (MeshTools are indispensable!): in 3ds Max use the Remove button again. Booleans attract most flak over the extra vertices generated but we don't have to keep them, they should always be tidied up. Then create edges where and how you want them.

Very good tips. Thanks.
Really nice sample.

I understand you're introducing extra curvature to the windows to reduce or eliminate the shading problem, but might it not be an idea to just make the mesh more detailed? A 24-segment cylinder is going to be smoother than a 16-segment one and today's PCs - hang it all, yesterday's PCs too - can handle poly counts larger than a few thousand. Just my tuppence ha'penny.

Yeah ! You're right. I'm just used to low poly usage wherever possible, it can be a bad habit sometimes. However, I decided to go with a low sided cylinder on purpose :
- If I started with a detailed mesh, spotting the artifacts and explaining why they're occuring would have been more difficult (my main goal was to detail one technique and show its pros and cons. Up to anyone to decide when its use suits the task) But yes, I just did it with a too simplistic fuselage
- One thing I also wanted to showcase is : you don't have to maintain the same amount of polygon usage everywhere on the model. Only the parts subject to "eye attraction" would require more details (angular edges on windows are pretty obvious, much less on a blank panel of the fuselage - but that's my own opinion )

One thing I can tell is : if I were to redo that windows cutting and shading fixes, I wouldn't be able to do it the same way I did it above and get a similar wireframe, because of a lot of reasons :
- taste at the moment
- new learned techniques in the meantime (pretty sure I'll use boolean)
- amount of time available
- and mostly, I would redo it with much more details and polygons anyway