What formula to calculate lat/lon in FSX?

[Testili]

Hi,

I have a problem with the following:

I made a proggy, which will use GC and partially Rhumb Line formulas to calculate the lat/lon of a point D, which is different nms out (with a given heading) of a starting point with a given lat/lon/heading.
Now all this working quite well. It works great for small distances.
So far, the farer the distance, the more "off course" the calculated point is.

Let's say the starting point is the starting point of the centre of a runway. Now I want a perfectly 180° turned around line to a point D, which is let's say 5nm out of the starting point.
No matter if I use GC or rhumb line, my point D is always a little bit off course (meaning a little bit right of a thought perfect line).
Interestingly, if I do the calculations in Europe, the mistake is not as big as in US.
Magnetic variation should not matter I think, since FS should calculate everything with true north-based headings, so i think it is not this (I also tried so far, did not help me much).

I would be very thankful about:
1) An idea, what might cause the off-course
2) A hint, what to do against (depends on 1 most probably)
3) Perhaps I should neither use GC nor Rhumb lines, but a - for me still secret - other formula?

Any comments are really very welcome!

Thanx alot,
Chris.

[scruffyduck]

I'm not entirely sure I understand but if you have a starting point in lat/lon, a bearing and distance then the formula for doing this based on spherical geometry is available on a number of web sites. Look for terminal co-ordinates or similar.

But maybe you have already done that and I have mis-understood the question (which I find very easy to do :) )

[Testili]

Thanx for the reply and the try to understand me :) I admit, my english is another problem to fix^^
Actually I use the formula. I tried almost all available. Still im off the course (slightly right or slightly left). Its very exact up to 1.2 nm at about. Afterwards it looks as if im not on a perfect line, but slightly off. The more the ditance, that more is the mistake. I wonder why?
If you cannot imagine, please tell me, I could make some screenshots to point out what I mean.

Sincerely,
Chris.

[scruffyduck]

Hi Chris

Well you might want to post the formula that you are using. I am using these over long distances and they are very accurate so I am not sure why you are having a problem

[Pete Dowson]

Its very exact up to 1.2 nm at about. Afterwards it looks as if im not on a perfect line, but slightly off. The more the ditance, that more is the mistake. I wonder why?

If, as you say, the error varies in different parts of the world then it is almost certainly down to using magnetic values somewhere instead of true.

Regards

Pete

[davidt]

Hi again :wave: ,

I have a bit of a follow-up question to this. In my ongoing endeavors to have another plane fly smoothly above me, I've thought about using a way point, based on my course, and then periodically adjusting the position of the AI plane.

My thinking is that if all goes well, the way the AI controlled plane is moving shouldn't be too far off if I'm giving new updates regularly enough.. right?

So I found some forumlas which I'm using to determine the Lat/Lon of my destination a fair distance away from me. I'm telling it to keep the same speed and altitude.

This seems to work as the AI plane doesn't wobble or try to drop right away as soon as the new position is given to it.

However I'd like to get a more accurate waypoint, using my pitch, bank and altitudes as well. I looked around on the Aviation Formulary page (http://williams.best.vwh.net/avform.htm - found the link elsewhere in the forums here), and figured out what to use to get my new lat/lon. None of the forumlas have anything to do with pitch/bank, though. Which makes sense since they're mostly focused on planning a course beforehand, rather than adjusting in flight.

So.. the forumlas I'm using are:

lat2 = asin(sin(lat1)*cos(d) + cos(lat1)*sin(d)*cos(heading))

lon2 = mod(lon - asin(sin(heading)*sin(d)/cos(lat)) + pi, 2pi) - pi


But I know if I'm in the air, and I'm turning, my course isn't going to be a straight line based on my heading. How do I account for that?

Thanks again!
DAvid

[DougMatthews]

Thanx for the reply and the try to understand me :) I admit, my english is another problem to fix^^
Actually I use the formula. I tried almost all available. Still im off the course (slightly right or slightly left). Its very exact up to 1.2 nm at about. Afterwards it looks as if im not on a perfect line, but slightly off. The more the ditance, that more is the mistake. I wonder why?
If you cannot imagine, please tell me, I could make some screenshots to point out what I mean.

Sincerely,
Chris.

Does the magnitude of the error increase as you get farther from the Equator?

-Doug

[FlapsOut]

Hi DavidT,

Check my other post, (you started it) called 'Getting User's Pitch, Bank, Heading, and Airspeed'.

There is more than one way to 'skin a bear'! :)

Some of my source may help. Dunno...

Flaps

[ziporama]

I've gone through a lot of pain recently trying to figure this out.

Because the Earth isn't a sphere, you can easily get into accuracy errors of several km depending what latitude you're at. I'd look up "Vincenty's direct and inverse method" (the first to compute lat/long given a point, distance and bearing, the second to compute distance/bearing given two points defined by lat/long), and I'd also punch in "WGS-84 calculations" in your search engine.

There're a few examples on code project in C# on the subject although I wrote my own implementation using Vincenty's 1975 paper (available online from NOAA). There's also a government site in Australia with implementations in Excel of these accurate techniques.

For FS use, you don't need to have millimeter accuracy, so you can reduce the accuracy and number of iterations for faster computations.

Depending on how far you're going, you can also switch models and use simpler spherical math or even straight flat trig. 20Km distance seems to be the magic number - use the simpler models over short distances.

There's also information on how to convert lat/lon to cartesian (x,y,z) using the same ellipsoidal math which is also useful in some cases so you can easily do math around an airport for example.

All the algorithms will give you true north, and not magnetic, so you would also need to compensate compass heading for this given your position.

Lastly, I need to review how FSX implements the spherical model of the earth to make sure the formulas use the same values it uses. I'm assuming WGS-84 but I didn't look at the SDK docs yet.

Cheers,

Etienne

[davidt]

Ok.. thanks Zip.. I'll have a look at those calculations, and see if I can find that excel thing.

[ziporama]

Checking the SDK terrain doc, FSX does use an ellipsoid but apparently not WGS-84.

From the terrain SDK:

In Flight Simulator, the earth is defined as an “oblate spheroid,” an ellipse rotated about its minor axis. In terms of shape, it's much like a sphere that is a little bit fat around the equator. In Flight Simulator, the earth has the following dimensions:
• Equatorial diameter=12756.27 km
• Equatorial circumference=40075.0 km
• Polar diameter=12734.62 km
• Polar circumference=40007.0 km

Major axis a radius = 6378.135 km
Minor axis b radius = 6367.31 km
I'm computing an inverse flattening (1/f) = 589.204
where f = (a-b)/a

For WGS-84:
a = 6378.137 km
b = 6356.7523142 km
(1/f) = 298.257223563

For WGS-72:
a = 6378.135 km (same as with FSX)
b = 6356.75
(1/f) = 298.26

I'm not sure why FS would use such a weird set of numbers that don't seem to match any reference ellipsoid models of the Earth I've been able to find (unless my math is wrong). The only thing I can think of is to reduce the computational errors using simpler formulas for ellipsoids. Hard to check not knowing what MS uses under the hood. Perhaps Adam or Phil at ACEs can explain this.

Etienne

[DougMatthews]

Checking the SDK terrain doc, FSX does use an ellipsoid but apparently not WGS-84.

From the terrain SDK:
Major axis a radius = 6378.135 km
Minor axis b radius = 6367.31 km
I'm computing an inverse flattening (1/f) = 589.204
where f = (a-b)/a

For WGS-84:
a = 6378.137 km
b = 6356.7523142 km
(1/f) = 298.257223563

For WGS-72:
a = 6378.135 km (same as with FSX)
b = 6356.75
(1/f) = 298.26

I'm not sure why FS would use such a weird set of numbers that don't seem to match any reference ellipsoid models of the Earth I've been able to find (unless my math is wrong). The only thing I can think of is to reduce the computational errors using simpler formulas for ellipsoids. Hard to check not knowing what MS uses under the hood. Perhaps Adam or Phil at ACEs can explain this.

Etienne

FSX is based on the WGS-84 spheroid. I have filed a bug against the SDK docs.

-Doug

[sjcomp]

Hello,

I have a similar problem. I would like to go back an forth between Cartesian and lat/lon coordinate systems. I use Haversine formula (http://www.ig.utexas.edu/outreach/googleearth/latlong.html). As ziporama said using it works fine within 20km radius of the origin for my Cartesian system.

FSX is based on the WGS-84 spheroid. I have filed a bug against the SDK docs.-Doug
As of 2009 and the sdk for the acceleration pack the documentation had not changed. Does it mean WGS-84 is not used?

I am going to give a try to Vincenty's direct and inverse methods (http://www.movable-type.co.uk/scripts/latlong-vincenty.html). But meanwhile maybe someone already has experience to share about this?

Thank you.

[rhumbaflappy]

WGS84 is used... the SDK docs are in error ( if I understand what Doug wrote ).

Dick

[sjcomp]

WGS84 is used... the SDK docs are in error ( if I understand what Doug wrote ).Doug wrote it in 2008. Though I can see how error would not be corrected in that time span :)

[rhumbaflappy]

The way of the betas is just because you find a flaw does not mean it gets corrected. Lots of corrections end up in the released products because the time and money to change them just aren't there.

The pace to a release version is incredibly frantic.

[sjcomp]

The pace to a release version is incredibly frantic.I can understand that. I wonder if we can actually test the FSX to see what is used. Doug maybe 100% correct, but how can we find out what they use? :)