Moderators: richierich, ua900, PanAm_DC10, hOMSaR

wardialer
Topic Author
Posts: 1243
Joined: Fri Sep 14, 2001 1:08 pm

### LAT/LONG - Most accurate unit measuring system in aviation?

What is the most accurate LAT/LONG units below for aviation?
DD MM.M'
or
DD MM'SS"

GalaxyFlyer
Posts: 8896
Joined: Fri Jan 01, 2016 4:44 am

### Re: LAT/LONG - Most accurate unit measuring system in aviation?

We use DD MM.MM, not seconds but tenths or hundredths of minutes.

wardialer
Topic Author
Posts: 1243
Joined: Fri Sep 14, 2001 1:08 pm

### Re: LAT/LONG - Most accurate unit measuring system in aviation?

Please post as an example on the format you stated.

Are you also referring to the FMS?
Does the FMS use the same format you mentioned without the SS?

glen
Posts: 373
Joined: Thu Jun 30, 2005 4:43 pm

### Re: LAT/LONG - Most accurate unit measuring system in aviation?

That's an easy calculation:
DD MM.M' gives you an accuracy of 1/10 of a Minute, i.e. 1/10 of a nautical mile
DD MM'SS' gives you an accuracy of 1/60 of a Minute, i.e. 1/60 of a nautical mile, so this is more exact.

We usually use DD MM.M'

GalaxyFlyer
Posts: 8896
Joined: Fri Jan 01, 2016 4:44 am

### Re: LAT/LONG - Most accurate unit measuring system in aviation?

wardialer wrote:
Please post as an example on the format you stated.

Are you also referring to the FMS?
Does the FMS use the same format you mentioned without the SS?

Yes, the FMS will use DD MM.M for entries, but some charts will show DD MM.MM. As stated above using tenths of a minute is accurate to 608’, plenty accurate for enroute waypoints. Hundredths of a minute are 60’mand thousandths are 6’. Waypoints are contained in the database, so rarely are they manually entered, never on the approach phases of flight.

Here’s the record for the BDL VORTAC, 41-56-27.638N / 072-41-18.881W. The databases use three digits to the right of the decimal, but as crew we wouldn’t enter the waypoint that manual, all out of the database.

Here’s a lookup site

https://www.airnav.com/airspace/fix/

CanukinUSA
Posts: 169
Joined: Sun Oct 25, 2020 5:06 pm

### Re: LAT/LONG - Most accurate unit measuring system in aviation?

The most accurate position data for a given point in Aviation is the Surveyors coordinates that have been obtained during surveying done for the Civil Aviation Authority in a particular country. Using Differential GPS surveying these positions are typically within cm of the correct position and obtained to the nearest 1/10000 of a second (4 decimal points in seconds) in Latitude and Longitude. These coordinates are rounded off to put in databases by the civil aviation authorities who control the positional data and database providers such as Jeppesen for use in avionics such as Flight Management Computers (FMS) etc.. These coordinates are much more accurate then is typically required by the flight crew and most avionic systems such as FMS, GLS, etc.. Coordinates that may be entered by the Flight Crew in their FMS are only allowed to the accuracy that the Avionics are certified for as stated in comments above. As more accuracy is required such as for GPS Landing Systems (GLS) approaches the crew is not allowed to manually enter location positions to avoid the possibly of manually making typing position errors into the system.
Another important consideration in the accuracy is the Horizontal and Vertical datum systems used to specify the position. The GPS system uses the WGS-84 datum which is a worldwide coordinate system, most countries have now published and/or have surveyed their aviation positions in the WGS-84 horizontal datum as recommended by the International Civil Aviation Organization (ICAO). There are still a few countries that have not surveyed their positions in WGS-84 and this can result in positional accuracies of as much as a few 100 meters horizontally in position from the correct position. Vertical position datum errors are now becoming an issue for things such as GLS approaches where there can be differences of a considerable amount in height from the normal sea level altitudes to WGS-84 vertical datum altitudes. When doing precision approaches an error of a few feet vertically can become critical.
Both ICAO and the FAA have published detailed manuals and advisory circulars which cover the required accuracy and techniques for the surveyors to use in determining positions for aviation use.

LH707330
Posts: 2558
Joined: Fri Jun 15, 2012 11:27 pm

### Re: LAT/LONG - Most accurate unit measuring system in aviation?

wardialer wrote:
What is the most accurate LAT/LONG units below for aviation?
DD MM.M'
or
DD MM'SS"

That's the precision you're after. Accuracy is how close the stated value is to the actual value. If I told you San Fransisco is "about 38N 122W," that's more accurate than saying 35 36' 42" N 123 53' 12" W, but less precise. I tend not to trust anything with too many decimal points, because it tells me the system designer was more concerned with precision than accuracy.

Yikes!
Posts: 366
Joined: Sat Oct 13, 2001 4:51 pm

### Re: LAT/LONG - Most accurate unit measuring system in aviation?

Most commercial FMS inputs use Degrees, Minutes and decimal minutes (to one place). Our surveillance FMS used degrees, minutes and decimal minutes (to two places) i.e.N75 27.55 vice a commercial notation of N75 27.6

Hope that helps.

### Who is online

Users browsing this forum: Flow2706 and 21 guests

### Popular Searches On Airliners.net

Top Photos of Last:   24 Hours  •  48 Hours  •  7 Days  •  30 Days  •  180 Days  •  365 Days  •  All Time

Military Aircraft Every type from fighters to helicopters from air forces around the globe

Classic Airliners Props and jets from the good old days

Flight Decks Views from inside the cockpit

Aircraft Cabins Passenger cabin shots showing seat arrangements as well as cargo aircraft interior

Cargo Aircraft Pictures of great freighter aircraft

Government Aircraft Aircraft flying government officials

Helicopters Our large helicopter section. Both military and civil versions

Blimps / Airships Everything from the Goodyear blimp to the Zeppelin

Night Photos Beautiful shots taken while the sun is below the horizon

Accidents Accident, incident and crash related photos

Air to Air Photos taken by airborne photographers of airborne aircraft

Special Paint Schemes Aircraft painted in beautiful and original liveries

Airport Overviews Airport overviews from the air or ground

Tails and Winglets Tail and Winglet closeups with beautiful airline logos