Page 1 of 1

### LAT/LONG - Most accurate unit measuring system in aviation?

Posted: Sun Nov 14, 2021 7:38 pm
What is the most accurate LAT/LONG units below for aviation?
DD MM.M'
or
DD MM'SS"

### Re: LAT/LONG - Most accurate unit measuring system in aviation?

Posted: Sun Nov 14, 2021 8:38 pm
We use DD MM.MM, not seconds but tenths or hundredths of minutes.

### Re: LAT/LONG - Most accurate unit measuring system in aviation?

Posted: Sun Nov 14, 2021 8:46 pm
Please post as an example on the format you stated.

Are you also referring to the FMS?
Does the FMS use the same format you mentioned without the SS?

### Re: LAT/LONG - Most accurate unit measuring system in aviation?

Posted: Sun Nov 14, 2021 9:00 pm
That's an easy calculation:
DD MM.M' gives you an accuracy of 1/10 of a Minute, i.e. 1/10 of a nautical mile
DD MM'SS' gives you an accuracy of 1/60 of a Minute, i.e. 1/60 of a nautical mile, so this is more exact.

We usually use DD MM.M'

### Re: LAT/LONG - Most accurate unit measuring system in aviation?

Posted: Sun Nov 14, 2021 9:20 pm
wardialer wrote:
Please post as an example on the format you stated.

Are you also referring to the FMS?
Does the FMS use the same format you mentioned without the SS?

Yes, the FMS will use DD MM.M for entries, but some charts will show DD MM.MM. As stated above using tenths of a minute is accurate to 608’, plenty accurate for enroute waypoints. Hundredths of a minute are 60’mand thousandths are 6’. Waypoints are contained in the database, so rarely are they manually entered, never on the approach phases of flight.

Here’s the record for the BDL VORTAC, 41-56-27.638N / 072-41-18.881W. The databases use three digits to the right of the decimal, but as crew we wouldn’t enter the waypoint that manual, all out of the database.

Here’s a lookup site

https://www.airnav.com/airspace/fix/

### Re: LAT/LONG - Most accurate unit measuring system in aviation?

Posted: Mon Nov 15, 2021 5:46 pm
The most accurate position data for a given point in Aviation is the Surveyors coordinates that have been obtained during surveying done for the Civil Aviation Authority in a particular country. Using Differential GPS surveying these positions are typically within cm of the correct position and obtained to the nearest 1/10000 of a second (4 decimal points in seconds) in Latitude and Longitude. These coordinates are rounded off to put in databases by the civil aviation authorities who control the positional data and database providers such as Jeppesen for use in avionics such as Flight Management Computers (FMS) etc.. These coordinates are much more accurate then is typically required by the flight crew and most avionic systems such as FMS, GLS, etc.. Coordinates that may be entered by the Flight Crew in their FMS are only allowed to the accuracy that the Avionics are certified for as stated in comments above. As more accuracy is required such as for GPS Landing Systems (GLS) approaches the crew is not allowed to manually enter location positions to avoid the possibly of manually making typing position errors into the system.
Another important consideration in the accuracy is the Horizontal and Vertical datum systems used to specify the position. The GPS system uses the WGS-84 datum which is a worldwide coordinate system, most countries have now published and/or have surveyed their aviation positions in the WGS-84 horizontal datum as recommended by the International Civil Aviation Organization (ICAO). There are still a few countries that have not surveyed their positions in WGS-84 and this can result in positional accuracies of as much as a few 100 meters horizontally in position from the correct position. Vertical position datum errors are now becoming an issue for things such as GLS approaches where there can be differences of a considerable amount in height from the normal sea level altitudes to WGS-84 vertical datum altitudes. When doing precision approaches an error of a few feet vertically can become critical.
Both ICAO and the FAA have published detailed manuals and advisory circulars which cover the required accuracy and techniques for the surveyors to use in determining positions for aviation use.

### Re: LAT/LONG - Most accurate unit measuring system in aviation?

Posted: Fri Nov 19, 2021 12:59 am
wardialer wrote:
What is the most accurate LAT/LONG units below for aviation?
DD MM.M'
or
DD MM'SS"

That's the precision you're after. Accuracy is how close the stated value is to the actual value. If I told you San Fransisco is "about 38N 122W," that's more accurate than saying 35 36' 42" N 123 53' 12" W, but less precise. I tend not to trust anything with too many decimal points, because it tells me the system designer was more concerned with precision than accuracy.

### Re: LAT/LONG - Most accurate unit measuring system in aviation?

Posted: Thu Dec 02, 2021 3:27 am
Most commercial FMS inputs use Degrees, Minutes and decimal minutes (to one place). Our surveillance FMS used degrees, minutes and decimal minutes (to two places) i.e.N75 27.55 vice a commercial notation of N75 27.6

Hope that helps.