| Re: Strange floating-point precision behavior [message #33924] |
Sat, 08 February 2003 18:31 |
Kenneth P. Bowman
Messages: 585
Registered: May 2000
|
Senior Member |
|
|
In article <e41fb407.0302081533.3874340f@posting.google.com>,
lloyd@lasp.colorado.edu (Tim Lloyd) wrote:
> I have written a routine that converts Earth-Centered Inertial
> coordinates in x/y/z to geodetic latitude/longitude/altitude using the
> WGS84 standard. I have one issue, however, that I believe is
> affecting my calculations of altitude so that they are accurate only
> to 1-meter resolution. I am defining the ECI coordinates as
> double-precision:
>
> IDL> boulder={x:-1283388.8693d0, $
> y:-4713016.9053d0, $
> z:4090191.0471d0} ;Boulder, CO, GPS station
>
> and yet IDL seems to be storing the data incorrectly:
>
> IDL> print,boulder,format='(3f20.10)'
> -1283388.8692999999 -4713016.9052999998 4090191.0471000001
>
> What am I doing wrong? I am fairly certain that this behavior is
> responsible for my calculations yielding 1674.6658 m as the altitude
> of the Boulder GPS station, and not 1674.7428 m (the actual altitude).
> This is on IDL 5.6 for Mac OS X.
Double precision gives you about 14 digits of decimal precision. You
are only providing about 10 digits, and you are getting out exactly what
you put in:
-1283388.8693 = -1283388.8692999999
within better than 14 digits.
Ken Bowman
|
|
|
|
| Re: Strange floating-point precision behavior [message #33925 is a reply to message #33924] |
Sat, 08 February 2003 18:28  |
tim
Messages: 9
Registered: July 1994
|
Junior Member |
|
|
In my haste I had neglected to do a proper search of this newsgroup's
archives. I see now that double-precision values are inherently off by a
small amount. Given that, I am looking for suggestions on how better to
handle double-precision values so that I get exact results from my
calculations.
Thanks.
On 8 Feb 2003, Tim Lloyd wrote:
> I have written a routine that converts Earth-Centered Inertial
> coordinates in x/y/z to geodetic latitude/longitude/altitude using the
> WGS84 standard. I have one issue, however, that I believe is
> affecting my calculations of altitude so that they are accurate only
> to 1-meter resolution. I am defining the ECI coordinates as
> double-precision:
>
> IDL> boulder={x:-1283388.8693d0, $
> y:-4713016.9053d0, $
> z:4090191.0471d0} ;Boulder, CO, GPS station
>
> and yet IDL seems to be storing the data incorrectly:
>
> IDL> print,boulder,format='(3f20.10)'
> -1283388.8692999999 -4713016.9052999998 4090191.0471000001
>
> What am I doing wrong? I am fairly certain that this behavior is
> responsible for my calculations yielding 1674.6658 m as the altitude
> of the Boulder GPS station, and not 1674.7428 m (the actual altitude).
> This is on IDL 5.6 for Mac OS X.
>
> Thanks,
> Tim Lloyd
> Laboratory for Atmospheric & Space Physics
>
Tim Lloyd, lloyd@lasp.colorado.edu
SNOE Mission Operations Lead Flight Controller
Laboratory for Atmospheric and Space Physics
"The eyes of the world now look into space, to the moon and to the
planets beyond, and we have vowed that we shall not see it governed
by a hostile flag of conquest, but by a banner of freedom and peace."
-- John F. Kennedy
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
