comp.lang.idl-pvwave archive: archive » Re: Strange floating-point precision behavior

Home » Public Forums » archive » Re: Strange floating-point precision behavior

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: Strange floating-point precision behavior [message #33924]

Sat, 08 February 2003 18:31

Kenneth P. Bowman
Messages: 585
Registered: May 2000

Senior Member

In article <e41fb407.0302081533.3874340f@posting.google.com>,
lloyd@lasp.colorado.edu (Tim Lloyd) wrote:

> I have written a routine that converts Earth-Centered Inertial
> coordinates in x/y/z to geodetic latitude/longitude/altitude using the
> WGS84 standard. I have one issue, however, that I believe is
> affecting my calculations of altitude so that they are accurate only
> to 1-meter resolution. I am defining the ECI coordinates as
> double-precision:
>
> IDL> boulder={x:-1283388.8693d0, $
> y:-4713016.9053d0, $
> z:4090191.0471d0} ;Boulder, CO, GPS station
>
> and yet IDL seems to be storing the data incorrectly:
>
> IDL> print,boulder,format='(3f20.10)'
> -1283388.8692999999 -4713016.9052999998 4090191.0471000001
>
> What am I doing wrong? I am fairly certain that this behavior is
> responsible for my calculations yielding 1674.6658 m as the altitude
> of the Boulder GPS station, and not 1674.7428 m (the actual altitude).
> This is on IDL 5.6 for Mac OS X.

Double precision gives you about 14 digits of decimal precision. You
are only providing about 10 digits, and you are getting out exactly what
you put in:

-1283388.8693 = -1283388.8692999999

within better than 14 digits.

Ken Bowman

Report message to a moderator

[Message index]

		Re: Strange floating-point precision behavior By: Kenneth P. Bowman on Sat, 08 February 2003 18:31
		Re: Strange floating-point precision behavior By: tim on Sat, 08 February 2003 18:28

Previous Topic:	Re: IDL executable compile options for FPE
Next Topic:	Suggestions on the use of lockfiles?

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Thu Oct 09 21:46:58 PDT 2025

Total time taken to generate the page: 0.47916 seconds