comp.lang.idl-pvwave archive: archive » Strange floating-point precision behavior

Home » Public Forums » archive » Strange floating-point precision behavior

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Strange floating-point precision behavior [message #33926]

Sat, 08 February 2003 15:33

lloyd
Messages: 16
Registered: February 2003

Junior Member

I have written a routine that converts Earth-Centered Inertial
coordinates in x/y/z to geodetic latitude/longitude/altitude using the
WGS84 standard. I have one issue, however, that I believe is
affecting my calculations of altitude so that they are accurate only
to 1-meter resolution. I am defining the ECI coordinates as
double-precision:

IDL> boulder={x:-1283388.8693d0, $
y:-4713016.9053d0, $
z:4090191.0471d0} ;Boulder, CO, GPS station

and yet IDL seems to be storing the data incorrectly:

IDL> print,boulder,format='(3f20.10)'
-1283388.8692999999 -4713016.9052999998 4090191.0471000001

What am I doing wrong? I am fairly certain that this behavior is
responsible for my calculations yielding 1674.6658 m as the altitude
of the Boulder GPS station, and not 1674.7428 m (the actual altitude).
This is on IDL 5.6 for Mac OS X.

Thanks,
Tim Lloyd
Laboratory for Atmospheric & Space Physics

Report message to a moderator

[Message index]

		Strange floating-point precision behavior By: lloyd on Sat, 08 February 2003 15:33
		Re: Strange floating-point precision behavior By: tim on Tue, 11 February 2003 09:49
		Re: Strange floating-point precision behavior By: Craig Markwardt on Mon, 10 February 2003 19:57
		Re: Strange floating-point precision behavior By: James Kuyper on Mon, 10 February 2003 12:44

Previous Topic:	drawing a single point in 3-D
Next Topic:	License options when starting IDL

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Wed Oct 08 19:43:02 PDT 2025

Total time taken to generate the page: 0.00469 seconds