comp.lang.idl-pvwave archive: archive » Re: DOUBLE precision no precise??

Home » Public Forums » archive » Re: DOUBLE precision no precise??

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: DOUBLE precision no precise?? [message #29621 is a reply to message #29619]

Tue, 05 March 2002 04:14

James Tappin
Messages: 54
Registered: December 1995

Member

David Williams wrote:

>
> I've always had heaps of help from the inhabitants of this newsgroup --
> for which I am eternally grateful -- despite my often stupid questions.
> So, when a mate of mine came across this `quirk' yesterday, and I wasn't
> sure how to help him out, I thought I'd ask this group.
>
> He has an array of numbers that he wants to apply a user-defined
> function to, but we're both a little disturbed by the fact that if you
> do the calculations with a pocket calculator, you get different numbers
> than if you perform the same calculation in IDL.
>
> To try and find where the problem is, we tried the following lines...
>
> IDL> a = DOUBLE(42766.080001)
> IDL> print,a,FORMAT='(F24.17)'
>
> 42766.07812500000000000
>
> As you see, the number we get out isn't the same as the number we
> entered. I'm guessing it's to do with the way IDL stores numbers in
> memory, but my understanding of low-level computational processes isn't
> great.
>
> Can anybody help me understand what's going on, and/or if there's a way
> around? I'd really appreciate whatever help is on offer, so thanks in
> advance.

The problem is that 42766.080001 is a single precision constant, so what's
happening is that you are storing the single-precision approximation to
42766.080001 in some scratch location, then converting that to double.

What you actually want is:
a=42766.080001D0

--
+------------------------+-------------------------------+-- -------+
| James Tappin | School of Physics & Astronomy | O__ |
| sjt@star.sr.bham.ac.uk | University of Birmingham | -- \/` |
| Ph: 0121-414-6462. Fax: 0121-414-3722 | |
+--------------------------------------------------------+-- -------+

Report message to a moderator

[Message index]

		Re: DOUBLE precision no precise?? By: Pete[1] on Tue, 05 March 2002 13:11
		Re: DOUBLE precision no precise?? By: dave_r_williams on Tue, 05 March 2002 09:20
		Re: DOUBLE precision no precise?? By: James Kuyper on Tue, 05 March 2002 08:29
		Re: DOUBLE precision no precise?? By: William Clodius on Tue, 05 March 2002 08:11
		Re: DOUBLE precision no precise?? By: Robert Stockwell on Tue, 05 March 2002 07:32
		Re: DOUBLE precision no precise?? By: Martin Downing on Tue, 05 March 2002 06:30
		Re: DOUBLE precision no precise?? By: Vincent Schut on Tue, 05 March 2002 07:07
		Re: DOUBLE precision no precise?? By: R.Bauer on Tue, 05 March 2002 06:07
		Re: DOUBLE precision no precise?? By: Vincent Schut on Tue, 05 March 2002 05:01
		Re: DOUBLE precision no precise?? By: James Tappin on Tue, 05 March 2002 04:14

Previous Topic:	bad magic number for shared library when using CALL_EXTERNAL
Next Topic:	CALL_EXTERNAL using C/Fortran shared library on HPUX11

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Wed Oct 08 18:31:48 PDT 2025

Total time taken to generate the page: 0.00190 seconds