| DOUBLE precision no precise?? [message #29622] |
Tue, 05 March 2002 03:54 |
David Williams
Messages: 13
Registered: February 2000
|
Junior Member |
|
|
I've always had heaps of help from the inhabitants of this newsgroup --
for which I am eternally grateful -- despite my often stupid questions.
So, when a mate of mine came across this `quirk' yesterday, and I wasn't
sure how to help him out, I thought I'd ask this group.
He has an array of numbers that he wants to apply a user-defined
function to, but we're both a little disturbed by the fact that if you
do the calculations with a pocket calculator, you get different numbers
than if you perform the same calculation in IDL.
To try and find where the problem is, we tried the following lines...
IDL> a = DOUBLE(42766.080001)
IDL> print,a,FORMAT='(F24.17)'
42766.07812500000000000
As you see, the number we get out isn't the same as the number we
entered. I'm guessing it's to do with the way IDL stores numbers in
memory, but my understanding of low-level computational processes isn't
great.
Can anybody help me understand what's going on, and/or if there's a way
around? I'd really appreciate whatever help is on offer, so thanks in
advance.
Dave
------------------------------------------------------------ ---------
David R. Williams, | BT7 1NN, Northern Ireland.
Astrophysics & Planetary Science, | d.williams@qub.ac.uk
Queen's University, Belfast, | http://star.pst.qub.ac.uk/~drw/
============================================================ =========
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
