| Re: Inaccuracies [message #5314 is a reply to message #5313] |
Mon, 13 November 1995 00:00  |
bowman
Messages: 121
Registered: September 1991
|
Senior Member |
|
|
In article <30A7BC4D.7018@cdc.noaa.gov>, Andy Loughe <afl@cdc.noaa.gov> wrote:
> IDL> a = findgen(15)*.2 - 1.4
> IDL> print, total(a)
> 7.15256e-07
~7 significant figures is what you expect from single precision arithmetic.
> IDL> a = dindgen(15)*(.2D)-1.4D
> IDL> print, total(a, /double)
> 4.4408921e-15
~15 is what you expect from double precision.
You can do this:
IDL> i = LINDGEN(15)*2L - 14L
IDL> print, i
-14 -12 -10 -8 -6 -4
-2 0 2 4 6 8
10 12 14
IDL> c = DOUBLE(i)/DOUBLE(10)
IDL> print, c
-1.4000000 -1.2000000 -1.0000000 -0.80000000
-0.60000000 -0.40000000 -0.20000000 0.0000000
0.20000000 0.40000000 0.60000000 0.80000000
1.0000000 1.2000000 1.4000000
which gives you an exact 0 (division of 0 by anything should(!) be exactly
0), but the other terms are not necessarily exact, and you still get
IDL> print, total(c)
2.2204460e-16
which is the best you can hope for.
That's floating point arithemtic ...
Regards, Ken Bowman
--
Kenneth P. Bowman, Assoc. Prof. 409-862-4060
Department of Meteorology 409-862-4132 fax
Texas A&M University bowman@csrp.tamu.edu
College Station, TX 77843-3150
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
