| Re: strange behaviour of bytscl by large arrays [message #79960 is a reply to message #79959] |
Mon, 23 April 2012 08:10   |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
Klemen writes:
> I read "the sky is falling" some time ago. I guess I understand how the numbers are saved in the computer. And perhaps this is not the best case I have shown over here. But still, as far as I understand float type, a variable has first 7 figures that should be ok. But I don't really care about the precision on the 7th figure - i have problems on the 1st figure. Look at this example.
>
> IDL> a=findgen(10000,10000)
> IDL> print, min(a), max(a)
> 0.000000 7.50000e+007 ;a huge error
> IDL> h=histogram(a, BINSIZE=10.^5)
> IDL> print, h
>
> Not just that the maximum value is way too small. The histogram seems strange too. I don't show here the wohle histogram output. As expected the h contains in the begining 100 000 values per bin. But this is just for the first 167 bins. in 168 there are 8 300 000 values, then following about 80 bins contain no values at all, then comes one with 25 000 000, then zeros again... 25 000 000.
>
> So this is not the precision. Is there a point that I am missing, perhaps by the mantisa of a number? If I use dindgen at the begining, everything seems ok. Or is the sky falling anyway? :)
I just think you are dealing with numbers that don't
have enough significant digits to do what you want to
do. You are going to have to multiply (or divide, as
the case may be) by a number that restores enough
significant values for you to perform your operation.
Or, alternatively, use double precision arrays on
the numbers you do have.
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.idlcoyote.com/
Sepore ma de ni thui. ("Perhaps thou speakest truth.")
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
