| Re: strange behaviour of bytscl by large arrays [message #79961 is a reply to message #79960] |
Mon, 23 April 2012 07:59   |
Klemen
Messages: 80
Registered: July 2009
|
Member |
|
|
Hi David,
I read "the sky is falling" some time ago. I guess I understand how the numbers are saved in the computer. And perhaps this is not the best case I have shown over here. But still, as far as I understand float type, a variable has first 7 figures that should be ok. But I don't really care about the precision on the 7th figure - i have problems on the 1st figure. Look at this example.
IDL> a=findgen(10000,10000)
IDL> print, min(a), max(a)
0.000000 7.50000e+007 ;a huge error
IDL> h=histogram(a, BINSIZE=10.^5)
IDL> print, h
Not just that the maximum value is way too small. The histogram seems strange too. I don't show here the wohle histogram output. As expected the h contains in the begining 100 000 values per bin. But this is just for the first 167 bins. in 168 there are 8 300 000 values, then following about 80 bins contain no values at all, then comes one with 25 000 000, then zeros again... 25 000 000.
So this is not the precision. Is there a point that I am missing, perhaps by the mantisa of a number? If I use dindgen at the begining, everything seems ok. Or is the sky falling anyway? :)
Cheers, Klemen
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
