|
|
| Re: decimal numbers [message #70914 is a reply to message #70913] |
Tue, 11 May 2010 16:30  |
penteado
Messages: 866
Registered: February 2018
|
Senior Member
Administrator |
|
|
On May 11, 8:15 pm, fgg <fabioguimaraesgoncal...@gmail.com> wrote:
> Hello,
>
> If
>
> IDL> a=358983
>
> how can I get a "b" = 0.358983? (that is, b = a/10^6)?... Why do I get
> the weird result below when I change "a" to double and divide it by
> 10^6?
>
> IDL> b=double(a)/10^6
> IDL> print, b
> 21.166450
Just use
a=358983d-6
10^6 overflows integers (16 bit by default):
IDL> print,10^6
16960
If you want to use 10^6, use either 1d6, 1e6, or if you want integers,
some larger integer type, such as 10L^6.
See
http://groups.google.com/group/comp.lang.idl-pvwave/browse_t hread/thread/775f355afcfa0b8a
|
|
|
|
| Re: decimal numbers [message #70915 is a reply to message #70914] |
Tue, 11 May 2010 16:25 |
R.G.Stockwell
Messages: 163
Registered: October 2004
|
Senior Member |
|
|
"fgg" <fabioguimaraesgoncalves@gmail.com> wrote in message
news:a43a5251-e8a0-45d2-a283-29388766de37@a2g2000prd.googleg roups.com...
> Hello,
>
> If
>
> IDL> a=358983
>
> how can I get a "b" = 0.358983? (that is, b = a/10^6)?... Why do I get
> the weird result below when I change "a" to double and divide it by
> 10^6?
>
> IDL> b=double(a)/10^6
> IDL> print, b
> 21.166450
>
> Thank you!
try this.
IDL> print,10^6
16960
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
