comp.lang.idl-pvwave archive: archive

Home » Public Forums » archive » Re: decimal numbers

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: decimal numbers [message #70914 is a reply to message #70913]

Tue, 11 May 2010 16:30

penteado
Messages: 866
Registered: February 2018

Senior Member
Administrator

On May 11, 8:15 pm, fgg <fabioguimaraesgoncal...@gmail.com> wrote:
> Hello,
>
> If
>
> IDL> a=358983
>
> how can I get a "b" = 0.358983? (that is, b = a/10^6)?... Why do I get
> the weird result below when I change "a" to double and divide it by
> 10^6?
>
> IDL> b=double(a)/10^6
> IDL> print, b
> 21.166450

Just use

a=358983d-6

10^6 overflows integers (16 bit by default):

IDL> print,10^6
16960

If you want to use 10^6, use either 1d6, 1e6, or if you want integers,
some larger integer type, such as 10L^6.

See

http://groups.google.com/group/comp.lang.idl-pvwave/browse_t hread/thread/775f355afcfa0b8a

Report message to a moderator

[Message index]

		Re: decimal numbers By: fgg on Tue, 11 May 2010 16:45
		Re: decimal numbers By: penteado on Tue, 11 May 2010 16:30
		Re: decimal numbers By: R.G.Stockwell on Tue, 11 May 2010 16:25

Previous Topic:	decimal numbers
Next Topic:	Re: SST data

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sat Oct 11 11:15:16 PDT 2025

Total time taken to generate the page: 0.24183 seconds