comp.lang.idl-pvwave archive: archive » Re: Numbers from nowhere?

Home » Public Forums » archive » Re: Numbers from nowhere?

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: Numbers from nowhere? [message #58725]

Sun, 17 February 2008 13:53

David Fanning
Messages: 11724
Registered: August 2001

Senior Member

elwood writes:

> But my question is more pointed:
> if you assign x=3.3
> and you know apriori that the floating point data type
> will not have enough bits to store this number precisely,
> why does "print" show this number as 3.3?
>
> Is it because it is rounding off with some kind of algorithm?
> Or am I misinterpreting how this number is being stored?

I presume it is because whatever number *is* stored,
when rounded to the 7-8 significant figures a float
can accurately represent, comes out to 3.300000.

IDL> x=3.3
IDL> print, x, format='(f0)'
3.300000

IDL> print, x, format='(f12.10)'
3.2999999523

Cheers,

David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming (www.dfanning.com)
Sepore ma de ni thui. ("Perhaps thou speakest truth.")

Report message to a moderator

[Message index]

		Re: Numbers from nowhere? By: David Fanning on Sun, 17 February 2008 13:53
		Re: Numbers from nowhere? By: elwood on Sun, 17 February 2008 12:19
		Re: Numbers from nowhere? By: David Fanning on Fri, 15 February 2008 14:51
		Re: Numbers from nowhere? By: Paul Van Delst[1] on Fri, 15 February 2008 13:31
		Re: Numbers from nowhere? By: bennette on Fri, 15 February 2008 13:12
		Re: Numbers from nowhere? By: Sven Utcke on Thu, 21 February 2008 05:06
		Re: Numbers from nowhere? By: elwood on Mon, 25 February 2008 09:57

Previous Topic:	Labeling the x-axis with a non standard time string
Next Topic:	idl format codes and output files

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Wed Oct 08 19:25:10 PDT 2025

Total time taken to generate the page: 0.00428 seconds