comp.lang.idl-pvwave archive: archive » Re: print and precision

Home » Public Forums » archive » Re: print and precision

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: print and precision [message #58747]

Thu, 14 February 2008 23:44

lasse
Messages: 48
Registered: February 2007

Member

Hi there

> As near as I can tell the print command pretty much just picks a
> random number of digits to print out.

Far from it! With the 32bits available for type float numbers, you can
only archive a relative precision of 1e-6 to 1e-7 so it makes
absolutely no sense to print more then 6-7 digits of a float type
variable.

> Of course floating point doesn't store 9 digits of precision, so I
> have no idea where the extra digits are coming from anyway...

Precisely. So why use a format code to print them out? If you do a

x = 3.3

then IDL saves precisely that number to a relative precision of the
float type.

Cheers
Lasse Clausen

Report message to a moderator

[Message index]

		Re: print and precision By: lasse on Thu, 14 February 2008 23:44
		Re: print and precision By: cmancone on Thu, 14 February 2008 11:47

Previous Topic:	keyboard shortcut in IDL 7
Next Topic:	Surface plotting some details

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Fri Oct 10 06:45:26 PDT 2025

Total time taken to generate the page: 0.64053 seconds