comp.lang.idl-pvwave archive: archive

Home » Public Forums » archive » print and precision

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

print and precision [message #58760]

Thu, 14 February 2008 08:50

elwood
Messages: 23
Registered: February 2007

Junior Member

I have a tiny understanding of how numbers are stored in computers
and how a float only has 32 bits to store a number, so for some
numbers it may run
out of bits before it can store the precise value that the user
intended.

My question is, what is the default form of the print statement doing?

for example:
x=3.3

If i Understand correctly, the floating point binary representation of
this number is
11.01001100110011001100110011001100
which exceeds 32 bits

so I'd expect to get something like
3.299999999813735
due to truncation

But if I print,x
I get 3.3
I'm sure theres some misunderstanding on my part here, but is there a
document I could read
concerning how the print command works with regard to floating point
precision?

Thanks

Report message to a moderator

[Message index]

print and precision

By: elwood on Thu, 14 February 2008 08:50

Previous Topic:	Re: IDLVM arguments
Next Topic:	N_ELEMENTS gt 0

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Fri Oct 10 17:39:44 PDT 2025

Total time taken to generate the page: 1.12301 seconds