comp.lang.idl-pvwave archive: archive » fix(4.70*100) is... 469

Home » Public Forums » archive » fix(4.70*100) is... 469

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: fix(4.70*100) is... 469 [message #53612 is a reply to message #53578]

Fri, 20 April 2007 09:31

James Kuyper
Messages: 425
Registered: March 2000

Senior Member

Paolo Grigis wrote:
> David Fanning wrote:
>> Jean H. writes:
>>
>>> What is being displayed is a bit different than what is being stored..
>>> IDL> print, 470.0 - (4.70*100)
>>> 3.05176e-005
>>
>> I guess I would argue that what is being displayed is EXACTLY
>> what is being stored:
>
> Is it?
> I always wondered if the garbage after the 16th digit does
> correspond indeed to the exact decimal representation of the
> double stored in memory in binary format, or is just a meaningless
> side product of the binary to decimal conversion...
>
> IDL> print,!DPi,format='(f68.64)'
> 3.1415926535897931159979634685441851615905761718750000000000 000000

The fractional part of a floating point number represented in binary
format can always be written as

>
>
> Ciao,
> Paolo
>
>
>>
>> IDL> print, 470, format='(f18.14)'
>> 470.00000000000000
>> IDL> print, 4.70*100, format='(f18.14)'
>> 469.99996948242187
>> IDL> print, 470.00000000000000D - 469.99996948242187D, format='(f18.14)'
>> 0.00003051757813
>>
>> Cheers,
>>
>> David

Report message to a moderator

[Message index]

		fix(4.70*100) is... 469 By: b_efremova@yahoo.com on Wed, 18 April 2007 15:48
		Re: fix(4.70*100) is... 469 By: mmeron on Fri, 20 April 2007 10:30
		Re: fix(4.70*100) is... 469 By: Bruce Bowler on Fri, 20 April 2007 09:47
		Re: fix(4.70*100) is... 469 By: James Kuyper on Fri, 20 April 2007 09:43
		Re: fix(4.70*100) is... 469 By: James Kuyper on Fri, 20 April 2007 09:31
		Re: fix(4.70*100) is... 469 By: James Kuyper on Fri, 20 April 2007 09:26
		Re: fix(4.70*100) is... 469 By: Paul Van Delst[1] on Fri, 20 April 2007 07:27
		Re: fix(4.70*100) is... 469 By: Paul Van Delst[1] on Thu, 19 April 2007 13:44
		Re: fix(4.70*100) is... 469 By: b_efremova@yahoo.com on Thu, 19 April 2007 12:43

Previous Topic:	idl 6.1 crashes on linux
Next Topic:	HDF5 data

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Tue Dec 02 11:34:32 PST 2025

Total time taken to generate the page: 0.56165 seconds