| Re: fix(4.70*100) is... 469 [message #53556 is a reply to message #53555] |
Thu, 19 April 2007 08:02   |
Paolo Grigis
Messages: 171
Registered: December 2003
|
Senior Member |
|
|
At the risk of reiterating what I already stated earlier:
fix truncates your number when converting it to integer.
Compare:
IDL> print,4.999999
5.00000
IDL> print,fix(4.999999)
4
You don't like this behavior? Fine, just use round instead:
IDL> print,round(4.999999)
5
Ciao,
Paolo
b_efremova@yahoo.com wrote:
> Sorry Guys, I should have made myself clearer.
> I'm afraid David this is not actually the question you describe in
> your article.
> and I do not expect better accuracy than I provide.
>
> There is nothing wrong here with the floating point accuracy.
> print,4.700*100.00
> 470.000
>
> It is the conversion to integer (I imagine) which makes no sence.
>
> print,fix(4.700*100.00)
> 469
> also (which is what I really needed)
>
> print,string(4.700*100.00,format='(i3)')
> 469
>
> Cheers
> Boryana
>
>> Hooboy! Haven't had one of these in awhile. :-)
>>
>> http://www.dfanning.com/math_tips/sky_is_falling.html
>>
>> Cheers,
>>
>> David
>> --
>> David Fanning, Ph.D.
>> Fanning Software Consulting, Inc.
>> Coyote's Guide to IDL Programming: http://www.dfanning.com/
>> Sepore ma de ni thui. ("Perhaps thou speakest truth.")
>
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
