| Re: round not behaving [message #46001 is a reply to message #46000] |
Thu, 27 October 2005 14:09   |
Dick Jackson
Messages: 347
Registered: August 1998
|
Senior Member |
|
|
<psbeps@hotmail.com> wrote in message
news:1130442231.788271.28210@f14g2000cwb.googlegroups.com...
> I am trying to round to the nearest tenth, and it works most of the
> time. It doesn't work if the value is 5.05 or 5.55, which get rounded
> to 5.0 and 5.5, respectively, instead of 5.1 and 5.6. When I type it
> into the command line, it works as expected, but not when I run the
> entire program.
>
> Here is the code (which I have divided into steps to see where it's
> going wrong--it still doesn't work if I do the same thing in one line
> of code):
>
> ;** Round off the value.
> help, value
> print, value
> temp = value * 10.0
> help, temp
> print, temp
> rounded_value = ROUND(temp)
> help, rounded_value
> print, rounded_value
> value = rounded_value/10.0
> help, value
> print, value
>
> Here is the output:
> VALUE FLOAT = 5.05000
> 5.05000
> TEMP FLOAT = 50.5000
> 50.5000
> ROUNDED_VALUE LONG = 50
> 50
> VALUE FLOAT = 5.00000
> 5.00000
>
> Why does it not round properly?
>
> Jill
>
Good question! And if I start with value of exactly 5.05, that's what I get:
IDL> value=5.05
IDL> help, value
VALUE FLOAT = 5.05000
IDL> print, value
5.05000
IDL> temp = value * 10.0
IDL> help, temp
TEMP FLOAT = 50.5000
IDL> print, temp
50.5000
IDL> rounded_value = ROUND(temp)
IDL> help, rounded_value
ROUNDED_VALUE LONG = 51
IDL> print, rounded_value
51
IDL> value = rounded_value/10.0
IDL> help, value
VALUE FLOAT = 5.10000
IDL> print, value
5.10000
But what if "value" only *looks* like 5.05, but only because of the display
precision... what if it is really 5.049999?
IDL> value=5.049999
IDL> help, value
VALUE FLOAT = 5.05000
IDL> print, value
5.05000
IDL> temp = value * 10.0
IDL> help, temp
TEMP FLOAT = 50.5000
IDL> print, temp
50.5000
IDL> rounded_value = ROUND(temp)
IDL> help, rounded_value
ROUNDED_VALUE LONG = 50
IDL> print, rounded_value
50
IDL> value = rounded_value/10.0
IDL> help, value
VALUE FLOAT = 5.00000
IDL> print, value
5.00000
That seems to be what was happening in your example. Can you check the value
in your program more closely?
IDL> value=5.049999
IDL> print,value
5.05000
IDL> print,value,Format='(F19.15)'
5.049999237060547
This is covered further at http://dfanning.com/math_tips/sky_is_falling.html
Cheers,
--
-Dick
Dick Jackson / dick@d-jackson.com
D-Jackson Software Consulting / http://www.d-jackson.com
Calgary, Alberta, Canada / +1-403-242-7398 / Fax: 241-7392
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
