| Re: round not behaving [message #46000] |
Thu, 27 October 2005 14:40  |
Robert Moss
Messages: 74
Registered: February 1996
|
Member |
|
|
psbeps@hotmail.com wrote:
> I am trying to round to the nearest tenth, and it works most of the
> time. It doesn't work if the value is 5.05 or 5.55, which get rounded
> to 5.0 and 5.5, respectively, instead of 5.1 and 5.6. When I type it
> into the command line, it works as expected, but not when I run the
> entire program.
>
> Here is the code (which I have divided into steps to see where it's
> going wrong--it still doesn't work if I do the same thing in one line
> of code):
>
> ;** Round off the value.
> help, value
> print, value
> temp = value * 10.0
> help, temp
> print, temp
> rounded_value = ROUND(temp)
> help, rounded_value
> print, rounded_value
> value = rounded_value/10.0
> help, value
> print, value
>
> Here is the output:
> VALUE FLOAT = 5.05000
> 5.05000
> TEMP FLOAT = 50.5000
> 50.5000
> ROUNDED_VALUE LONG = 50
> 50
> VALUE FLOAT = 5.00000
> 5.00000
>
> Why does it not round properly?
>
> Jill
>
If it works at the command line (which this code does), but it does not
work in a program, you are probably looking in the wrong place for the
error. In other words, the error is not in the lines above, but probably
somewhere after that (assuming that the lines above tell the whole
story). Post the actual problem code and we can probably help you.
--
Robert Moss, PhD
|
|
|
|
| Re: round not behaving [message #46001 is a reply to message #46000] |
Thu, 27 October 2005 14:09  |
Dick Jackson
Messages: 347
Registered: August 1998
|
Senior Member |
|
|
<psbeps@hotmail.com> wrote in message
news:1130442231.788271.28210@f14g2000cwb.googlegroups.com...
> I am trying to round to the nearest tenth, and it works most of the
> time. It doesn't work if the value is 5.05 or 5.55, which get rounded
> to 5.0 and 5.5, respectively, instead of 5.1 and 5.6. When I type it
> into the command line, it works as expected, but not when I run the
> entire program.
>
> Here is the code (which I have divided into steps to see where it's
> going wrong--it still doesn't work if I do the same thing in one line
> of code):
>
> ;** Round off the value.
> help, value
> print, value
> temp = value * 10.0
> help, temp
> print, temp
> rounded_value = ROUND(temp)
> help, rounded_value
> print, rounded_value
> value = rounded_value/10.0
> help, value
> print, value
>
> Here is the output:
> VALUE FLOAT = 5.05000
> 5.05000
> TEMP FLOAT = 50.5000
> 50.5000
> ROUNDED_VALUE LONG = 50
> 50
> VALUE FLOAT = 5.00000
> 5.00000
>
> Why does it not round properly?
>
> Jill
>
Good question! And if I start with value of exactly 5.05, that's what I get:
IDL> value=5.05
IDL> help, value
VALUE FLOAT = 5.05000
IDL> print, value
5.05000
IDL> temp = value * 10.0
IDL> help, temp
TEMP FLOAT = 50.5000
IDL> print, temp
50.5000
IDL> rounded_value = ROUND(temp)
IDL> help, rounded_value
ROUNDED_VALUE LONG = 51
IDL> print, rounded_value
51
IDL> value = rounded_value/10.0
IDL> help, value
VALUE FLOAT = 5.10000
IDL> print, value
5.10000
But what if "value" only *looks* like 5.05, but only because of the display
precision... what if it is really 5.049999?
IDL> value=5.049999
IDL> help, value
VALUE FLOAT = 5.05000
IDL> print, value
5.05000
IDL> temp = value * 10.0
IDL> help, temp
TEMP FLOAT = 50.5000
IDL> print, temp
50.5000
IDL> rounded_value = ROUND(temp)
IDL> help, rounded_value
ROUNDED_VALUE LONG = 50
IDL> print, rounded_value
50
IDL> value = rounded_value/10.0
IDL> help, value
VALUE FLOAT = 5.00000
IDL> print, value
5.00000
That seems to be what was happening in your example. Can you check the value
in your program more closely?
IDL> value=5.049999
IDL> print,value
5.05000
IDL> print,value,Format='(F19.15)'
5.049999237060547
This is covered further at http://dfanning.com/math_tips/sky_is_falling.html
Cheers,
--
-Dick
Dick Jackson / dick@d-jackson.com
D-Jackson Software Consulting / http://www.d-jackson.com
Calgary, Alberta, Canada / +1-403-242-7398 / Fax: 241-7392
|
|
|
|
| Re: round not behaving [message #46146 is a reply to message #46001] |
Thu, 27 October 2005 14:54 |
psbeps
Messages: 8
Registered: March 2005
|
Junior Member |
|
|
I suspected it might be something like that going on, and sure enough,
you were right. I printed it out how you suggested and got this:
5.049999713897705
So I guess I just have to accept that it is rounding properly after all
(will go read the sky is falling link).
Thank you,
Jill
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
