| Re: IDL mathematics [message #2703] |
Wed, 31 August 1994 13:07  |
velt
Messages: 19
Registered: June 1994
|
Junior Member |
|
|
>
> How come the following expression gives the wrong answer.
>
> J = 201+((1461*(1994+4799))/4)-(3*((1994+4899)/100)/4)-2465022
>
> J = -2457713
>
> It should be:
>
> J = 201+((1461.*(1994+4799.))/4.)-(3.*((1994+4899.)/100.)/4.)-24 65022.
>
> J = 16270.5
>
>
> Kelly Dean
Most of your calculations are done in 2-byte integer, upto the last
subtraction. The intermediate results are overflowing, so you get
the wrong answer. Throw in a few longs at strategic places:
J = 201+((1461*(1994L+4799))/4)-(3*((1994L+4899)/100)/4)-2465022 print,J
--> 16271
Robert Velthuizen,
Digital Medical Imaging Program of the
H. Lee Moffitt Cancer Center and Research Institute at the
University of South Florida.
|
|
|
|
| Re: IDL mathematics [message #2704 is a reply to message #2703] |
Wed, 31 August 1994 13:20  |
paul
Messages: 22
Registered: June 1991
|
Junior Member |
|
|
In article <342fs2$8sg@yuma.ACNS.ColoState.EDU>, Kelly Dean writes:
>
> How come the following expression gives the wrong answer.
>
> J = 201+((1461*(1994+4799))/4)-(3*((1994+4899)/100)/4)-2465022
>
> J = -2457713
>
The problem is with the large vlaue of 1461*(1994+4799), which overflows
one could do
IDL> print, 201+((LONG(1461)*(1994+4799))/4-3*((1994+4899)/100)/4)-24650 22
IDL> 16271
note that
IDL> print, 1461.*(1994+4799)
9.92457e+06
is like
IDL> print, fix( 9.92457E06)
28634
and not like
IDL> print, long( 9.92457E06)
9924570
Using LONG rather than converting to floating point preserves any integer
truncation properties that may (or may not) be desired.
Paul Schopf
Oceans and Ice Branch
NASA GSFC
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
