comp.lang.idl-pvwave archive: archive

Home » Public Forums » archive » Re: IDL mathematics

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: IDL mathematics [message #2703]

Wed, 31 August 1994 13:07

velt
Messages: 19
Registered: June 1994

Junior Member

>
> How come the following expression gives the wrong answer.
>
> J = 201+((1461*(1994+4799))/4)-(3*((1994+4899)/100)/4)-2465022
>
> J = -2457713
>
> It should be:
>
> J = 201+((1461.*(1994+4799.))/4.)-(3.*((1994+4899.)/100.)/4.)-24 65022.
>
> J = 16270.5
>
>
> Kelly Dean

Most of your calculations are done in 2-byte integer, upto the last
subtraction. The intermediate results are overflowing, so you get
the wrong answer. Throw in a few longs at strategic places:

J = 201+((1461*(1994L+4799))/4)-(3*((1994L+4899)/100)/4)-2465022 print,J
--> 16271

Robert Velthuizen,
Digital Medical Imaging Program of the
H. Lee Moffitt Cancer Center and Research Institute at the
University of South Florida.

Report message to a moderator

[Message index]

		Re: IDL mathematics By: velt on Wed, 31 August 1994 13:07
		Re: IDL mathematics By: paul on Wed, 31 August 1994 13:20

Previous Topic:	Re: More Q's on TV and Postscript
Next Topic:	More Q's on TV and Postscript

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Tue Dec 02 12:52:58 PST 2025

Total time taken to generate the page: 0.15682 seconds