comp.lang.idl-pvwave archive: archive

Home » Public Forums » archive » Re: IDL mathematics

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: IDL mathematics [message #2704 is a reply to message #2703]

Wed, 31 August 1994 13:20

paul
Messages: 22
Registered: June 1991

Junior Member

In article <342fs2$8sg@yuma.ACNS.ColoState.EDU>, Kelly Dean writes:
>
> How come the following expression gives the wrong answer.
>
> J = 201+((1461*(1994+4799))/4)-(3*((1994+4899)/100)/4)-2465022
>
> J = -2457713
>
The problem is with the large vlaue of 1461*(1994+4799), which overflows
one could do

IDL> print, 201+((LONG(1461)*(1994+4799))/4-3*((1994+4899)/100)/4)-24650 22
IDL> 16271

note that

IDL> print, 1461.*(1994+4799)
9.92457e+06

is like

IDL> print, fix( 9.92457E06)
28634

and not like

IDL> print, long( 9.92457E06)
9924570

Using LONG rather than converting to floating point preserves any integer
truncation properties that may (or may not) be desired.

Paul Schopf
Oceans and Ice Branch
NASA GSFC

Report message to a moderator

[Message index]

		Re: IDL mathematics By: velt on Wed, 31 August 1994 13:07
		Re: IDL mathematics By: paul on Wed, 31 August 1994 13:20

Previous Topic:	Re: More Q's on TV and Postscript
Next Topic:	More Q's on TV and Postscript

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sun Oct 12 08:33:28 PDT 2025

Total time taken to generate the page: 1.99942 seconds