comp.lang.idl-pvwave archive: archive » Re: accuracy problems

Home » Public Forums » archive » Re: accuracy problems

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: accuracy problems [message #19920 is a reply to message #19904]

Thu, 27 April 2000 00:00

davidf
Messages: 2866
Registered: September 1996

Senior Member

J�rg Schliwa (joerg.schliwa@ewetel.net) writes:

> I've a very strange problem: I'm running IDL 5.2.1 on a Pentium III 500 with
> Windows NT. When I read data from a file into a variable, IDL always makes a
> truncation. Even when I type it at the IDL prompt.
> For example:
>
> IDL> a=2.83456
> IDL> print, a
>
> IDL prints : 2.00000
>
> Its equal if I define the variable as float or double, the result is always
> the same.
>
> Does anyone have a idea how this can happen,

You have the Student Edition of IDL. All numbers are
automatically truncated to the 10 counting digits you
have available on your hands. A compiler option will
extend this to the toe digits as well.

Cheers,

David

P.S. Let's just say there is something seriously wrong
with IDL, or you are not telling us *something* about
how you created those numbers. You aren't trying to get
the variable a by reading it from the keyboard into a
integer variable you? :-)

--
David Fanning, Ph.D.
Fanning Software Consulting
Phone: 970-221-0438 E-Mail: davidf@dfanning.com
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Toll-Free IDL Book Orders: 1-888-461-0155

Report message to a moderator

[Message index]

		Re: accuracy problems By: davidf on Fri, 28 April 2000 00:00
		Re: accuracy problems By: J�rg Schliwa on Fri, 28 April 2000 00:00
		Re: accuracy problems By: davidf on Thu, 27 April 2000 00:00

Previous Topic:	Interploation/extrapolation problem
Next Topic:	Re: surface,/horizontal ;inadequate

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sat Nov 29 13:20:00 PST 2025

Total time taken to generate the page: 1.20097 seconds