comp.lang.idl-pvwave archive: archive » Re: accuracy problems

Home » Public Forums » archive » Re: accuracy problems

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: accuracy problems [message #19908 is a reply to message #19904]

Fri, 28 April 2000 00:00

J�rg Schliwa
Messages: 12
Registered: April 2000

Junior Member

HY David!

The solution of my problem was just the different country specific setting
in IDL and Windows NT.
In Windows NT I had the German setting with comma as decimal point. This
seem to cause the trouble in IDL. I changed the NT settings and now IDL
works just normal.

Cheerio,
J�rg

David Fanning schrieb in Nachricht ...
> J�rg Schliwa (joerg.schliwa@ewetel.net) writes:
>
>> I've a very strange problem: I'm running IDL 5.2.1 on a Pentium III 500
with
>> Windows NT. When I read data from a file into a variable, IDL always
makes a
>> truncation. Even when I type it at the IDL prompt.
>> For example:
>>
>> IDL> a=2.83456
>> IDL> print, a
>>
>> IDL prints : 2.00000
>>
>> Its equal if I define the variable as float or double, the result is
always
>> the same.
>>
>> Does anyone have a idea how this can happen,
>
> You have the Student Edition of IDL. All numbers are
> automatically truncated to the 10 counting digits you
> have available on your hands. A compiler option will
> extend this to the toe digits as well.
>
> Cheers,
>
> David
>
> P.S. Let's just say there is something seriously wrong
> with IDL, or you are not telling us *something* about
> how you created those numbers. You aren't trying to get
> the variable a by reading it from the keyboard into a
> integer variable you? :-)
>
> --
> David Fanning, Ph.D.
> Fanning Software Consulting
> Phone: 970-221-0438 E-Mail: davidf@dfanning.com
> Coyote's Guide to IDL Programming: http://www.dfanning.com/
> Toll-Free IDL Book Orders: 1-888-461-0155

Report message to a moderator

[Message index]

		Re: accuracy problems By: davidf on Fri, 28 April 2000 00:00
		Re: accuracy problems By: J�rg Schliwa on Fri, 28 April 2000 00:00
		Re: accuracy problems By: davidf on Thu, 27 April 2000 00:00

Previous Topic:	Interploation/extrapolation problem
Next Topic:	Re: surface,/horizontal ;inadequate

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Fri Oct 10 17:46:21 PDT 2025

Total time taken to generate the page: 1.23090 seconds