| Re: accuracy problems [message #19904] |
Fri, 28 April 2000 00:00 |
davidf
Messages: 2866
Registered: September 1996
|
Senior Member |
|
|
J�rg Schliwa (joerg.schliwa@ewetel.net) writes:
> The solution of my problem was just the different country specific setting
> in IDL and Windows NT.
> In Windows NT I had the German setting with comma as decimal point. This
> seem to cause the trouble in IDL. I changed the NT settings and now IDL
> works just normal.
Whew! My faith in IDL is restored. :-)
Thanks for letting us know what the problem was.
Cheers,
David
P.S. Let's just say I'm feeling better today, and I've
written a note to myself that I placed on my computer
screen. It says, "It's too nice a day to be GRUMPY."
I'm going to try to read it before I make any more
newsgroup postings. :-)
--
David Fanning, Ph.D.
Fanning Software Consulting
Phone: 970-221-0438 E-Mail: davidf@dfanning.com
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Toll-Free IDL Book Orders: 1-888-461-0155
|
|
|
|
| Re: accuracy problems [message #19908 is a reply to message #19904] |
Fri, 28 April 2000 00:00  |
J�rg Schliwa
Messages: 12
Registered: April 2000
|
Junior Member |
|
|
HY David!
The solution of my problem was just the different country specific setting
in IDL and Windows NT.
In Windows NT I had the German setting with comma as decimal point. This
seem to cause the trouble in IDL. I changed the NT settings and now IDL
works just normal.
Cheerio,
J�rg
David Fanning schrieb in Nachricht ...
> J�rg Schliwa (joerg.schliwa@ewetel.net) writes:
>
>> I've a very strange problem: I'm running IDL 5.2.1 on a Pentium III 500
with
>> Windows NT. When I read data from a file into a variable, IDL always
makes a
>> truncation. Even when I type it at the IDL prompt.
>> For example:
>>
>> IDL> a=2.83456
>> IDL> print, a
>>
>> IDL prints : 2.00000
>>
>> Its equal if I define the variable as float or double, the result is
always
>> the same.
>>
>> Does anyone have a idea how this can happen,
>
> You have the Student Edition of IDL. All numbers are
> automatically truncated to the 10 counting digits you
> have available on your hands. A compiler option will
> extend this to the toe digits as well.
>
> Cheers,
>
> David
>
> P.S. Let's just say there is something seriously wrong
> with IDL, or you are not telling us *something* about
> how you created those numbers. You aren't trying to get
> the variable a by reading it from the keyboard into a
> integer variable you? :-)
>
> --
> David Fanning, Ph.D.
> Fanning Software Consulting
> Phone: 970-221-0438 E-Mail: davidf@dfanning.com
> Coyote's Guide to IDL Programming: http://www.dfanning.com/
> Toll-Free IDL Book Orders: 1-888-461-0155
|
|
|
|
| Re: accuracy problems [message #19920 is a reply to message #19904] |
Thu, 27 April 2000 00:00 |
davidf
Messages: 2866
Registered: September 1996
|
Senior Member |
|
|
J�rg Schliwa (joerg.schliwa@ewetel.net) writes:
> I've a very strange problem: I'm running IDL 5.2.1 on a Pentium III 500 with
> Windows NT. When I read data from a file into a variable, IDL always makes a
> truncation. Even when I type it at the IDL prompt.
> For example:
>
> IDL> a=2.83456
> IDL> print, a
>
> IDL prints : 2.00000
>
> Its equal if I define the variable as float or double, the result is always
> the same.
>
> Does anyone have a idea how this can happen,
You have the Student Edition of IDL. All numbers are
automatically truncated to the 10 counting digits you
have available on your hands. A compiler option will
extend this to the toe digits as well.
Cheers,
David
P.S. Let's just say there is something seriously wrong
with IDL, or you are not telling us *something* about
how you created those numbers. You aren't trying to get
the variable a by reading it from the keyboard into a
integer variable you? :-)
--
David Fanning, Ph.D.
Fanning Software Consulting
Phone: 970-221-0438 E-Mail: davidf@dfanning.com
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Toll-Free IDL Book Orders: 1-888-461-0155
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
