| Re: Problem with FLOAT in CW_FORM [message #16010] |
Tue, 22 June 1999 00:00 |
Martin Schultz
Messages: 515
Registered: August 1997
|
Senior Member |
|
|
Axel vom Endt wrote:
>
> Steffen Keitel wrote:
>>
>> Hello!
>> Building a widget based fit program (IDL 5.1 / Linux) I encountered the
>> following problem: I use the CW_FORM widget to enter my initial values
>> of the fit parameter. If I type e.g. "0.350" in a FLOAT field, IDL will
>> work with "0.3499999940" or something like that. How can I teach IDL to
>> use the _exact_ value?
>>
>> Thanks in advance, Steffen
>
> Hi steffen,
>
> that's a more general problem: How do you tell a computer to work with
> 0.350 ?
>
> short answer: .35 cannot be represented by a finite binary fraction.
true, but at least the program should be able to identify 0.35 as 0.35
whenever it needs to. The kind of roundoff error you describe is very
common when you change the data type from float to double or vice versa.
What would be needed to make you happy is a DOUBLE type in CW_FORM. Then
again, who really needs this?
Martin
|||||||||||||||\\\\\\\\\\\\\-------------------///////////// //|||||||||||||||
Martin Schultz, DEAS, Harvard University, 29 Oxford St., Pierce 109,
Cambridge, MA 02138 phone (617) 496 8318 fax (617) 495 4551
e-mail mgs@io.harvard.edu web http://www-as/people/staff/mgs/
|
|
|
|
| Re: Problem with FLOAT in CW_FORM [message #16018 is a reply to message #16010] |
Tue, 22 June 1999 00:00  |
Axel vom Endt
Messages: 9
Registered: March 1999
|
Junior Member |
|
|
Steffen Keitel wrote:
>
> Hello!
> Building a widget based fit program (IDL 5.1 / Linux) I encountered the
> following problem: I use the CW_FORM widget to enter my initial values
> of the fit parameter. If I type e.g. "0.350" in a FLOAT field, IDL will
> work with "0.3499999940" or something like that. How can I teach IDL to
> use the _exact_ value?
>
> Thanks in advance, Steffen
Hi steffen,
that's a more general problem: How do you tell a computer to work with
0.350 ?
short answer: .35 cannot be represented by a finite binary fraction.
Try the same in c:
#include <stdio.h>
int main(){
float x = 0.35;
printf("x = %12.10f\n",x);
return 0;
}
For a more detailed answer, refer to any textbook on numerical methods.
Axel
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
