| Re: Floating-point accuracy problem [message #72108] |
Mon, 09 August 2010 09:09 |
Michael Galloy
Messages: 1114
Registered: April 2006
|
Senior Member |
|
|
On 8/9/10 9:30 AM, Ramy wrote:
> I have a problem related to float-point accuracy
> If I type in: 50d - 1d-9, I get 50.000000
>
> And here lies my problem, I'm doing a numerical simulation where such
> an arithmetic is common place, and as a result i get a lot or errors.
>
> I know for example, that if i simply type
> print, 50d - 1d-9, format = '(f.20.10)' , i'll get:
> 49.9999999990
>
> But how can I convince IDL to do it on its own during computations?
>
> Any suggestions will be massively welcomed!
You are confusing the computations with the printed output. IDL will do
double precision arithmetic if one of the arguments is a double. When
you print it out, you can choose the format you want to print it out in:
IDL> d = 50d - 1d-9
IDL> print, d
50.000000
IDL> print, d, format='(F20.10)'
49.9999999990
Mike
--
www.michaelgalloy.com
Research Mathematician
Tech-X Corporation
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
