comp.lang.idl-pvwave archive: archive » Floating-point accuracy problem

Home » Public Forums » archive » Floating-point accuracy problem

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Floating-point accuracy problem [message #72109]

Mon, 09 August 2010 08:30

Ramy
Messages: 1
Registered: August 2010

Junior Member

I have a problem related to float-point accuracy
If I type in: 50d - 1d-9, I get 50.000000

And here lies my problem, I'm doing a numerical simulation where such
an arithmetic is common place, and as a result i get a lot or errors.

I know for example, that if i simply type
print, 50d - 1d-9, format = '(f.20.10)' , i'll get:
49.9999999990

But how can I convince IDL to do it on its own during computations?

Any suggestions will be massively welcomed!

Thank you

Report message to a moderator

[Message index]

Floating-point accuracy problem

By: Ramy on Mon, 09 August 2010 08:30

Previous Topic:	svdfit-measure_errors
Next Topic:	Re: Floating-point accuracy problem

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Wed Oct 08 16:17:48 PDT 2025

Total time taken to generate the page: 0.00174 seconds