comp.lang.idl-pvwave archive
Messages from Usenet group comp.lang.idl-pvwave, compiled by Paulo Penteado
Home » Public Forums » archive » precision problem
Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend  		Switch to threaded view of this topic Create a new topic Submit Reply
precision problem [message #67664] Tue, 18 August 2009 05:37
skymaxwell@gmail.com is currently offline  skymaxwell@gmail.com
Messages: 127
Registered: January 2007
Senior Member
I have problem with precision.
I typing

IDL> b=13.37943889D

and then i typing

IDL> help,b
B DOUBLE = 13.379439

how get my 13.37943889 for calculus ?
If IDL rounding value, i've got worst results
Even for test example

How avoid this problem ?

Thanks

Report message to a moderator

  Switch to threaded view of this topic Create a new topic Submit Reply
Previous Topic: What a Wild, Wacky Place
Next Topic: Precision Problem

-=] Back to Top [=-
[ Syndicate this forum (XML) ] [ RSS ] [ PDF ]

Current Time: Wed Oct 08 13:49:02 PDT 2025

Total time taken to generate the page: 0.00468 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.7.
Copyright ©2001-2013 FUDforum Bulletin Board Software