comp.lang.idl-pvwave archive: archive

Home » Public Forums » archive » accuracy problems

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

accuracy problems [message #19921]

Thu, 27 April 2000 00:00

J�rg Schliwa
Messages: 12
Registered: April 2000

Junior Member

HY to all!

I've a very strange problem: I'm running IDL 5.2.1 on a Pentium III 500 with
Windows NT. When I read data from a file into a variable, IDL always makes a
truncation. Even when I type it at the IDL prompt.
For example:

IDL> a=2.83456
IDL> print, a

IDL prints : 2.00000

Its equal if I define the variable as float or double, the result is always
the same.

Does anyone have a idea how this can happen,

J�rg

Report message to a moderator

[Message index]

		accuracy problems By: J�rg Schliwa on Thu, 27 April 2000 00:00
		Re: Accuracy problem By: kehcheng on Sat, 11 March 2006 22:45
		Re: Accuracy problem By: Juan Arrieta on Sat, 11 March 2006 19:37
		Re: Accuracy problem By: mmeron on Sat, 11 March 2006 16:25
		Re: Accuracy problem By: David Fanning on Sat, 11 March 2006 06:14
		Re: Accuracy problem By: James Kuyper on Sat, 11 March 2006 03:56
		Re: Accuracy problem By: mmeron on Fri, 10 March 2006 22:52
		Re: Accuracy problem By: David Fanning on Fri, 10 March 2006 21:43

Previous Topic:	Re: How can I find the central pixel?
Next Topic:	Floats

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sun Oct 12 10:30:53 PDT 2025

Total time taken to generate the page: 0.72004 seconds