comp.lang.idl-pvwave archive: archive » Re: Precision Problem

Home » Public Forums » archive » Re: Precision Problem

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: Precision Problem [message #49397 is a reply to message #49394]

Mon, 24 July 2006 12:19

Bruce Bowler
Messages: 128
Registered: September 1998

Senior Member

On Mon, 24 Jul 2006 09:51:40 -0700, DMac put fingers to keyboard and said:

> shapefile. The X coordinates are fine. The X coordinates are 6 digits
> to the left of the decimal place while the Y coordinates are 7 digits
> to the left of the decimal place

As Paul pointed out, you're getting single precision rather then double
precision.

Are the 6 and 7 digits before or after the conversion? If after, the
problems are both related as 7 digits is typically the limit of the number
of significant digits in a SP number (I think :-). The range is
10^(+/-)38, but # of significant digits is much less.

Bruce

--
+-------------------+--------------------------------------- ------------+
Bruce Bowler | People want economy and they will pay any price to
1.207.633.9600 | get it. - Lee Iacocca
bbowler@bigelow.org |
+-------------------+--------------------------------------- ------------+

Report message to a moderator

[Message index]

		Re: Precision Problem By: Paul Van Delst[1] on Mon, 24 July 2006 12:34
		Re: Precision Problem By: Paul Van Delst[1] on Mon, 24 July 2006 12:33
		Re: Precision Problem By: Bruce Bowler on Mon, 24 July 2006 12:19
		Re: Precision Problem By: Paul Van Delst[1] on Mon, 24 July 2006 12:00
		Re: Precision Problem By: DMac on Mon, 24 July 2006 14:05

Previous Topic:	Close window command?
Next Topic:	Re: problem loading color palette

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sun Oct 12 09:44:44 PDT 2025

Total time taken to generate the page: 1.28257 seconds