comp.lang.idl-pvwave archive: archive » Re: Precision Problem

Home » Public Forums » archive » Re: Precision Problem

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: Precision Problem [message #49394 is a reply to message #49393]

Mon, 24 July 2006 12:33

Paul Van Delst[1]
Messages: 1157
Registered: April 2002

Senior Member

Bruce Bowler wrote:
> On Mon, 24 Jul 2006 09:51:40 -0700, DMac put fingers to keyboard and said:
>
>
>> shapefile. The X coordinates are fine. The X coordinates are 6 digits
>> to the left of the decimal place while the Y coordinates are 7 digits
>> to the left of the decimal place
>
>
> As Paul pointed out, you're getting single precision rather then double
> precision.
>
> Are the 6 and 7 digits before or after the conversion? If after, the
> problems are both related as 7 digits is typically the limit of the number
> of significant digits in a SP number (I think :-). The range is
> 10^(+/-)38, but # of significant digits is much less.

Ah, yeah. I misunderstood the explanation and totaly missed the 6-7 digit clue there. I
thought the y-values were just integers, no decimals at all.

You're correct. If you want more than 6/7 decimal place precision, you need double (at
least, for it to be meaningful):

IDL> x=10000000.0
IDL> print, x, format='(f12.2)'
10000000.00
IDL> x=x+0.1
IDL> print, x, format='(f12.2)'
10000000.00

IDL> x=10000000.0d
IDL> print, x, format='(f12.2)'
10000000.00
IDL> x=x+0.1d
IDL> print, x, format='(f12.2)'
10000000.10

paulv

--
Paul van Delst Ride lots.
CIMSS @ NOAA/NCEP/EMC Eddy Merckx
Ph: (301)763-8000 x7748
Fax:(301)763-8545

Report message to a moderator

[Message index]

		Re: Precision Problem By: Paul Van Delst[1] on Mon, 24 July 2006 12:34
		Re: Precision Problem By: Paul Van Delst[1] on Mon, 24 July 2006 12:33
		Re: Precision Problem By: Bruce Bowler on Mon, 24 July 2006 12:19
		Re: Precision Problem By: Paul Van Delst[1] on Mon, 24 July 2006 12:00
		Re: Precision Problem By: DMac on Mon, 24 July 2006 14:05

Previous Topic:	Close window command?
Next Topic:	Re: problem loading color palette

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sat Oct 11 20:21:41 PDT 2025

Total time taken to generate the page: 1.84354 seconds