comp.lang.idl-pvwave archive: archive » Re: Precision Problem

Home » Public Forums » archive » Re: Precision Problem

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: Precision Problem [message #49393]

Mon, 24 July 2006 12:34

Paul Van Delst[1]
Messages: 1157
Registered: April 2002

Senior Member

Paul Van Delst wrote:
> You're correct. If you want more than 6/7 decimal place precision, you

I really should have said if you want more than 6/7 meaningful significant figures...

> need double (at least, for it to be meaningful):
>
> IDL> x=10000000.0
> IDL> print, x, format='(f12.2)'
> 10000000.00
> IDL> x=x+0.1
> IDL> print, x, format='(f12.2)'
> 10000000.00
>
> IDL> x=10000000.0d
> IDL> print, x, format='(f12.2)'
> 10000000.00
> IDL> x=x+0.1d
> IDL> print, x, format='(f12.2)'
> 10000000.10
>
>
> paulv
>

--
Paul van Delst Ride lots.
CIMSS @ NOAA/NCEP/EMC Eddy Merckx
Ph: (301)763-8000 x7748
Fax:(301)763-8545

Report message to a moderator

Re: Precision Problem [message #49394 is a reply to message #49393]

Mon, 24 July 2006 12:33

Paul Van Delst[1]
Messages: 1157
Registered: April 2002

Senior Member

Bruce Bowler wrote:
> On Mon, 24 Jul 2006 09:51:40 -0700, DMac put fingers to keyboard and said:
>
>
>> shapefile. The X coordinates are fine. The X coordinates are 6 digits
>> to the left of the decimal place while the Y coordinates are 7 digits
>> to the left of the decimal place
>
>
> As Paul pointed out, you're getting single precision rather then double
> precision.
>
> Are the 6 and 7 digits before or after the conversion? If after, the
> problems are both related as 7 digits is typically the limit of the number
> of significant digits in a SP number (I think :-). The range is
> 10^(+/-)38, but # of significant digits is much less.

Ah, yeah. I misunderstood the explanation and totaly missed the 6-7 digit clue there. I
thought the y-values were just integers, no decimals at all.

You're correct. If you want more than 6/7 decimal place precision, you need double (at
least, for it to be meaningful):

IDL> x=10000000.0
IDL> print, x, format='(f12.2)'
10000000.00
IDL> x=x+0.1
IDL> print, x, format='(f12.2)'
10000000.00

IDL> x=10000000.0d
IDL> print, x, format='(f12.2)'
10000000.00
IDL> x=x+0.1d
IDL> print, x, format='(f12.2)'
10000000.10

paulv

--
Paul van Delst Ride lots.
CIMSS @ NOAA/NCEP/EMC Eddy Merckx
Ph: (301)763-8000 x7748
Fax:(301)763-8545

Report message to a moderator

Re: Precision Problem [message #49397 is a reply to message #49394]

Mon, 24 July 2006 12:19

Bruce Bowler
Messages: 128
Registered: September 1998

Senior Member

On Mon, 24 Jul 2006 09:51:40 -0700, DMac put fingers to keyboard and said:

> shapefile. The X coordinates are fine. The X coordinates are 6 digits
> to the left of the decimal place while the Y coordinates are 7 digits
> to the left of the decimal place

As Paul pointed out, you're getting single precision rather then double
precision.

Are the 6 and 7 digits before or after the conversion? If after, the
problems are both related as 7 digits is typically the limit of the number
of significant digits in a SP number (I think :-). The range is
10^(+/-)38, but # of significant digits is much less.

Bruce

--
+-------------------+--------------------------------------- ------------+
Bruce Bowler | People want economy and they will pay any price to
1.207.633.9600 | get it. - Lee Iacocca
bbowler@bigelow.org |
+-------------------+--------------------------------------- ------------+

Report message to a moderator

Re: Precision Problem [message #49398 is a reply to message #49397]

Mon, 24 July 2006 12:00

Paul Van Delst[1]
Messages: 1157
Registered: April 2002

Senior Member

DMac wrote:
> Hello All,
> Using IDL version 6.2 on a Windows XP machine I am reading in a set of
> coordinates from a binary file (it is LiDAR data stored in LAS Version
> 1.0 format) stored in UTM NAD83 Zone 11 and these points are stored as
> 4 byte long integers in the binary file . The coordinates are stored
> in the binary file with an offset such that they need to be divided by
> 100 to obtain the actual coordinates. The binary file is read into a
> heap variable (Data below). The X and Y coordinates are than written
> from the heap variable into a double precision array as follows:
> x = dblarr(Num_pts)
> x = TEMPORARY(data.x*1.000e-002)
>
> y = dblarr(Num_pts)
> y = TEMPORARY(data.y*1.000e-002)

That is going to give you single precision arrays.

IDL> x=dblarr(num_pts)
IDL> x = TEMPORARY(data.x*1.000e-002)
IDL> help, x
X FLOAT = Array[100]

You need to do
IDL> x = TEMPORARY(data.x*1.000d-002)
IDL> help, x
X DOUBLE = Array[100]

i.e. using 1.000d-002 rather than the single precision literal 1.000e-002.

I don't think the TEMPORARY() is doing anything either since the argument is an expression
which is pretty much temporary already.

All you need to is:

scale_factor=1.0d-02
x = data.x*scale_factor
y = data.y*scale_factor

to get the double precision arrays at the required size.

As to why your final "y" array is bogus, I don't know. Maybe data.y is somehow buggered up
prior to your scaling?. My little tests gave the expected result.

Maybe on Windows if you do a TEMPORARY(data.x*1.000e-002) is whacks the entire structure?
(I hope not)

paulv

--
Paul van Delst Ride lots.
CIMSS @ NOAA/NCEP/EMC Eddy Merckx
Ph: (301)763-8000 x7748
Fax:(301)763-8545

Report message to a moderator

Re: Precision Problem [message #49486 is a reply to message #49393]

Mon, 24 July 2006 14:05

DMac
Messages: 6
Registered: May 2006

Junior Member

Thanks a bunch guys. That was the problem i.e. I should use 1.000d-002
rather than the single precision literal 1.000e-002.

Solved all my problems.

I really appreciate the prompt and accurate replies.

Derek M.

Paul Van Delst wrote:
> Paul Van Delst wrote:
>> You're correct. If you want more than 6/7 decimal place precision, you
>
> I really should have said if you want more than 6/7 meaningful significant figures...
>
>
>> need double (at least, for it to be meaningful):
>>
>> IDL> x=10000000.0
>> IDL> print, x, format='(f12.2)'
>> 10000000.00
>> IDL> x=x+0.1
>> IDL> print, x, format='(f12.2)'
>> 10000000.00
>>
>> IDL> x=10000000.0d
>> IDL> print, x, format='(f12.2)'
>> 10000000.00
>> IDL> x=x+0.1d
>> IDL> print, x, format='(f12.2)'
>> 10000000.10
>>
>>
>> paulv
>>
>
>
> --
> Paul van Delst Ride lots.
> CIMSS @ NOAA/NCEP/EMC Eddy Merckx
> Ph: (301)763-8000 x7748
> Fax:(301)763-8545

Report message to a moderator

Previous Topic:	Close window command?
Next Topic:	Re: problem loading color palette

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sat Oct 11 08:49:59 PDT 2025

Total time taken to generate the page: 1.03932 seconds