| bug in plotting with large long integers [message #14689] |
Tue, 23 March 1999 00:00  |
David Ritscher
Messages: 30
Registered: August 1995
|
Member |
|
|
I just came across an interesting bug in IDL and PV-Wave:
plot, /ynozero, lindgen(2000) + long(1e9)
It seems to be thrown off by numeric problems, resulting in a staircase
instead of a straight line.
David Ritscher
--
Cardiac Rhythm Management Laboratory
Department of Medicine
University of Alabama at Birmingham
B168 Volker Hall - 1670 University Boulevard
Birmingham AL 35294-0019
Tel: (205) 975-2122 Fax: (205) 975-4720
|
|
|
|
| Re: bug in plotting with large long integers [message #14773 is a reply to message #14689] |
Wed, 24 March 1999 00:00  |
Axel vom Endt
Messages: 9
Registered: March 1999
|
Junior Member |
|
|
David Ritscher wrote:
>
> I just came across an interesting bug in IDL and PV-Wave:
>
> plot, /ynozero, lindgen(2000) + long(1e9)
>
> It seems to be thrown off by numeric problems, resulting in a staircase
> instead of a straight line.
>
> David Ritscher
Yes, but this behaviour is documented in the online help of PLOT:
"This argument is converted to single-precision floating-point before
plotting. Plots created with PLOT are limited to the range and precision
of single-precision floating-point values."
Your long(1e9) is converted to float. Try
IDL> print, float(long(1e9) + 1L) - long(1e9)
0.00000
Hope that helps
Axel
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
