| Set Precision??????????? [message #15790] |
Mon, 14 June 1999 00:00  |
akk
Messages: 10
Registered: July 1998
|
Junior Member |
|
|
Hi All
I'd like create a varible which always holds numbers out to 6 places
passed the decimal point, and 4 numbers before the decimal point (e.g.
1999.123456). I tried double
precision but this seems to only hold 8 digits total for decimal numbers.
However I would like to create a variable corresonding to a fraction of a
Year (e.g 1995.123456). I have not been able to find a procedure allowing
me to create 10 digit decimal number , without the number being rounded
off to 8 digits.
Does anyone know of a way to declare a variable to hold a 10 digit
decimal number?
- Thanks in advance
--------------------------
|
|
|
|
| Re: Set Precision??????????? [message #15838 is a reply to message #15790] |
Wed, 16 June 1999 00:00  |
Martin Schultz
Messages: 515
Registered: August 1997
|
Senior Member |
|
|
wbiagiot@suffolk.lib.ny.us wrote:
>
>> However I would like to create a variable corresonding to a fraction
> of a
>> Year (e.g 1995.123456). I have not been able to find a procedure
> allowing
>> me to create 10 digit decimal number , without the number being
> rounded
>> off to 8 digits.
>> Does anyone know of a way to declare a variable to hold a 10 digit
>> decimal number?
>
> The other replies were on the money with the correct answer, but also
> consider this: there are only 365 days in a year. I'm pretty sure you
> could get that resolute with even a "float" (4 byte decimal).
>
> - Bill B.
>
Bill,
you had better not hit the send button for this message!! A reply
like yours is what caused the Y2K bug (admitted: it has created many
jobs). Sure, if you are only after "daily" resolution, then a simple
float is fine, but as soon as you want to extract seconds, you get
terrible roundoff errors. And believe it or not: there are datasets in
1-second resolution (or finer) that are stored with the day of the year.
More systematically:
1 year ~ 3e7 secs ~ 3e10 msecs ~ 3e13 usecs
i.e. with 7 digits you can get to about 10 secs, with 15 digits you
achieve better than microsecond resolution.
Regards,
Martin
--
|||||||||||||||\\\\\\\\\\\\\-------------------///////////// //|||||||||||||||
Martin Schultz, DEAS, Harvard University, 29 Oxford St., Pierce 109,
Cambridge, MA 02138 phone (617) 496 8318 fax (617) 495 4551
e-mail mgs@io.harvard.edu web http://www-as/people/staff/mgs/
|
|
|
|
|
|
| Re: Set Precision??????????? [message #15927 is a reply to message #15790] |
Thu, 17 June 1999 00:00 |
wbiagiot
Messages: 59
Registered: January 1999
|
Member |
|
|
Sure, if you are only after "daily" resolution, then a simple
> float is fine, but as soon as you want to extract seconds, you get
> terrible roundoff errors. And believe it or not: there are datasets in
> 1-second resolution (or finer) that are stored with the day of the
year.
Martin,
This is true. I only took the original message at face value. As for
the Y2K problem, if the year was numerically stored, even minimally, as
a byte, you would still have at least 256 instances. The problem was not
one of resolution, but representation, I believe. ASCII characters.
BTW, for those Y2K trivia buffs with Sun workstations, I read somewhere
that there's a Y-203x problem coming. Something about the internal
counter (32 bits?) reaching max count in the year 203x. Also, the
current problem wasn't an original one. Back in the sixties, one
character was allocated for the year.
Cheers,
Bill B.
Sent via Deja.com http://www.deja.com/
Share what you know. Learn what you don't.
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
