comp.lang.idl-pvwave archive: archive » bizarre number transformation

Home » Public Forums » archive » bizarre number transformation

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: bizarre number transformation [message #31659 is a reply to message #31489]

Fri, 26 July 2002 10:14

mperrin+news
Messages: 81
Registered: May 2001

Member

Paul van Delst <paul.vandelst@noaa.gov> wrote:
> James Kuyper wrote:
>> Precisely. I also do scientific programming, where such needs are pretty
>> common. But the vast bulk of the world's programming involves numbers
>> that can be represented with adequate accuracy using single precision
>> floating point. For instance, how many million-dollar quantities are
>> actually measured with a precision of +/-$1?
>
> I would hope the answer is not any at all. I would think that 10th's of a US dollar/Euro
> (or even 100th's) would be used. Exchange rates are usually quoted to 1000th's of a US
> Dollar. This is all a hopeful thought on my part since my past and forecast number of
> million dollar transactions is, unfortunately, zero.

Some brief experimentation with Microsoft Excel (I know, not the world's
paragon of financial software, but it's what I have on hand.) seem to indicate
that it uses doubles for storing constants entered into cells. It happily
handles up to 15 decimal digits without trouble and truncates digits entered
after that (not rounds). This allows you to handle up to ten trillion dollars
retaining down to the penny - good enough for most purposes indeed.

Quicken on the other hand restricts amounts entered to
|n| < 9,999,999.99
and doesn't allow precision greater than 0.01 to be entered at all.
This is either a limitation imposed by the UI which doesn't reflect
the limitations of their internal representation, or they're using
fixed-width BCD in units of one cent - which would strike me as the
Correct Thing to do in this case.

My gut instinct is that most financial software out there probably uses
either doubles or BCD. Roundoff errors could easily become a major
financial problem if you're Visa.

- Marshall

Report message to a moderator

[Message index]

		bizarre number transformation By: merlecorp on Thu, 25 July 2002 04:46
		Re: bizarre number transformation By: R.Bauer on Mon, 29 July 2002 02:04
		Re: bizarre number transformation By: David Fanning on Fri, 26 July 2002 10:36
		Re: bizarre number transformation By: David Fanning on Fri, 26 July 2002 10:33
		Re: bizarre number transformation By: mperrin+news on Fri, 26 July 2002 10:14
		Re: bizarre number transformation By: Paul Van Delst[1] on Fri, 26 July 2002 07:39

Previous Topic:	compare ascii files, EOF
Next Topic:	Re: compare ascii files, EOF

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Thu Dec 04 14:30:00 PST 2025

Total time taken to generate the page: 1.83663 seconds