comp.lang.idl-pvwave archive - RDF feed
type: channel
description: Messages from Usenet group comp.lang.idl-pvwave, compiled by Paulo Penteado
items: Problems with double precision in IDL

Problems with double precision in IDL
type: item
description: We have a potentially VERY serious problem with the COBE data analysis because of the way IDL seems to be (mis?)handling double precision numbers. Here is an example of how IDL treats floating point numbers when converting them to double precision. The operations were performed on a DECstation. z=.32 print,f2,double(z) ; If a variable is declared DOUBLE this is what happens. 0.319999992847 print,f2,.32d ; If the "d" notation is used instead it's accurate. 0.320000000000 print,double(z)-.32D -7.15255737e-09 print,[double(z)-.32D]/.32D -2.23517418e-08 An illustration of the difference between FORTRAN and IDL follows. Two parallel programs were written, one in each language, which do the following: 1. Read variables declared as single precision from a file (typically 2-5 numbers). 2. Convert the variables to double precision 3. Perform various computations on the double precision numbers (the computations involved only a few addition/subtraction operations and some divison). 4. Convert the answers to single precision and write to a file. The difference in the answers shows that IDL only gives accuracy to four significant digits (or less) under these circumstances. That's really not acceptable for many applications. COBE Variable Name FORTRAN IDL .coad_spec_data.XCAL_SIGMA 8.096988e-02 8.096990e-02 .coad_spec_data.ICAL_SIGMA 7.917210e-05 7.912043e-05 .coad_spec_data.SKYHORN_SIGMA 4.511252e-04 4.511047e-04 .coad_spec_data.REFHORN_SIGMA 6.387254e-05 6.390238e-05 .coad_spec_data.DIHEDRAL_SIGMA 9.627582e-02 9.627583e-02 .coad_spec_data.COLLIMATOR_SIGMA 1.346936e-02 1.346943e-02 .coad_spec_data.BOL_ASSEM_SIGMA 4.545850e-03 4.545857e-03 .coad_spec_data.BOL_ASSEM_SIGMA 4.577306e-03 4.577292e-03 .coad_spec_data.BOL_ASSEM_SIGMA 4.641583e-03 4.641529e-03 .coad_spec_data.BOL_ASSEM_SIGMA 6.197933e-03 6.197947e-03 .spec_data.RESP_SIGMA 3.442960e-03 3.442946e-03 .spec_data.TC_SIGMA 1.057141e-06 1.057107e-06 .spec_data.PC_SIGMA 8.645144e-05 8.645145e-05 .spec_data.QRAD_SIGMA 8.201977e…
creator: isaacman
date: 1993-06-16T16:12:00-00:00