Is there any particular reason why IDL would struggle reading a really
large number of binary files? I'm trying to compute a number of
statistics from the ECWMF ERA40 reanalysis 4*daily data, which is a
HUGE amount of data. The data was originally in Grib format, which I
converted to ieee binary with the wgrib program. Each file contains a
month's worth of data, which has 23 pressure levels. In trying to
track down the cause of the error I'm getting, I've pared the program
down to what you can see below (just compute the long-term average
zonal wind). FWIW, I'm running this on a Mac PPC Quad G5 with 8GB of
RAM, and it's IDL 6.3.
What happens is that I get an error message like this:
% READU: Corrupted f77 unformatted file detected. Unit: 100, File:
daily_4/u196710.bin
% Execution halted at: READUDAILY 19 /Users/mark/Datasets/
era40/readudaily.pro
% $MAIN$
And on repeated trials, the error always occurs in a different file.
And I know that none of the files are actually corrupted.
Interestingly, I wrote a similar program in fortran and using the
NAGWare compiler it has similar problems, but it worked successfully
using the xlf fortran compiler. However, I'd prefer to have it work
in IDL, but I cannot for the life of me figure out what I'm doing
wrong.
Any ideas?
Thanks in advance,
Mark Branson
CSU Dept. of Atmospheric Science
pro readudaily
; read 4*daily ECMWF ERA-40 reanalysis wind data
x = fltarr(144,73)
ubar = fltarr(144,73,23)
infiles = FILE_SEARCH('daily_4/u*.bin')
icnt = 0L & totcnt = 0L
mm = 9 ; first file is sept 1957
for i = 0,n_elements(infiles)-1 do begin
openr, lun, infiles[i], /get_lun, /binary, /f77_unformatted
print, infiles[i]
iday = 1 & ihr = 0
while not EOF(lun) do begin
for ilev = 0,22 do begin
readu, lun, x
ubar[*,*,ilev] = ubar[*,*,ilev] + x[*,*]
endfor
icnt = icnt+1
endwhile
free_lun, lun
case 1 of
(mm eq 2) : begin
if icnt NE 112 then $
print, '>>> file = ',infiles[i],' icnt = ',icnt
end
(mm eq 1 or mm eq 3 or mm eq 5 or mm eq 7 or mm eq 8 or mm eq 10
or mm eq 12) : begin
if icnt NE 124 then $
print, '>>> file = ',infiles[i],' icnt = ',icnt
end
(mm eq 2 or mm eq 4 or mm eq 6 or mm eq 9 or mm eq 11) : begin
if icnt NE 120 then $
print, '>>> file = ',infiles[i],' icnt = ',icnt
end
endcase
if mm EQ 12 then mm = 1 else mm = mm+1
totcnt = totcnt + icnt
icnt = 0L
endfor
print, '>>> totcnt = ',totcnt
; compute mean
for ilev = 0,22 do $
ubar[*,*,ilev] = ubar[*,*,ilev]/float(totcnt)
openw, lun, 'ubar-annual-idl.data', /get_lun
writeu, lun, ubar
free_lun, lun
end
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
