| Re: error reading a large number of binary files [message #53727 is a reply to message #53600] |
Mon, 23 April 2007 10:16   |
Mark Branson
Messages: 9
Registered: May 2006
|
Junior Member |
|
|
On Apr 23, 10:03 am, Paul van Delst <Paul.vanDe...@noaa.gov> wrote:
> Hi,
>
> Hmm. Is there *no* pattern as to which file (and where in the file) the failure occurs?
>
> Is there a way to rewrite your IDL code that completely separates the grib read from what
> follows? E.g.
>
> begin file loop...
>
> call read function....
>
> do other stuff....
>
> end file loop
>
> Maybe with the file reader completely separated from the body of your code you can test it
> differently.
>
> What happens if you *only* read the file data, but don't do anything with it? E.g.:
>
> pro readudaily
> ; Just because.....
> CATCH, Error_Status
> IF ( Error_Status NE 0 ) THEN BEGIN
> CATCH, /CANCEL
> MESSAGE, !ERROR_STATE.MSG, /CONTINUE
> RETURN
> ENDIF
>
> ; read 4*daily ECMWF ERA-40 reanalysis wind data
> x = fltarr(144,73)
> infiles = FILE_SEARCH('daily_4/u*.bin')
> icnt = 0L & totcnt = 0L
>
> for i = 0,n_elements(infiles)-1 do begin
>
> ; Read the file
> openr, lun, infiles[i], /get_lun, /f77_unformatted
> print, infiles[i]
> while not EOF(lun) do begin
> for ilev = 0,22 do begin
> readu, lun, x
> endfor
> icnt = icnt+1
> endwhile
> free_lun, lun
>
> ; Increment counters
> totcnt = totcnt + icnt
> icnt = 0L
>
> endfor
> print, '>>> totcnt = ',totcnt
>
> end
>
> ??
>
> Do you still get failures?
>
> cheers,
>
> paulv
>
Hi Paul. Thanks for taking an interest in this. On Friday I actually
tried something similar to your idea of just reading the files and not
doing any computations, and that also failed. And just now I took the
program you created above, saved it as readudaily2.pro and tried it
three times. Here are the results:
attempt #1:
daily_4/u196112.bin
daily_4/u196201.bin
daily_4/u196202.bin
% READUDAILY2: READU: Corrupted f77 unformatted file detected. Unit:
100, File: daily_4/u196202.bin
attempt #2:
daily_4/u198409.bin
daily_4/u198410.bin
daily_4/u198411.bin
% READUDAILY2: READU: Corrupted f77 unformatted file detected. Unit:
100, File: daily_4/u198411.bin
attempt #3:
daily_4/u199610.bin
daily_4/u199611.bin
% READUDAILY2: READU: Corrupted f77 unformatted file detected. Unit:
100, File: daily_4/u199611.bin
As you can see, it fails on a different file each time. Although
since it seemed to get further along on each successive trial, maybe I
should try one or two more times and perhaps I'll get lucky??? :)
Actually, I think this is just random and not a trend.
I wonder if the same error would occur if I generated a large number
of binary files of a smaller size (say a 5x5 array of random numbers),
because maybe it's not how large the files are but how many of them
there are. ?????
Thanks,
Mark
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
