| Re: error reading a large number of binary files [message #53697 is a reply to message #53600] |
Wed, 25 April 2007 07:21   |
![Paul Van Delst[1] is currently offline](theme/default/images/xoffline.png.pagespeed.ic.XRkd1fkXye.png "Paul Van Delst[1] is currently offline")
Paul Van Delst[1]
Messages: 1157
Registered: April 2002
|
Senior Member |
|
|
mark@atmos.colostate.edu wrote:
> ** UPDATE on the status of my problem **
>
> I decided to reprocess the original grib files with wgrib again but
> this time with the -nh switch turned on, which eliminates the f77-
> style header record. Then in the IDL read routine I just had to turn
> off the /f77_unformatted switch. Well, this has worked flawlessly in
> the three tests I have tried!!!
>
> That made me wonder if my problem was with IDL reading ANY fortran-
> style binary files, or just the ones created by wgrib. So I wrote a
> fortran-90 program to generate roughly the same size and number of
> binary files but just writing random numbers to the files, and then a
> simple readbin.pro IDL routine. Sure enough, it crashes in random
> places just like my previous set of ECWMF binary files with f77
> headers did. Here's some sample output from a few trials -- these
> random number files are just named x001.bin, x002.bin, ..., x520.bin.
>
> test #1
>>>> x062.bin
>>>> x063.bin
>>>> x064.bin
> % READU: Corrupted f77 unformatted file detected. Unit: 100, File:
> x064.bin
> % Execution halted at: READBIN2 10 /Users/mark/Datasets/
> era40/testbin/readbin2.pro
> % $MAIN$
>
> test #2
>>>> x300.bin
>>>> x301.bin
>>>> x302.bin
>>>> x303.bin
> % READU: Corrupted f77 unformatted file detected. Unit: 100, File:
> x303.bin
> % Execution halted at: READBIN2 10 /Users/mark/Datasets/
> era40/testbin/readbin2.pro
> % $MAIN$
>
> AND the problem seems to only occur once I make the files very large.
> i.e., when i just wrote out 520 files of arrays of size 5, it flies
> through all of those with no problems.
Interesting. Exactly how large do the individual files need to be for this problem to surface?
cheers,
paulv
> I'd really love to get to the bottom of this problem, and I should
> mention that one of the ITTVIS consultants has indeed been emailing
> with me and has been extremely helpful. But in the meantime I will
> obviously proceed with the "no-header" files for this project.
>
> Thanks to all of you who offered suggestions -- it's very helpful and
> encouraging!
>
> Cheers,
> Mark
>
--
Paul van Delst Ride lots.
CIMSS @ NOAA/NCEP/EMC Eddy Merckx
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
