| Re: error reading a large number of binary files [message #53599] |
Fri, 20 April 2007 16:25  |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
mark@atmos.colostate.edu writes:
> And on repeated trials, the error always occurs in a different file.
> And I know that none of the files are actually corrupted.
> Interestingly, I wrote a similar program in fortran and using the
> NAGWare compiler it has similar problems, but it worked successfully
> using the xlf fortran compiler. However, I'd prefer to have it work
> in IDL, but I cannot for the life of me figure out what I'm doing
> wrong.
From your description, I would say the chances are MUCH
better than they usually are that this is NOT operator
error. :-)
But this might be the wrong place to look for an answer.
I think I would bump this one up to ITTVIS.
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Sepore ma de ni thui. ("Perhaps thou speakest truth.")
|
|
|
|
| Re: error reading a large number of binary files [message #53705 is a reply to message #53599] |
Tue, 24 April 2007 14:11  |
Mark Branson
Messages: 9
Registered: May 2006
|
Junior Member |
|
|
** UPDATE on the status of my problem **
I decided to reprocess the original grib files with wgrib again but
this time with the -nh switch turned on, which eliminates the f77-
style header record. Then in the IDL read routine I just had to turn
off the /f77_unformatted switch. Well, this has worked flawlessly in
the three tests I have tried!!!
That made me wonder if my problem was with IDL reading ANY fortran-
style binary files, or just the ones created by wgrib. So I wrote a
fortran-90 program to generate roughly the same size and number of
binary files but just writing random numbers to the files, and then a
simple readbin.pro IDL routine. Sure enough, it crashes in random
places just like my previous set of ECWMF binary files with f77
headers did. Here's some sample output from a few trials -- these
random number files are just named x001.bin, x002.bin, ..., x520.bin.
test #1
>>> x062.bin
>>> x063.bin
>>> x064.bin
% READU: Corrupted f77 unformatted file detected. Unit: 100, File:
x064.bin
% Execution halted at: READBIN2 10 /Users/mark/Datasets/
era40/testbin/readbin2.pro
% $MAIN$
test #2
>>> x300.bin
>>> x301.bin
>>> x302.bin
>>> x303.bin
% READU: Corrupted f77 unformatted file detected. Unit: 100, File:
x303.bin
% Execution halted at: READBIN2 10 /Users/mark/Datasets/
era40/testbin/readbin2.pro
% $MAIN$
AND the problem seems to only occur once I make the files very large.
i.e., when i just wrote out 520 files of arrays of size 5, it flies
through all of those with no problems.
I'd really love to get to the bottom of this problem, and I should
mention that one of the ITTVIS consultants has indeed been emailing
with me and has been extremely helpful. But in the meantime I will
obviously proceed with the "no-header" files for this project.
Thanks to all of you who offered suggestions -- it's very helpful and
encouraging!
Cheers,
Mark
|
|
|
|
| Re: error reading a large number of binary files [message #53734 is a reply to message #53599] |
Mon, 23 April 2007 07:45 |
Mark Branson
Messages: 9
Registered: May 2006
|
Junior Member |
|
|
On Apr 20, 5:25 pm, David Fanning <n...@dfanning.com> wrote:
> m...@atmos.colostate.edu writes:
>> And on repeated trials, the error always occurs in a different file.
>> And I know that none of the files are actually corrupted.
>> Interestingly, I wrote a similar program in fortran and using the
>> NAGWare compiler it has similar problems, but it worked successfully
>> using the xlf fortran compiler. However, I'd prefer to have it work
>> in IDL, but I cannot for the life of me figure out what I'm doing
>> wrong.
>
> From your description, I would say the chances are MUCH
> better than they usually are that this is NOT operator
> error. :-)
>
> But this might be the wrong place to look for an answer.
> I think I would bump this one up to ITTVIS.
>
> Cheers,
>
> David
> --
> David Fanning, Ph.D.
> Fanning Software Consulting, Inc.
> Coyote's Guide to IDL Programming:http://www.dfanning.com/
> Sepore ma de ni thui. ("Perhaps thou speakest truth.")
Thanks, David. Good suggestion. I sent a report to ITTVIS over the
weekend. I'm now taking bets as to if/when I actually hear back from
them... :)
BTW, I have you to blame for my lack of sleep lately. I recently
bought the first three Anna Pigeon novels based on your
recommendation, and you're right -- they're just about impossible to
put down!
Cheers,
Mark
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
