| Re: reading in binary data [message #17144] |
Thu, 16 September 1999 00:00 |
wbiagiot
Messages: 59
Registered: January 1999
|
Member |
|
|
In article <MPG.1249d49594627d649898f4@news.frii.com>,
davidf@dfanning.com (David Fanning) wrote:
>
> P.S. Do I remember reading somewhere that IDL 5.3 has
> implemented some kind of file compression scheme into
> its SAVE files? Can't put my hands on it now, but it
> sure sticks in my head somehow.
>
Just for the record: from my own efforts on creating IDL application
install floppies, my .SAV files achieve roughly a 7-to-1 reduction in
size when zipped. In fact, reading the characters in a .SAV file will
reveal that a vast majority of the bytes are null!
-Bill B.
--
"They don't think it be like it is, but it do."
Oscar Gamble, NY Yankees
Sent via Deja.com http://www.deja.com/
Share what you know. Learn what you don't.
|
|
|
|
| Re: reading in binary data [message #17149 is a reply to message #17144] |
Wed, 15 September 1999 00:00  |
davidf
Messages: 2866
Registered: September 1996
|
Senior Member |
|
|
Brian Nagy (lazzar@gte.net) writes:
> I'm trying to read in a binary data file that is broken down into
> various tuples (these are blocks of different data within the file).
> Each tuple starts with a specific header byte to identify it, which is
> proceeded by a byte that can be used to check if the data in that tuple
> is the correct type. The way I currently read in these files is to read
> in an entire file then using a WHILE loop check each byte for a given
> tuple type, if yes then read in the tuple and store it, if no check for
> the next tuple type. On the surface this seems to work with no
> problems, but it is very slow and limited based on file size. Some of
> the files I'm trying to read in create arrays that are 1000x1500 and
> that is a relatively small size. I would like to find a faster way to
> read this data, as well as find a way not to bog IDL down with hugh
> arrays. Is it possible to access a binary file without reading it all
> in at once?
The standard method for reading a portion of a binary
file is to use the Associated Variable method to read
just that portion of the file desired. This is almost
like a random access method, where file pointers are
used to position you at the correct location in the file.
I can't tell from your description of the file if this
file lends itself to this method, but I think it is likely.
> Also is it possible to write to an IDL save file
> incrementally, so that I can unload some of the array to disk and free
> up the extra memory?
No.
> Along the same line is it possible to read in only
> a portion of an IDL save file if, for example, I only want certain
> variables from it but not all of the variables?
No.
> Another issue of note is the size of the IDL save files. When I convert
> one of my binary files to an IDL save file it increases the size of that
> file by about 3 times (a 953k binary file equals an equivalent IDL save
> file of 3.02meg). Is there any way to reduce the size of the IDL save
> file (I already remove any zeros from the array by indexing it and store
> the index and values in separate variables), or to save in a different
> format that is better on compression but still is quick to read?
This surprises me. There is obviously some overhead in
creating an IDL save file, but this seems a bit excessive.
Usually in these situations I always blame the programmer.
I'm wrong about 0.05 percent of the time. :-)
And, anyway, why are you so enamored with IDL SAVE files?
I use them occasionally, of course, but I'm scratching my
head to think of the need for them here. What exactly are
you trying to do?
Cheers,
David
P.S. Do I remember reading somewhere that IDL 5.3 has
implemented some kind of file compression scheme into
its SAVE files? Can't put my hands on it now, but it
sure sticks in my head somehow.
--
David Fanning, Ph.D.
Fanning Software Consulting
Phone: 970-221-0438 E-Mail: davidf@dfanning.com
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Toll-Free IDL Book Orders: 1-888-461-0155
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
