| reading in binary data [message #17158] |
Wed, 15 September 1999 00:00  |
Lazzar
Messages: 9
Registered: February 1999
|
Junior Member |
|
|
I have problem that maybe someone out there can help with,
I'm trying to read in a binary data file that is broken down into
various tuples (these are blocks of different data within the file).
Each tuple starts with a specific header byte to identify it, which is
proceeded by a byte that can be used to check if the data in that tuple
is the correct type. The way I currently read in these files is to read
in an entire file then using a WHILE loop check each byte for a given
tuple type, if yes then read in the tuple and store it, if no check for
the next tuple type. On the surface this seems to work with no
problems, but it is very slow and limited based on file size. Some of
the files I'm trying to read in create arrays that are 1000x1500 and
that is a relatively small size. I would like to find a faster way to
read this data, as well as find a way not to bog IDL down with hugh
arrays. Is it possible to access a binary file without reading it all
in at once? Also is it possible to write to an IDL save file
incrementally, so that I can unload some of the array to disk and free
up the extra memory? Along the same line is it possible to read in only
a portion of an IDL save file if, for example, I only want certain
variables from it but not all of the variables?
Another issue of note is the size of the IDL save files. When I convert
one of my binary files to an IDL save file it increases the size of that
file by about 3 times (a 953k binary file equals an equivalent IDL save
file of 3.02meg). Is there any way to reduce the size of the IDL save
file (I already remove any zeros from the array by indexing it and store
the index and values in separate variables), or to save in a different
format that is better on compression but still is quick to read?
Thanks for any help you can provide,
Brian
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
