| Re: memory allocation for structure arrays [message #11583] |
Thu, 30 April 1998 00:00 |
Peter Mason
Messages: 145
Registered: June 1996
|
Senior Member |
|
|
On Wed, 29 Apr 1998, Ian Sprod wrote:
> I am trying to read a pretty large data file (40Mb) into IDL. The file
> is 925,801 records, each 44 bytes long. I can describe the 44 bytes as a
> data structure and then replicate this to make a structure array (albeit
> a very large one). Then, in theory, reading in the data is a breeze.
>
> The problem is that IDL runs out of memory trying to read in the file.
> It seems that each line of the structure array is somehow requiring MORE
> than 44 bytes of memory. Poking around with top and free shows that it
> seems to be using ~312 bytes for each line instead. At this rate I can
> only read in the first ~225,000 lines of the file.
>
> Does anyone know exactly how IDL allocates memory for structures?
>
> Should I be using an associative array to do this?
>
> I am running IDL 5.0.2 on a Linux box with 128Mb of physical RAM and
> twice that of swap space.
Are you doing an ASCII (READF) or binary (READU) read? If ASCII then it
might be some sort of buffer problem. Try using OPEN,BUFSIZE=0, ...
(Really assuming a binary read from here on...)
Do you get the error during the READU call? (i.e., Not during the REPLICATE
call?) If so then there's something very odd going on here.
Can you read the first few (say 10) records correctly? If so, then I'd say
that there might be a bug in IDL on Linux.
Type: HELP,my_struc,/STRUC to see how much actual memory is used by each
instance of your struc. (My_struc being either your array or your original
struc def.) Note that this includes IDL's padding bytes, and will most
probably be a little more than 44 bytes (this is OK). If it's a *lot* more
than 44 bytes then you have a problem in your struc def.
If your structure contains no strings then I would have expected the I/O to
work with READU. With READU and WRITEU, padding bytes are handled
transparently by IDL, and not stored on disk.
If it contains strings then (assuming the file was not generated by IDL) they
will have a fixed amount of space in the original structure. The easiest way
to accomodate this in IDL is to use BYTARRs of the required size instead of
strings in your structure definition. (You can get the strings later when
you need them - STRING(a_bytarr) will find and use any NULL-termination in
a_bytarr.) String structure-members in IDL are dynamically sized and it
is a nightmare to do binary I/O with them.
Do not use ASSOC to try to read the file unless you know that it was written
by an IDL program using ASSOC. I believe that ASSOC with structures does not
have the intelligence of READU/WRITEU - it reads and writes the padding bytes
from/to disk and so is incompatible with READU and WRITEU (and essentially
any external program that does not know exactly how IDL has padded the
structure on your particular platform).
Not very conclusive, but I hope it helped a bit.
Peter Mason
|
|
|
|
| Re: memory allocation for structure arrays [message #11584 is a reply to message #11583] |
Thu, 30 April 1998 00:00  |
korpela
Messages: 59
Registered: September 1993
|
Member |
|
|
In article <3547B2B8.5220E8F6@ngdc.noaa.gov>,
Ian Sprod <ian@ngdc.noaa.gov> wrote:
>
> Does anyone know exactly how IDL allocates memory for structures?
>
I'm not sure exactly how it does, but it should...
1) create a structure definition
2) allocate a block of memory for the structure data
3) assign the data block and structure definition to to the variable.
> Should I be using an associative array to do this?
Probably, but I hate ASSOC. I've got my own memory mapped files
IDL routine that I use for large files. It doesn't yet work
with structures, but works fine with normal arrays. I've never
tried it on linux, but it should work with very little modification.
(The only modification that probably need to be made is that some of
the include files reside in a different location on Linux systems vs
SunOS system). The program and instructions for use reside at
http://sag-www.ssl.berkeley.edu/~korpela/mmap/
Eric
--
Eric Korpela | An object at rest can never be
korpela@ssl.berkeley.edu | stopped.
<a href="http://sag-www.ssl.berkeley.edu/~korpela">Click for home page.</a>
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
