| Re: Fragmented memory with IDL [message #1044] |
Tue, 18 May 1993 17:47  |
thompson
Messages: 584
Registered: August 1991
|
Senior Member |
|
|
mayor@vaxine.larc.nasa.gov writes:
> IDLers,
> The following is a question that I sent to IDL tech support today.
> Below it is the reply from IDL. Thought I'd post it to see if
> anybody else had similar problems or solutions.
>> We've noticed that occasionally we get the following error message:
>>
>> % Unable to allocate memory: to make array.
>> not enough core
>>
>> The interesting part is that it never occurs when we run the program
>> immediately after starting IDL. It does happen after running the
>> program multiple times EVEN with a RETALL at the end of each run.
>> Somehow, it appears that consecutive runs cause it to reserve
>> more and more memory - even though each run shouldn't require any
>> more than the previous runs in the same IDL session. Is there a
>> command that I can use to assure that all memory is cleared out
>> even after a retall?
> Alan Youngblood of RSI replies:
> The circumstances you describe happen when memory becomes fragmented.
> Unfortunately, there is nothing you can do except use less memory in
> your application, or get more for the system to work with.
Yes, this is a well known problem. Memory gets fragmented just like disks do.
Suppose you allocate two megabytes for a temporary 1024x1024 integer array.
Then you release this memory and create several smaller arrays. Chances are
these new arrays will fall into some of the space used by the original large
array. Now you want to allocate another 2 megabytes temporarily. You can't
use the memory you used before because part of it is taken up. Worse, an array
has to be stored contiguously, so any free memory that occurs between used
sections can't be used to store this temporary array.
There was a long discussion of memory fragmentation in this newsgroup some time
ago. It's probably worth putting some of the net wisdom on this issue in the
new FAQ.
One particularly common cause of memory fragmentation is when large arrays are
allowed to grow by concatenation. For instance, suppose you were reading a
large data file into a array with something like the following:
DATA = FLTARR(10)
READF,UNIT,DATA
ACCUMULATED = DATA
WHILE NOT EOF(UNIT) DO BEGIN
READF,UNIT,DATA
ACCUMULATED = [ACCUMULATED,DATA]
ENDWHILE
then every step through this loop would require that a new array be created. A
better way to accomplish the same thing would be
ACCUMULATED = FLTARR(1000)
DATA = FLTARR(10)
N_DATA = 0
WHILE NOT EOF(UNIT) DO BEGIN
IF N_DATA EQ N_ELEMENTS(ACCUMULATED) THEN $
ACCUMULATED = [ACCUMULATED,FLTARR(1000)]
READF,UNIT,DATA
ACCUMULATED(N_DATA) = DATA
N_DATA = N_DATA + 10
ENDWHILE
ACCUMULATED = ACCUMULATED(0:N_DATA)
Letting the array grow by fewer but bigger jumps will help minimize the effects
of memory fragmentation.
Another way to avoid memory fragmentation is to use the TEMPORARY function
where appropriate.
Hope this helps. I'm sure other people also have insights into memory
fragmentation to share with the net.
Bill Thompson.
|
|
|
|
| Re: Fragmented memory with IDL [message #1045 is a reply to message #1044] |
Tue, 18 May 1993 18:58  |
deutsch
Messages: 19
Registered: February 1992
|
Junior Member |
|
|
In article <1tbs6sINN6eh@rave.larc.nasa.gov>, mayor@vaxine.larc.nasa.gov writes:
> IDLers,
>
>> We've noticed that occasionally we get the following error message:
>>
>> % Unable to allocate memory: to make array.
>> not enough core
>>
>> The interesting part is that it never occurs when we run the program
>> immediately after starting IDL. It does happen after running the
>> program multiple times EVEN with a RETALL at the end of each run.
>> Somehow, it appears that consecutive runs cause it to reserve
>> more and more memory - even though each run shouldn't require any
>> more than the previous runs in the same IDL session. Is there a
>> command that I can use to assure that all memory is cleared out
>> even after a retall?
>
> Alan Youngblood of RSI replies:
>
> The circumstances you describe happen when memory becomes fragmented.
> Unfortunately, there is nothing you can do except use less memory in
> your application, or get more for the system to work with.
>
This is, in a nutshell, correct. I believe that it is a problem of IDL
reserving memory from the OS (in this case VMS, I assume) in lots of little
chucks and at no point can any sort of garbage collection of memory
compaction really occur. In short, after allocating and freeing memory for
lots of arrays (especially large ones) the memory becomes "fragmented" such
that there is no contiguous memory chunk availble. Besides doing what Alan
suggests (look into the temporary() function if you're not using it) you
might try:
IDL> tmp=intarr(2048,3000,/nozero)
IDL> delvar,tmp
right after entering IDL. Change the 2048 and 3000 dimension sizes to be
about as big as your page file quota will allow. In theory, this gives IDL
one huge chunk of memory to work with and should allow it to manage variables
better than if it got them in little chunks. Note that if you have a huge
page file quota, you may only want to reserve roughly the amount of memory
you'll be using.
No guarantees that this will solve or help your problems, but do try it; it may.
Eric
Eric Deutsch
University of Washington
Department of Astronomy FM-20
Seattle, WA 98195
deutsch@astro.washington.edu or UWSPAN::JANUS::DEUTSCH
>
>
> ============================================================ ====================
> Shane D. Mayor, Lidar Applications Group, NASA Langley Research Center,
> Mail Stop 401A, Hampton, Virginia 23681-0001
> Internet: MAYOR@VAXINE.LARC.NASA.GOV Phone: 804-864-7598 Fax: 804-864-7790
> ============================================================ ====================
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
