comp.lang.idl-pvwave archive - RDF feed
type: channel
description: Messages from Usenet group comp.lang.idl-pvwave, compiled by Paulo Penteado
items: Re: How to expand large arrays?

Re: How to expand large arrays?
type: item
description: In article <3n142g$j4k@aplinfo.jhuapl.edu>, art.croucher@jhuapl.edu writes: > I am trying to read in a large but unknown amount of data, and am > getting (IDL 3.6.1 VMS and Windows) 'unable to allocate core' crashes > long before I should be out of memory. Is there something that causes > IDL to allocate something like 5x the size of the array being used? I > understand that a temporary array is created, but I thought this would > limit me to an array 1/2 the size of the available memory. > I create a large array and expand it by either concatenation or by > creation of a new array and pasting. In both instances, I crash when > using 1/5 of my memory quota and less than half the physical memory > available (according to IDL help,/memory. The respective operating > systems say I'm at the limit of both physical and virtual). Yes, I > created enough huge images to use my expected quota - got just the > numbers I expected. > Can anybody tell me what is going on, and how can I create an array > that is large but nowhere near the quota? > The two methods I tried were: > temp=(~10000,2) > data=[temporary(data),temp] > and > temp=fltarr(npts+10000,2) > temp(0,0)=data > data=temp > temp=0 > Both crashed when the IDL-reported memory usage was 9MB of my 50 MB VMS > quota (VMS said I was using all 50MB). I think this is discussed in the FAQ, (and if not it should be!). The problem is memory fragmentation. When you do: IDL> data=[temporary(data),temp] you release the memory which IDL was using for "data". However, IDL now needs to allocate a larger CONTIGUOUS chunk of memory for the new larger version of "data". It can't re-use any of the memory you just released for this purpose, because it won't be part of a contiguous free memory region. The problem just continues as your array grows bigger and bigger, all of the previously freed memory is unusable for your growing array. It will be re-used whenever possible for future smaller arrays. Each time IDL calls "malloc" the operatin…
creator: rivers
date: 1995-04-19T07:00:00-00:00