| Re: about memory [message #25138] |
Tue, 22 May 2001 09:21 |
Craig Markwardt
Messages: 1869
Registered: November 1996
|
Senior Member |
|
|
davidf@dfanning.com (David Fanning) writes:
> DELVAR can only be used at the IDL main level. Another
> alternative is to use something like my UNDEFINE program,
> which can be used anywhere and has the effect of undefining
> a program variable.
>
> http://www.dfanning.com/programs/undefine.pro
I use UNDEFINE all the time. It's great. When I'm writing an
exportable program, where I'm not sure that UNDEFINE will be
available, then I use the essence of the algorithm:
a = 0 & dummy = temporary(a)
Then A is undefined and the memory is unallocated.
Craig
--
------------------------------------------------------------ --------------
Craig B. Markwardt, Ph.D. EMAIL: craigmnet@cow.physics.wisc.edu
Astrophysics, IDL, Finance, Derivatives | Remove "net" for better response
------------------------------------------------------------ --------------
|
|
|
|
|
|
| Re: about memory [message #25145 is a reply to message #25143] |
Tue, 22 May 2001 07:30  |
Med Bennett
Messages: 109
Registered: April 1997
|
Senior Member |
|
|
Craig Markwardt wrote:
> Amar Nayegandhi <anayegan@csee.usf.edu> writes:
>
>> This is a general question about memory management.
>> can i free memory allocated to an array if I have finished using it in
>> the program? suppose
>> arr_a = fltarr(10000,10000)
>> if i don't need array arr_a anymore, would
>> arr_a = 0
>> free the memory used by arr_a? I would expect it to now utilize only
>> 2(or 4) bytes of memory.
>
> The answer to the question is "yes," but also "it depends." The
> answer is "yes" because the memory is indeed made available again to
> the same IDL process. I use
>
> A = 0
>
> every day when I am done with a variable. It's totally legit. Hong
> may need go no further than this, although I agree that he's dealing
> with extremely large arrays which might be better suited with a tiled
> or chunked approach.
>
> The answer is also "it depends," because it depends on which platform
> you are using. I believe that under Windows the memory is actually
> returned to the OS. Thus, other programs are able to use the memory
> again. Under Unix this is much less likely, so once the memory is
> allocated to one session of IDL, it stays there until the session ends
> (and is not available to another process). [ I think this is not
> *always* true, but mostly true. ]
>
> Every so often somebody asks, "why is IDL eating all my memory?" It's
> basically unavoidable. Sorry for this diversion.
>
> Craig
>
> --
> ------------------------------------------------------------ --------------
> Craig B. Markwardt, Ph.D. EMAIL: craigmnet@cow.physics.wisc.edu
> Astrophysics, IDL, Finance, Derivatives | Remove "net" for better response
> ------------------------------------------------------------ --------------
Just curious - why not 'delvar,a'? Is this not as effective (or more so) than
'a=0'?
|
|
|
|
| Re: about memory [message #25147 is a reply to message #25145] |
Mon, 21 May 2001 17:08 |
Craig Markwardt
Messages: 1869
Registered: November 1996
|
Senior Member |
|
|
Amar Nayegandhi <anayegan@csee.usf.edu> writes:
> This is a general question about memory management.
> can i free memory allocated to an array if I have finished using it in
> the program? suppose
> arr_a = fltarr(10000,10000)
> if i don't need array arr_a anymore, would
> arr_a = 0
> free the memory used by arr_a? I would expect it to now utilize only
> 2(or 4) bytes of memory.
The answer to the question is "yes," but also "it depends." The
answer is "yes" because the memory is indeed made available again to
the same IDL process. I use
A = 0
every day when I am done with a variable. It's totally legit. Hong
may need go no further than this, although I agree that he's dealing
with extremely large arrays which might be better suited with a tiled
or chunked approach.
The answer is also "it depends," because it depends on which platform
you are using. I believe that under Windows the memory is actually
returned to the OS. Thus, other programs are able to use the memory
again. Under Unix this is much less likely, so once the memory is
allocated to one session of IDL, it stays there until the session ends
(and is not available to another process). [ I think this is not
*always* true, but mostly true. ]
Every so often somebody asks, "why is IDL eating all my memory?" It's
basically unavoidable. Sorry for this diversion.
Craig
--
------------------------------------------------------------ --------------
Craig B. Markwardt, Ph.D. EMAIL: craigmnet@cow.physics.wisc.edu
Astrophysics, IDL, Finance, Derivatives | Remove "net" for better response
------------------------------------------------------------ --------------
|
|
|
|
| Re: about memory [message #25148 is a reply to message #25147] |
Mon, 21 May 2001 15:38 |
Amar Nayegandhi
Messages: 15
Registered: May 2001
|
Junior Member |
|
|
This is a general question about memory management.
can i free memory allocated to an array if I have finished using it in
the program? suppose
arr_a = fltarr(10000,10000)
if i don't need array arr_a anymore, would
arr_a = 0
free the memory used by arr_a? I would expect it to now utilize only
2(or 4) bytes of memory.
this may be helpful to you, Hong. if you have multiple Tb arrays, and
are not using them anymore in the program, you can free some memory by
initializing Tb to 0(if it works!)
-amar
William Thompson wrote:
>
> Hong Gang <honggang@uni-bremen.de> writes:
>
>> Hallo all,
>
>> The following is the error given by my program.
>> % Unable to allocate memory: to make array.
>> Cannot allocate memory
>
>> I have some large arrays as Tb(632*664, 880). I do not know what is the
>> matter with the program. Can anyone give me some idea?
>
> IDL is telling you that the operating system is not allowing it to allocate
> enough memory to create the array. It's not surprising; a floating point array
> of that size would take well over a gigabyte of memory. Generally speaking,
> you'd need several times that much memory to manipulate such an array.
>
> You don't mention what operating system you're using. If Unix, then you can
> find out how much memory the operating system will allow you to have by
> entering in the command "limit" (before calling IDL). You can also type
> "limit -h" to find the hard limits. The relevant number is "datasize". You
> can increase this, e.g. "limit datasize 200000" would give you 200 megabytes of
> memory.
>
> Probably, you should rewrite your software to manipulate your data in pieces,
> instead of all at once. One way to do this is to write your data to a file,
> and then use the ASSOC() function to address parts of the file.
>
> William Thompson
--
*************************************************
Murphy's Law of Research:
Enough research will tend to support your theory.
*************************************************
Amar Nayegandhi
Graduate Student
Department of Computer Science,
University of South Florida, Tampa.
|
|
|
|
| Re: about memory [message #25149 is a reply to message #25148] |
Mon, 21 May 2001 15:01 |
thompson
Messages: 584
Registered: August 1991
|
Senior Member |
|
|
Hong Gang <honggang@uni-bremen.de> writes:
> Hallo all,
> The following is the error given by my program.
> % Unable to allocate memory: to make array.
> Cannot allocate memory
> I have some large arrays as Tb(632*664, 880). I do not know what is the
> matter with the program. Can anyone give me some idea?
IDL is telling you that the operating system is not allowing it to allocate
enough memory to create the array. It's not surprising; a floating point array
of that size would take well over a gigabyte of memory. Generally speaking,
you'd need several times that much memory to manipulate such an array.
You don't mention what operating system you're using. If Unix, then you can
find out how much memory the operating system will allow you to have by
entering in the command "limit" (before calling IDL). You can also type
"limit -h" to find the hard limits. The relevant number is "datasize". You
can increase this, e.g. "limit datasize 200000" would give you 200 megabytes of
memory.
Probably, you should rewrite your software to manipulate your data in pieces,
instead of all at once. One way to do this is to write your data to a file,
and then use the ASSOC() function to address parts of the file.
William Thompson
|
|
|
|
| Re: about memory [message #25150 is a reply to message #25149] |
Mon, 21 May 2001 14:33 |
Jaco van Gorkom
Messages: 97
Registered: November 2000
|
Member |
|
|
Hong Gang wrote:
> The following is the error given by my program.
> % Unable to allocate memory: to make array.
> Cannot allocate memory
>
> I have some large arrays as Tb(632*664, 880). I do not know what is the
> matter with the program. Can anyone give me some idea?
Usually this means that IDL has run out of memory. An array of dimensions
419648 (632*664) by 880 would have 369290240 elements. If it is of type
FLOAT, then each element takes up 4 bytes, coming to a grand total of
almost 1.4 Gigabyte. Do you have that much memory?
It might be possible to increase the amount of virtual memory (= swap space
on disk) available, but swapping memory from disk invariably slows down
programs quite a lot. A much better approach would be to rewrite your program
so that it needs less memory, e.g. by processing the data in smaller blocks.
hope this helps,
Jaco
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
