| "Unable to allocate memory" when using H5_parse iteratively [message #88394] |
Mon, 21 April 2014 14:59  |
robindesvolcans
Messages: 2
Registered: April 2014
|
Junior Member |
|
|
Hi all,
I'm Robin, a new member on this group, using IDL to process satellite images for volcano monitoring.
I've written a code to process large number of data from the ozone monitoring instrument. These are HDF5 files weighing 28 Mb each.
The code interactively calls the H5_parse function to open the files and extract the necessary variables, which undergo some basic processing (gridding, mapping and calculating the total SO2 mass) The variables are overwritten at each loop iteration. However after 30-40 iterations the H5_parse function returns me the unable to allocate memory error message. It looks as if the H5_parse function was not emptying its temporary memory use, but when i look at the task manager's diplay of the memory use, it stays constant and low (20%).
Does someone have an idea of how to solve the problem?
Thanks a lot
robin
|
|
|
|
| Re: "Unable to allocate memory" when using H5_parse iteratively [message #88395 is a reply to message #88394] |
Mon, 21 April 2014 15:09  |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
robindesvolcans@gmail.com writes:
> I'm Robin, a new member on this group, using IDL to process satellite images for volcano monitoring.
> I've written a code to process large number of data from the ozone monitoring instrument. These are HDF5 files weighing 28 Mb each.
> The code interactively calls the H5_parse function to open the files
and extract the necessary variables, which undergo some basic processing
(gridding, mapping and calculating the total SO2 mass) The variables are
overwritten at each loop iteration. However after 30-40 iterations the
H5_parse function returns me the unable to allocate memory error
message. It looks as if the H5_parse function was not emptying its
temporary memory use, but when i look at the task manager's display of
the memory use, it stays constant and low (20%).
> Does someone have an idea of how to solve the problem?
You may be overwriting the variables, but it is extremely unlikely you
are cleaning up all the pointers and other heap variables returned by
the H5_Parse function. I would try doing this at the end of each file
processed:
for j=0,whatever do begin
h5data = HD_Parse(file[j])
...
Heap_Free, h5data
endfor
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.idlcoyote.com/
Sepore ma de ni thue. ("Perhaps thou speakest truth.")
|
|
|
|
| Re: "Unable to allocate memory" when using H5_parse iteratively [message #88398 is a reply to message #88395] |
Mon, 21 April 2014 16:40 |
robindesvolcans
Messages: 2
Registered: April 2014
|
Junior Member |
|
|
Hi David and Thank You for your reply.
I added this line in my code but unfortunately the problem remains. Indeed the memory use increases with time, and the error occurs when it reaches about 350 Mb (on a total of 2000)
I tried processing the data by smaller bunches, but it happens that after processing without problem a smaller bunch of 05 files, the program bugs when i run it for processing the next one, for the same error.
Le lundi 21 avril 2014 17:09:24 UTC-5, David Fanning a écrit :
> robindesvolcans@gmail.com writes:
>
>
>
>> I'm Robin, a new member on this group, using IDL to process satellite images for volcano monitoring.
>
>> I've written a code to process large number of data from the ozone monitoring instrument. These are HDF5 files weighing 28 Mb each.
>
>> The code interactively calls the H5_parse function to open the files
>
> and extract the necessary variables, which undergo some basic processing
>
> (gridding, mapping and calculating the total SO2 mass) The variables are
>
> overwritten at each loop iteration. However after 30-40 iterations the
>
> H5_parse function returns me the unable to allocate memory error
>
> message. It looks as if the H5_parse function was not emptying its
>
> temporary memory use, but when i look at the task manager's display of
>
> the memory use, it stays constant and low (20%).
>
>> Does someone have an idea of how to solve the problem?
>
>
>
> You may be overwriting the variables, but it is extremely unlikely you
>
> are cleaning up all the pointers and other heap variables returned by
>
> the H5_Parse function. I would try doing this at the end of each file
>
> processed:
>
>
>
> for j=0,whatever do begin
>
> h5data = HD_Parse(file[j])
>
> ...
>
> Heap_Free, h5data
>
> endfor
>
>
>
> Cheers,
>
>
>
> David
>
> --
>
> David Fanning, Ph.D.
>
> Fanning Software Consulting, Inc.
>
> Coyote's Guide to IDL Programming: http://www.idlcoyote.com/
>
> Sepore ma de ni thue. ("Perhaps thou speakest truth.")
|
|
|
|
| Re: "Unable to allocate memory" when using H5_parse iteratively [message #88568 is a reply to message #88398] |
Tue, 13 May 2014 08:20 |
james.n.sweet
Messages: 1
Registered: May 2014
|
Junior Member |
|
|
On Monday, April 21, 2014 5:40:50 PM UTC-6, robinde...@gmail.com wrote:
> Hi David and Thank You for your reply.
>
> I added this line in my code but unfortunately the problem remains. Indeed the memory use increases with time, and the error occurs when it reaches about 350 Mb (on a total of 2000)
>
> I tried processing the data by smaller bunches, but it happens that after processing without problem a smaller bunch of 05 files, the program bugs when i run it for processing the next one, for the same error.
>
> Hi,
I've had a possibly similar problem. When using h5 parse, the memory usage would increase as more files were processed. I informed envi but the didn't follow up. My solution was to use the h5 open/read/close functions explicitly rather than using h5 parse. It's a pain but it is faster than parse (in my case anyway) and the memory problem is gone.
Hope this helps!
Jim
>
> Le lundi 21 avril 2014 17:09:24 UTC-5, David Fanning a écrit :
>
>> robindesvolcans@gmail.com writes:
>
>>
>
>>
>
>>
>
>>> I'm Robin, a new member on this group, using IDL to process satellite images for volcano monitoring.
>
>>
>
>>> I've written a code to process large number of data from the ozone monitoring instrument. These are HDF5 files weighing 28 Mb each.
>
>>
>
>>> The code interactively calls the H5_parse function to open the files
>
>>
>
>> and extract the necessary variables, which undergo some basic processing
>
>>
>
>> (gridding, mapping and calculating the total SO2 mass) The variables are
>
>>
>
>> overwritten at each loop iteration. However after 30-40 iterations the
>
>>
>
>> H5_parse function returns me the unable to allocate memory error
>
>>
>
>> message. It looks as if the H5_parse function was not emptying its
>
>>
>
>> temporary memory use, but when i look at the task manager's display of
>
>>
>
>> the memory use, it stays constant and low (20%).
>
>>
>
>>> Does someone have an idea of how to solve the problem?
>
>>
>
>>
>
>>
>
>> You may be overwriting the variables, but it is extremely unlikely you
>
>>
>
>> are cleaning up all the pointers and other heap variables returned by
>
>>
>
>> the H5_Parse function. I would try doing this at the end of each file
>
>>
>
>> processed:
>
>>
>
>>
>
>>
>
>> for j=0,whatever do begin
>
>>
>
>> h5data = HD_Parse(file[j])
>
>>
>
>> ...
>
>>
>
>> Heap_Free, h5data
>
>>
>
>> endfor
>
>>
>
>>
>
>>
>
>> Cheers,
>
>>
>
>>
>
>>
>
>> David
>
>>
>
>> --
>
>>
>
>> David Fanning, Ph.D.
>
>>
>
>> Fanning Software Consulting, Inc.
>
>>
>
>> Coyote's Guide to IDL Programming: http://www.idlcoyote.com/
>
>>
>
>> Sepore ma de ni thue. ("Perhaps thou speakest truth.")
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
