| Random 'Unable to allocate memory' error [message #92093] |
Tue, 13 October 2015 06:43  |
Kai Heckel
Messages: 51
Registered: April 2015
|
Member |
|
|
Hello everyone!
I am currently processing a time series. For some reasons the 'Unable to allocate memory' error occurs sometimes. I don't know the reasons as the appearance ist not systematic. Sometimes the error occurs sometime not. I can say that is not file specific.
Any Ideas?
Thanks,
Kai
|
|
|
|
|
|
|
|
|
|
|
|
| Re: Random 'Unable to allocate memory' error [message #92104 is a reply to message #92099] |
Wed, 14 October 2015 04:15  |
Haje Korth
Messages: 651
Registered: May 1997
|
Senior Member |
|
|
Hm, I thought you could indeed be suffering from memory fragmentation, but I thought this is only a windows issue. Since you are on linux, you should be automatically using 64 bit as the 32 bit version is no longer included in recent releases. In this case, I really don't know what causes your issue. I would check with support.
On Wednesday, October 14, 2015 at 3:44:35 AM UTC-4, Kai Heckel wrote:
> Am Dienstag, 13. Oktober 2015 20:11:47 UTC+2 schrieb Haje Korth:
>> I assume this is Windows OS. Did you try the 64-bit version of IDL?
>>
>> On Tuesday, October 13, 2015 at 9:43:42 AM UTC-4, Kai Heckel
>> wrote:
>>> Hello everyone!
>>>
>>> I am currently processing a time series. For some reasons the 'Unable to allocate memory' error occurs sometimes. I don't know the reasons as the appearance ist not systematic. Sometimes the error occurs sometime not. I can say that is not file specific.
>>>
>>> Any Ideas?
>>>
>>> Thanks,
>>> Kai
>
>
> Thanks for your suggestions so far!
>
> I am processing on a linux server. So it shouldn't be anything Windows related.
|
|
|
|
| Re: Random 'Unable to allocate memory' error [message #92107 is a reply to message #92104] |
Wed, 14 October 2015 07:59 |
Paul Van Delst[1]
Messages: 1157
Registered: April 2002
|
Senior Member |
|
|
Hello,
On 10/14/15 07:15, Haje Korth wrote:
> Hm, I thought you could indeed be suffering from memory
> fragmentation,
> but I thought this is only a windows issue. Since you are on linux, you
> should be automatically using 64 bit as the 32 bit version is no longer
> included in recent releases. In this case, I really don't know what
> causes your issue. I would check with support.
This is probably unrelated, but I experienced a similar issue recently.
In my case it was because I was attempting to do some IDL processing on
a frontend (or login) node of the supacomputer I was using. That is
(somehow) restricted by the system setup. I'm guessing everything was
working fine until I passed some memory usage limit or there were too
many other folks on the login node - I was not doing anything too
complicated or big.
When I started an interactive job on a compute node the 'Unable to
allocate memory' error went away. (Aside: The sysdocs said to run IDL, a
whole node should be taken! I guess memory management is an issue in
general with IDL.)
So, if the OP is using IDL on a shared system (s/he did say "server"
so...other users?) maybe the other users are requesting enough resources
to make the OP's script run out of available memory in a seemingly
random way?
(I'm waving my hands about in the air trying to make the scenario seem
reasonable :o)
cheers,
paulv
>
> On Wednesday, October 14, 2015 at 3:44:35 AM UTC-4, Kai Heckel wrote:
>> Am Dienstag, 13. Oktober 2015 20:11:47 UTC+2 schrieb Haje Korth:
>>> I assume this is Windows OS. Did you try the 64-bit version of IDL?
>>>
>>> On Tuesday, October 13, 2015 at 9:43:42 AM UTC-4, Kai Heckel
>>> wrote:
>>>> Hello everyone!
>>>>
>>>> I am currently processing a time series. For some reasons the
>>>> 'Unable to allocate memory' error occurs sometimes. I don't know the
>>>> reasons as the appearance ist not systematic. Sometimes the error occurs
>>>> sometime not. I can say that is not file specific.
>>>>
>>>> Any Ideas?
>>>>
>>>> Thanks,
>>>> Kai
>>
>>
>> Thanks for your suggestions so far!
>>
>> I am processing on a linux server. So it shouldn't be anything Windows related.
>
|
|
|
|
| Re: Random 'Unable to allocate memory' error [message #92118 is a reply to message #92107] |
Thu, 15 October 2015 04:01 |
Kai Heckel
Messages: 51
Registered: April 2015
|
Member |
|
|
Am Mittwoch, 14. Oktober 2015 16:59:35 UTC+2 schrieb Paul van Delst:
> Hello,
>
> On 10/14/15 07:15, Haje Korth wrote:
>> Hm, I thought you could indeed be suffering from memory
>> fragmentation,
>> but I thought this is only a windows issue. Since you are on linux, you
>> should be automatically using 64 bit as the 32 bit version is no longer
>> included in recent releases. In this case, I really don't know what
>> causes your issue. I would check with support.
>
> This is probably unrelated, but I experienced a similar issue recently.
>
> In my case it was because I was attempting to do some IDL processing on
> a frontend (or login) node of the supacomputer I was using. That is
> (somehow) restricted by the system setup. I'm guessing everything was
> working fine until I passed some memory usage limit or there were too
> many other folks on the login node - I was not doing anything too
> complicated or big.
>
> When I started an interactive job on a compute node the 'Unable to
> allocate memory' error went away. (Aside: The sysdocs said to run IDL, a
> whole node should be taken! I guess memory management is an issue in
> general with IDL.)
>
> So, if the OP is using IDL on a shared system (s/he did say "server"
> so...other users?) maybe the other users are requesting enough resources
> to make the OP's script run out of available memory in a seemingly
> random way?
>
> (I'm waving my hands about in the air trying to make the scenario seem
> reasonable :o)
>
> cheers,
>
> paulv
>
>
>>
>> On Wednesday, October 14, 2015 at 3:44:35 AM UTC-4, Kai Heckel wrote:
>>> Am Dienstag, 13. Oktober 2015 20:11:47 UTC+2 schrieb Haje Korth:
>>>> I assume this is Windows OS. Did you try the 64-bit version of IDL?
>>>>
>>>> On Tuesday, October 13, 2015 at 9:43:42 AM UTC-4, Kai Heckel
>>>> wrote:
>>>> > Hello everyone!
>>>> >
>>>> > I am currently processing a time series. For some reasons the
>>>> > 'Unable to allocate memory' error occurs sometimes. I don't know the
>>>> > reasons as the appearance ist not systematic. Sometimes the error occurs
>>>> > sometime not. I can say that is not file specific.
>>>> >
>>>> > Any Ideas?
>>>> >
>>>> > Thanks,
>>>> > Kai
>>>
>>>
>>> Thanks for your suggestions so far!
>>>
>>> I am processing on a linux server. So it shouldn't be anything Windows related.
>>
Good idea Paul but I am the only one processing on the server.... However, 32 GB of RAM should be enough for me and others to process.
Here is another idea I have:
In all processes large arrays are created. Not only one but multiple arrays. Is possible that these variables need to be deleted after they were used in each file loop?
BTW: Is it possible to delete all variable except for one?
|
|
|
|
|
|
|
|
|
|
|
|
| Re: Random 'Unable to allocate memory' error [message #92166 is a reply to message #92094] |
Thu, 22 October 2015 11:20 |
Chip Helms
Messages: 24
Registered: November 2012
|
Junior Member |
|
|
On Tuesday, October 13, 2015 at 10:31:24 AM UTC-4, Kai Heckel wrote:
> One more thing I should mention.
> I am processing parallel (2 processes). Those two should not exceed the memory limit of the server.
Are you using the IDL-IDL bridge for this? If this is the case, this thread might help you:
https://groups.google.com/d/topic/comp.lang.idl-pvwave/J2QMU UvmPLY/discussion
Supposedly, the issue was fixed in 8.3 (of course, I'm still using 8.2.3 due to budget issues :) ). I'm not entirely sure how I solved the issue (or if I did at all). I can try digging a bit more if this is, in fact, related to your issue.
Cheers,
Chip
|
|
|
|
| Re: Random 'Unable to allocate memory' error [message #92169 is a reply to message #92166] |
Thu, 22 October 2015 23:55 |
Kai Heckel
Messages: 51
Registered: April 2015
|
Member |
|
|
Am Donnerstag, 22. Oktober 2015 20:20:05 UTC+2 schrieb Chip Helms:
> On Tuesday, October 13, 2015 at 10:31:24 AM UTC-4, Kai Heckel wrote:
>> One more thing I should mention.
>> I am processing parallel (2 processes). Those two should not exceed the memory limit of the server.
>
> Are you using the IDL-IDL bridge for this? If this is the case, this thread might help you:
> https://groups.google.com/d/topic/comp.lang.idl-pvwave/J2QMU UvmPLY/discussion
>
> Supposedly, the issue was fixed in 8.3 (of course, I'm still using 8.2.3 due to budget issues :) ). I'm not entirely sure how I solved the issue (or if I did at all). I can try digging a bit more if this is, in fact, related to your issue.
>
> Cheers,
> Chip
Hey Chip,
thanks for the suggestion, but I am using 8.4 and this has already been fixed I guess. Anyways I think that it could be CPU related because on the 1st server 2 processes run properly (16 cores, 32GB RAM). On the 2nd server only one process can be run (8 cores, 32GB RAM).
So I am only running 3 processes now, which is fine for me.
However, thank you all for your suggestions!
Cheers,
Kai
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
