| Re: Large array memory problem. [message #46398] |
Tue, 22 November 2005 11:23  |
Rick Towler
Messages: 821
Registered: August 1998
|
Senior Member |
|
|
On a related note, is anyone using IDL x86-64 for linux?
-Rick
Carolina wrote:
> Hi,
>
> I am running IDL 5.5 in a computer that has 2GB of RAM and a paging
> file size of 4GB. However it seems that IDL will not give more than
> about 1.2GB for array storage per IDL session. Unfortunately, I need to
> do an array multiplication where the arrays are double precision
> complex 11000 by 11000 square arrays, and IDL will not even give me
> enough memory for a single dcomplexarr(6500,6500) array!
>
> Am I being too ambitious? Could anyone please tell me if it is possible
> to store such a large array?
>
> I though IDL was the best language for large array manipulations, is it
> true? or are there other languages that are better suited for this?
>
> Many thanks for your time,
>
> Carolina.
>
|
|
|
|
| Re: Large array memory problem. [message #46400 is a reply to message #46398] |
Tue, 22 November 2005 11:14  |
Mark Hadfield
Messages: 783
Registered: May 1995
|
Senior Member |
|
|
Carolina wrote:
> Yes, I was using Windows. I remember seeing some articles in this
> websites that mentioned how to get a couple of mb more of memory, but
> that is definetly not good enough for what I want to do. I didn't think
> it was related to memory fragmentation, since this problem is present
> even with a new IDL session, without any other arrays defined
> previously. I am currently working in a unix machine and I can now
> access around 4gb of memory, and I can allocate 3 complex square arrays
> with dimensions 11000 by 11000, as long as they aren't double precision
> (these occupied 2.9gb all together). It wouldn't let me allocate 2
> double precision ones (these would take about 4gb).
There is an IDL procedure called memtest (or mem_test) that will report
on the areas of *contiguous* memory available to IDL. You can get it here:
http://www.rsinc.com/services/techtip.asp?ttid=3441
You might might want to look here as well:
http://www.rsinc.com/services/techtip.asp?ttid=3512
On my system (Windows 2000 Service Pack 4, 1 GiB RAM, a few GiB of swap
space), when IDL is freshly started, memtest reports the following
Memory block # 1: 1033 Mb (total: 1033 Mb)
Memory block # 2: 388 Mb (total: 1421 Mb)
Memory block # 3: 203 Mb (total: 1624 Mb)
Memory block # 4: 61 Mb (total: 1685 Mb)
Memory block # 5: 58 Mb (total: 1743 Mb)
Memory block # 6: 48 Mb (total: 1791 Mb)
Memory block # 7: 33 Mb (total: 1824 Mb)
Memory block # 8: 21 Mb (total: 1845 Mb)
Memory block # 9: 20 Mb (total: 1865 Mb)
Memory block #10: 17 Mb (total: 1882 Mb)
(where the "Mb"s are supposed to be "MB"s, ie megabytes rather than
megabits).
I doubt that you're goin gto do much better than this in Windows.
--
Mark Hadfield "Kei puwaha te tai nei, Hoea tahi tatou"
m.hadfield@niwa.co.nz
National Institute for Water and Atmospheric Research (NIWA)
|
|
|
|
|
|
| Re: Large array memory problem. [message #46410 is a reply to message #46402] |
Tue, 22 November 2005 05:34 |
Paolo Grigis
Messages: 171
Registered: December 2003
|
Senior Member |
|
|
Is this on windows? There was a thread here dealing about memory
fragmentation etc. some time ago, if I remember correctly the
conclusion was that there is not so much one can do in windows,
but a few tricks could help a bit, and that linux manages the
memory better. But anyway with idl 6.0 or less there is no chance
to allocate a very large array because of the "array has too many
elements" limit anyway.
Can you allocate two double (real) array instead of a complex one?
Ciao,
Paolo
Carolina wrote:
> Hi,
>
> I am running IDL 5.5 in a computer that has 2GB of RAM and a paging
> file size of 4GB. However it seems that IDL will not give more than
> about 1.2GB for array storage per IDL session. Unfortunately, I need to
> do an array multiplication where the arrays are double precision
> complex 11000 by 11000 square arrays, and IDL will not even give me
> enough memory for a single dcomplexarr(6500,6500) array!
>
> Am I being too ambitious? Could anyone please tell me if it is possible
> to store such a large array?
>
> I though IDL was the best language for large array manipulations, is it
> true? or are there other languages that are better suited for this?
>
> Many thanks for your time,
>
> Carolina.
>
|
|
|
|
|
|
|
|
|
|
| Re: Large array memory problem. [message #46545 is a reply to message #46398] |
Thu, 24 November 2005 01:40 |
Nigel Wade
Messages: 286
Registered: March 1998
|
Senior Member |
|
|
Rick Towler wrote:
> On a related note, is anyone using IDL x86-64 for linux?
>
> -Rick
We have it here. I've only just got the 6.2 license manager up and running, so
no-one has tested it except me as yet (I need to impose ulimits first).
IDL> bigarr=bindgen(1024,1024,1024,14)
IDL> help,bigarr,/memory
heap memory used: 15033176469, max: 15033176469, gets: 414, frees: 110
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14482 nmw 20 0 14.1g 11g 4776 R 43.2 73.7 0:24.93 idl
It takes a bit of CPU time to initialise 14Gigs of memory...
--
Nigel Wade, System Administrator, Space Plasma Physics Group,
University of Leicester, Leicester, LE1 7RH, UK
E-mail : nmw@ion.le.ac.uk
Phone : +44 (0)116 2523548, Fax : +44 (0)116 2523555
|
|
|
|
| Re: Large array memory problem. [message #46570 is a reply to message #46474] |
Mon, 28 November 2005 16:34 |
Rick Towler
Messages: 821
Registered: August 1998
|
Senior Member |
|
|
R.G. Stockwell wrote:
> "Rick Towler" wrote...
>
>> On a related note, is anyone using IDL x86-64 for linux?
>>
> Ours has just come online. I have not actually used it, but I
> have snooped around.
> Anything in particular you want to see?
A WinXP-64 port ;)
Actually I rarely have the need to address huge amounts of memory. I'm
just interested in seeing IDL move forward in this area as I know it is
a concern of colleagues and some in this newsgroup. With the death of
the Itanium near, Intel's dream of pushing IA-64 into the low end HPC
market is dead. Itanium has already killed PA-RISC and the beloved
Alpha. uSparc? That's comedy... Even if Fujitsu delivers it isn't an
especially compelling platform. Apple+PPC? We know where that
love-fest is headed. IBM can deliver some excitement with Power but it
will cost you (and that isn't counting the dozens of doughnuts you'll
need to bribe your IT department with to run AIX in their server room).
So those of us with modest budgets and large problems are left with
x86-64, warts and all, for the foreseeable future.
So I guess what I wanted to see was just what I am seeing. Interest in
the linux x86-64 version which will hopefully fuel a WinXP-64 port.
-Rick
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
