| Re: Help Large_array [message #52614] |
Tue, 20 February 2007 09:27  |
JD Smith
Messages: 850
Registered: December 1999
|
Senior Member |
|
|
On Tue, 20 Feb 2007 12:15:07 +0000, Nigel Wade wrote:
>> b) What's the solution to the operation of large array once and for
>> all(I mean by software not changing hardware)?
>
> There is no solution if your hardware is 32bit. You need 64bit hardware, a 64bit
> OS and a 64bit version of IDL if you want to use arrays of that size.
Or you need to reconsider your methods to use only as much memory as your
system can support at a given time, working on data in chunks. ASSOC is
one (limited) IDL method for doing this, but there are others. It also
really helps if you layout your data such that the pieces which are needed
for a given calculation are nearby each other in memory.
JD
|
|
|
|
| Re: Help Large_array [message #52616 is a reply to message #52614] |
Tue, 20 February 2007 04:15  |
Nigel Wade
Messages: 286
Registered: March 1998
|
Senior Member |
|
|
wx_f@sohu.com wrote:
> w...@sohu.com wrote:
>
>>> I need to build a ut=dblarr(36000,500,25) in IDL's pro---sfit.pro.
>
>> It is not possible to build such an array with IDL 32 bit version,
>> no matter how much memory your system has.
>>
>> Ciao,
>> Paolo
>
> Mr Pallo
> Since ut=dblarr(20000,500,25)
this requires 2GB of contiguous memory...
> and ut=fltarr(36000,500,25)can be
this requires 1.8GB of contiguous memory...
> created by my PC(linux_IDL6.0),why it cannot do better(3.6 vs 2,double
> vs float)?
for double it would require 3.8GB of contiguous memory.
> a) What's the reason.Could you give a formula to explain
> quantitatively?
There is a limit to the amount of memory which your OS can address. There is
also a limit below this as to the amount of contiguous memory which the OS can
allocate to a process. For a 32bit OS the absolute maximum amount of
addressable memory is 4GB. But since the OS requires space, and there are
likely to be many other running processes which are using chunks of that
address space the chance of you being able to allocate 3.8GB of the available
4GB for your array are vanishingly small. There is no quantitative formula for
the maximum size of an array which you can allocate, it varies with the amount
of memory in use and the amount of fragmentation of the allocated memory
chunks. The only rule is that it will be less than the maximum.
> b) What's the solution to the operation of large array once and for
> all(I mean by software not changing hardware)?
There is no solution if your hardware is 32bit. You need 64bit hardware, a 64bit
OS and a 64bit version of IDL if you want to use arrays of that size.
--
Nigel Wade, System Administrator, Space Plasma Physics Group,
University of Leicester, Leicester, LE1 7RH, UK
E-mail : nmw@ion.le.ac.uk
Phone : +44 (0)116 2523548, Fax : +44 (0)116 2523555
|
|
|
|
| Re: Help Large_array [message #52617 is a reply to message #52616] |
Tue, 20 February 2007 02:53 |
Paolo Grigis
Messages: 171
Registered: December 2003
|
Senior Member |
|
|
wx_f@sohu.com wrote:
> w...@sohu.com wrote:
>
>>> I need to build a ut=dblarr(36000,500,25) in IDL's pro---sfit.pro.
>
>> It is not possible to build such an array with IDL 32 bit version,
>> no matter how much memory your system has.
>>
>> Ciao,
>> Paolo
>
> Mr Pallo
> Since ut=dblarr(20000,500,25) and ut=fltarr(36000,500,25)can be
> created by my PC(linux_IDL6.0),why it cannot do better(3.6 vs 2,double
> vs float)?
> a) What's the reason.Could you give a formula to explain
> quantitatively?
Take the number of elements in your array, call it n.
Take the size in byte of your data type (i.e. 4 for float,
8 for doubles etc.), call it m.
Then m*n is limited to 2^31.
As for the reason, we could speculate that IDL (32 bit) uses
long, signed integers internally to access the memory locations
where the elements of the array reside.
> b) What's the solution to the operation of large array once and for
> all(I mean by software not changing hardware)?
You'll be the one who has to find the best solution suited to
your problem, somehow you'll have to work with smaller chunks
of data.
Ciao,
Paolo
>
> thanks
> wxf
>
>
>
|
|
|
|
| Re: Help Large_array [message #52618 is a reply to message #52617] |
Tue, 20 February 2007 01:44 |
wx_f
Messages: 2
Registered: February 2007
|
Junior Member |
|
|
w...@sohu.com wrote:
>> I need to build a ut=dblarr(36000,500,25) in IDL's pro---sfit.pro.
> It is not possible to build such an array with IDL 32 bit version,
> no matter how much memory your system has.
>
> Ciao,
> Paolo
Mr Pallo
Since ut=dblarr(20000,500,25) and ut=fltarr(36000,500,25)can be
created by my PC(linux_IDL6.0),why it cannot do better(3.6 vs 2,double
vs float)?
a) What's the reason.Could you give a formula to explain
quantitatively?
b) What's the solution to the operation of large array once and for
all(I mean by software not changing hardware)?
thanks
wxf
|
|
|
|
|
|
|
|
| Re: Help Large_array [message #53144 is a reply to message #52616] |
Tue, 27 March 2007 07:20 |
liamgumley
Messages: 74
Registered: June 2005
|
Member |
|
|
On Feb 20, 7:15 am, Nigel Wade <n...@ion.le.ac.uk> wrote:
> There is no solution if your hardware is 32bit. You need 64bit hardware, a 64bit
> OS and a 64bit version of IDL if you want to use arrays of that size.
The array can be created in 64-bit IDL:
IDL Version 6.2 (linux x86_64 m64). (c) 2005, Research Systems, Inc.
IDL> b=dblarr(36000, 500, 25)
IDL> print, memory()
3600805575 452
134 3600805575
Hardware is a Sun Fire V40z with 8GB RAM running 64-bit CentOS.
Cheers,
Liam.
Practical IDL Programming
http://www.gumley.com/
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
