comp.lang.idl-pvwave archive - RDF feed
type: channel
description: Messages from Usenet group comp.lang.idl-pvwave, compiled by Paulo Penteado
items: Re: Help Large_array

Re: Help Large_array
type: item
description: This is truly annoying at times and I have never found a way around it. Have a look at http://www.dfanning.com/misc_tips/submemory.html for a discussion of this. Brian ------------------------------------------------------------ --------- Brian A. Larsen Dept. of Physics Space Science and Engineering Lab (SSEL) Montana State University - Bozeman Bozeman, MT 59717
creator: Brian Larsen
date: 2007-02-19T21:57:11-00:00

Re: Help Large_array
type: item
description: wx_f@sohu.com wrote: > [...] > I need to build a ut=dblarr(36000,500,25) in IDL's pro---sfit.pro. It is not possible to build such an array with IDL 32 bit version, no matter how much memory your system has. Ciao, Paolo
creator: Paolo Grigis
date: 2007-02-20T08:02:04-00:00

Re: Help Large_array
type: item
description: w...@sohu.com wrote: >> I need to build a ut=dblarr(36000,500,25) in IDL's pro---sfit.pro. > It is not possible to build such an array with IDL 32 bit version, > no matter how much memory your system has. > > Ciao, > Paolo Mr Pallo Since ut=dblarr(20000,500,25) and ut=fltarr(36000,500,25)can be created by my PC(linux_IDL6.0),why it cannot do better(3.6 vs 2,double vs float)? a) What's the reason.Could you give a formula to explain quantitatively? b) What's the solution to the operation of large array once and for all(I mean by software not changing hardware)? thanks wxf
creator: wx_f
date: 2007-02-20T09:44:46-00:00

Re: Help Large_array
type: item
description: wx_f@sohu.com wrote: > w...@sohu.com wrote: > >>> I need to build a ut=dblarr(36000,500,25) in IDL's pro---sfit.pro. > >> It is not possible to build such an array with IDL 32 bit version, >> no matter how much memory your system has. >> >> Ciao, >> Paolo > > Mr Pallo > Since ut=dblarr(20000,500,25) and ut=fltarr(36000,500,25)can be > created by my PC(linux_IDL6.0),why it cannot do better(3.6 vs 2,double > vs float)? > a) What's the reason.Could you give a formula to explain > quantitatively? Take the number of elements in your array, call it n. Take the size in byte of your data type (i.e. 4 for float, 8 for doubles etc.), call it m. Then m*n is limited to 2^31. As for the reason, we could speculate that IDL (32 bit) uses long, signed integers internally to access the memory locations where the elements of the array reside. > b) What's the solution to the operation of large array once and for > all(I mean by software not changing hardware)? You'll be the one who has to find the best solution suited to your problem, somehow you'll have to work with smaller chunks of data. Ciao, Paolo > > thanks > wxf > > >
creator: Paolo Grigis
date: 2007-02-20T10:53:58-00:00

Re: Help Large_array
type: item
description: wx_f@sohu.com wrote: > w...@sohu.com wrote: > >>> I need to build a ut=dblarr(36000,500,25) in IDL's pro---sfit.pro. > >> It is not possible to build such an array with IDL 32 bit version, >> no matter how much memory your system has. >> >> Ciao, >> Paolo > > Mr Pallo > Since ut=dblarr(20000,500,25) this requires 2GB of contiguous memory... > and ut=fltarr(36000,500,25)can be this requires 1.8GB of contiguous memory... > created by my PC(linux_IDL6.0),why it cannot do better(3.6 vs 2,double > vs float)? for double it would require 3.8GB of contiguous memory. > a) What's the reason.Could you give a formula to explain > quantitatively? There is a limit to the amount of memory which your OS can address. There is also a limit below this as to the amount of contiguous memory which the OS can allocate to a process. For a 32bit OS the absolute maximum amount of addressable memory is 4GB. But since the OS requires space, and there are likely to be many other running processes which are using chunks of that address space the chance of you being able to allocate 3.8GB of the available 4GB for your array are vanishingly small. There is no quantitative formula for the maximum size of an array which you can allocate, it varies with the amount of memory in use and the amount of fragmentation of the allocated memory chunks. The only rule is that it will be less than the maximum. > b) What's the solution to the operation of large array once and for > all(I mean by software not changing hardware)? There is no solution if your hardware is 32bit. You need 64bit hardware, a 64bit OS and a 64bit version of IDL if you want to use arrays of that size. -- Nigel Wade, System Administrator, Space Plasma Physics Group, University of Leicester, Leicester, LE1 7RH, UK E-mail : nmw@ion.le.ac.uk Phone : +44 (0)116 2523548, Fax : +44 (0)116 2523555
creator: Nigel Wade
date: 2007-02-20T12:15:07-00:00

Re: Help Large_array
type: item
description: On Tue, 20 Feb 2007 12:15:07 +0000, Nigel Wade wrote: >> b) What's the solution to the operation of large array once and for >> all(I mean by software not changing hardware)? > > There is no solution if your hardware is 32bit. You need 64bit hardware, a 64bit > OS and a 64bit version of IDL if you want to use arrays of that size. Or you need to reconsider your methods to use only as much memory as your system can support at a given time, working on data in chunks. ASSOC is one (limited) IDL method for doing this, but there are others. It also really helps if you layout your data such that the pieces which are needed for a given calculation are nearby each other in memory. JD
creator: JD Smith
date: 2007-02-20T17:27:03-00:00

Re: Help Large_array
type: item
description: On Feb 20, 7:15 am, Nigel Wade <n...@ion.le.ac.uk> wrote: > There is no solution if your hardware is 32bit. You need 64bit hardware, a 64bit > OS and a 64bit version of IDL if you want to use arrays of that size. The array can be created in 64-bit IDL: IDL Version 6.2 (linux x86_64 m64). (c) 2005, Research Systems, Inc. IDL> b=dblarr(36000, 500, 25) IDL> print, memory() 3600805575 452 134 3600805575 Hardware is a Sun Fire V40z with 8GB RAM running 64-bit CentOS. Cheers, Liam. Practical IDL Programming http://www.gumley.com/
creator: liamgumley
date: 2007-03-27T14:20:35-00:00