| Re: How to make large array [message #24436] |
Thu, 29 March 2001 00:36 |
Martin Schultz
Messages: 515
Registered: August 1997
|
Senior Member |
|
|
renjie wrote:
>
> I tried to make large array, however, in Windows,
> the largest I can made is 256MB, and in Unix is 1 GB,
> it's supposed that the limitation is 2GB, do you have any
> idea about the largest array in IDL?
Hmmmm. First of all, I don't think it is necessary to create arrays
larger than a couple of MBs in memory. And they get really clumsy and
slow: you must be extremely careful about when do you pass something
by reference or by value and when IDL will make a temporary copy of
the array. Think hard whether you can come up with a solution where
most of the data resides in a file and is only accessed piecewise.
Second, I recall that Erik Korpela
(http://albert.ssl.berkeley.edu/~korpela/mmap/) wrote some stuff to
allow memory mapping of IDL arrays. From his web page it seems like
this would solve your problems in a very nice way.
Good luck,
Martin
--
[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[ [[[[[[[
[[ Dr. Martin Schultz Max-Planck-Institut fuer Meteorologie [[
[[ Bundesstr. 55, 20146 Hamburg [[
[[ phone: +49 40 41173-308 [[
[[ fax: +49 40 41173-298 [[
[[ martin.schultz@dkrz.de [[
[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[ [[[[[[[
|
|
|
|
| Re: How to make large array [message #24444 is a reply to message #24436] |
Wed, 28 March 2001 14:53  |
Craig Markwardt
Messages: 1869
Registered: November 1996
|
Senior Member |
|
|
renjie <renjie.he@uth.tmc.edu> writes:
> I tried to make large array, however, in Windows,
> the largest I can made is 256MB, and in Unix is 1 GB,
> it's supposed that the limitation is 2GB, do you have any
> idea about the largest array in IDL?
I have no idea about Windows. Sometimes you encounter artificial
limits under Unix because of your process limits are set to low. Use
"limit datasize unlimited" before running IDL under tcsh. Use the
"ulimit" command under sh.
Craig
--
------------------------------------------------------------ --------------
Craig B. Markwardt, Ph.D. EMAIL: craigmnet@cow.physics.wisc.edu
Astrophysics, IDL, Finance, Derivatives | Remove "net" for better response
------------------------------------------------------------ --------------
|
|
|
|
| Re: How to make large array [message #24451 is a reply to message #24444] |
Wed, 28 March 2001 13:21 |
renjie
Messages: 10
Registered: October 1999
|
Junior Member |
|
|
Great.
If you can explain the manner I made 1GB in win2000, the method does not
work in win98 for producing 500MB.
Mark Hadfield wrote:
> "renjie" <renjie.he@uth.tmc.edu> wrote in message
> news:3AC228EE.6A26844C@uth.tmc.edu...
>> The results below are obtained in WIN98 [i.e. 256MB limit on array size],
> in WIN2000
>> I can make single array of 512MB [and 1 GB], can someone explain it.
>
> Windows 9x is a 32-bit shell on a 16-bit operating system originally written
> for an 8-bit processor by a 4-bit company without 2 bits of sense.
> (Apologies to the unknown originator of this saying for any misquoting.)
> Windows 2000 & NT aren't (except for the 4-bit company thing).
>
> I.e. I don't know exactly why Win98 has a limit on array size that Win 2000
> and NT don't, but I'm not at all surprised.
>
> ---
> Mark Hadfield
> m.hadfield@niwa.cri.nz http://katipo.niwa.cri.nz/~hadfield
> National Institute for Water and Atmospheric Research
|
|
|
|
| Re: How to make large array [message #24453 is a reply to message #24451] |
Wed, 28 March 2001 13:00 |
Mark Hadfield
Messages: 783
Registered: May 1995
|
Senior Member |
|
|
"renjie" <renjie.he@uth.tmc.edu> wrote in message
news:3AC228EE.6A26844C@uth.tmc.edu...
> The results below are obtained in WIN98 [i.e. 256MB limit on array size],
in WIN2000
> I can make single array of 512MB [and 1 GB], can someone explain it.
Windows 9x is a 32-bit shell on a 16-bit operating system originally written
for an 8-bit processor by a 4-bit company without 2 bits of sense.
(Apologies to the unknown originator of this saying for any misquoting.)
Windows 2000 & NT aren't (except for the 4-bit company thing).
I.e. I don't know exactly why Win98 has a limit on array size that Win 2000
and NT don't, but I'm not at all surprised.
---
Mark Hadfield
m.hadfield@niwa.cri.nz http://katipo.niwa.cri.nz/~hadfield
National Institute for Water and Atmospheric Research
|
|
|
|
| Re: How to make large array [message #24456 is a reply to message #24453] |
Wed, 28 March 2001 10:33 |
renjie
Messages: 10
Registered: October 1999
|
Junior Member |
|
|
<!doctype html public "-//w3c//dtd html 4.0 transitional//en">
<html>
I can make array around 1GB in WIN2000 as below:
<p>IDL> b=bytarr(1024,1024,512)
<br>IDL> a=bytarr(1024,1024,512)
<br>IDL> c=bytarr(1024,1024,512)
<br>% Unable to allocate memory: to make array.
<br>% Execution halted at: $MAIN$
<br>IDL> help,/mem
<br>heap memory used: 1074113975, max: 1074114003, gets:
448, frees: 209
<br>IDL> a=0
<br>IDL> b=0
<br>IDL> a=bytarr(1024,1024,1000)
<br>IDL> help,/mem
<br>heap memory used: 1048948020, max: 1074113975, gets:
454, frees: 216
<br>
<p>renjie wrote:
<blockquote TYPE=CITE>The results below are obtained in WIN98, in WIN2000
I can make single array of 512MB, can someone explain it.
<p>renjie wrote:
<blockquote TYPE=CITE>
<blockquote TYPE=CITE>
<pre></pre>
</blockquote>
<p><br>Below are my tests, though the heap memory can be increased till
the maximum of RAM+VM (my machine is 512MB+2GB), the
<br>maximum size for a single arrary can not be larger than 256MB, I really
need to make array around 1GB, if anyboy can config out
<br>what is wrong with my system. Thanks.
<blockquote TYPE=CITE>
<pre>IDL> b=bytarr(1024,1024,255)
IDL> help, /mem
heap memory used: 267652748, max: 535039712, gets: 395, frees: 148
IDL> b=bytarr(1024,1024,256)
% Unable to allocate memory: to make array.
% Execution halted at: $MAIN$ & ;nbsp; & ;nbsp;
IDL> help, /mem
heap memory used: 267652748, max: 267652776, gets: 398, frees: 151
IDL> c=bytarr(1024,1024,255)
IDL> help, /mem
heap memory used: 535039699, max: 535039727, gets: 402, frees: 154
IDL> d=bytarr(1024,1024,255)
% Unable to allocate memory: to make array.
% Execution halted at: $MAIN$ & ;nbsp; & ;nbsp;
IDL> help, /mem
heap memory used: 535039699, max: 535039727, gets: 405, frees: 157
IDL> d=bytarr(1024,1024,125)
IDL> help, /mem
heap memory used: 666111805, max: 666111833, gets: 409, frees: 160</pre>
</blockquote>
Alex Schuster wrote:
<blockquote TYPE=CITE>renjie wrote:
<p>> I tried to make large array, however, in Windows,
<br>> the largest I can made is 256MB, and in Unix is 1 GB,
<br>> it's supposed that the limitation is 2GB, do you have any
<br>> idea about the largest array in IDL?
<p>We habe encountered a problem with IDL and NT, where IDL is only able
to
<br>allocate 800 MB of memory in one step, even if it can allocate more
<br>memory afterwards. But 256 MB is much too low, presuming there is enough
<br>physical+virtual memory.
<p>In Unix, I can allocate more than 1 GB:
<p>--> idl
<br>IDL Version 5.2.1 (sunos sparc). (c) 1999, Research Systems, Inc.
<br>Installation number: 80864-0.
<br>Licensed for use by: MPI Forschung
<p>IDL> b=bytarr(1024,1024,1200)
<br>IDL> help, /mem
<br>heap memory used: 1258497902, max: 1258497930, gets:
227,
<br> frees: & ;nbsp; 89
<p>The limit command tells you about the datasize you are allowed to
<br>allocate, maybe this is set too low for you. And maybe there's a similar
<br>setting for Windows?
<p> Alex
<br>--
<br> Alex Schuster Wonko@weird.cologne.de
PGP Key available
<br>
alex@pet.mpin-koeln.mpg.de</blockquote>
</blockquote>
</blockquote>
</html>
|
|
|
|
| Re: How to make large array [message #24457 is a reply to message #24456] |
Wed, 28 March 2001 10:09 |
renjie
Messages: 10
Registered: October 1999
|
Junior Member |
|
|
<!doctype html public "-//w3c//dtd html 4.0 transitional//en">
<html>
The results below are obtained in WIN98, in WIN2000 I can make single array
of 512MB, can someone explain it.
<p>renjie wrote:
<blockquote TYPE=CITE>
<blockquote TYPE=CITE>
<pre></pre>
</blockquote>
<p><br>Below are my tests, though the heap memory can be increased till
the maximum of RAM+VM (my machine is 512MB+2GB), the
<br>maximum size for a single arrary can not be larger than 256MB, I really
need to make array around 1GB, if anyboy can config out
<br>what is wrong with my system. Thanks.
<blockquote TYPE=CITE>
<pre>IDL> b=bytarr(1024,1024,255)
IDL> help, /mem
heap memory used: 267652748, max: 535039712, gets: 395, frees: 148
IDL> b=bytarr(1024,1024,256)
% Unable to allocate memory: to make array.
% Execution halted at: $MAIN$ & ;nbsp; & ;nbsp;
IDL> help, /mem
heap memory used: 267652748, max: 267652776, gets: 398, frees: 151
IDL> c=bytarr(1024,1024,255)
IDL> help, /mem
heap memory used: 535039699, max: 535039727, gets: 402, frees: 154
IDL> d=bytarr(1024,1024,255)
% Unable to allocate memory: to make array.
% Execution halted at: $MAIN$ & ;nbsp; & ;nbsp;
IDL> help, /mem
heap memory used: 535039699, max: 535039727, gets: 405, frees: 157
IDL> d=bytarr(1024,1024,125)
IDL> help, /mem
heap memory used: 666111805, max: 666111833, gets: 409, frees: 160</pre>
</blockquote>
Alex Schuster wrote:
<blockquote TYPE=CITE>renjie wrote:
<p>> I tried to make large array, however, in Windows,
<br>> the largest I can made is 256MB, and in Unix is 1 GB,
<br>> it's supposed that the limitation is 2GB, do you have any
<br>> idea about the largest array in IDL?
<p>We habe encountered a problem with IDL and NT, where IDL is only able
to
<br>allocate 800 MB of memory in one step, even if it can allocate more
<br>memory afterwards. But 256 MB is much too low, presuming there is enough
<br>physical+virtual memory.
<p>In Unix, I can allocate more than 1 GB:
<p>--> idl
<br>IDL Version 5.2.1 (sunos sparc). (c) 1999, Research Systems, Inc.
<br>Installation number: 80864-0.
<br>Licensed for use by: MPI Forschung
<p>IDL> b=bytarr(1024,1024,1200)
<br>IDL> help, /mem
<br>heap memory used: 1258497902, max: 1258497930, gets:
227,
<br> frees: & ;nbsp; 89
<p>The limit command tells you about the datasize you are allowed to
<br>allocate, maybe this is set too low for you. And maybe there's a similar
<br>setting for Windows?
<p> Alex
<br>--
<br> Alex Schuster Wonko@weird.cologne.de
PGP Key available
<br>
alex@pet.mpin-koeln.mpg.de</blockquote>
</blockquote>
</html>
|
|
|
|
| Re: How to make large array [message #24458 is a reply to message #24457] |
Wed, 28 March 2001 10:03 |
renjie
Messages: 10
Registered: October 1999
|
Junior Member |
|
|
<!doctype html public "-//w3c//dtd html 4.0 transitional//en">
<html>
<blockquote TYPE=CITE>
<pre></pre>
</blockquote>
<p><br>Below are my tests, though the heap memory can be increased till
the maximum of RAM+VM (my machine is 512MB+2GB), the
<br>maximum size for a single arrary can not be larger than 256MB, I really
need to make array around 1GB, if anyboy can config out
<br>what is wrong with my system. Thanks.
<blockquote TYPE=CITE>
<pre>IDL> b=bytarr(1024,1024,255)
IDL> help, /mem
heap memory used: 267652748, max: 535039712, gets: 395, frees: 148
IDL> b=bytarr(1024,1024,256)
% Unable to allocate memory: to make array.
% Execution halted at: $MAIN$ & ;nbsp; & ;nbsp;
IDL> help, /mem
heap memory used: 267652748, max: 267652776, gets: 398, frees: 151
IDL> c=bytarr(1024,1024,255)
IDL> help, /mem
heap memory used: 535039699, max: 535039727, gets: 402, frees: 154
IDL> d=bytarr(1024,1024,255)
% Unable to allocate memory: to make array.
% Execution halted at: $MAIN$ & ;nbsp; & ;nbsp;
IDL> help, /mem
heap memory used: 535039699, max: 535039727, gets: 405, frees: 157
IDL> d=bytarr(1024,1024,125)
IDL> help, /mem
heap memory used: 666111805, max: 666111833, gets: 409, frees: 160</pre>
</blockquote>
<p>Alex Schuster wrote:
<blockquote TYPE=CITE>renjie wrote:
<p>> I tried to make large array, however, in Windows,
<br>> the largest I can made is 256MB, and in Unix is 1 GB,
<br>> it's supposed that the limitation is 2GB, do you have any
<br>> idea about the largest array in IDL?
<p>We habe encountered a problem with IDL and NT, where IDL is only able
to
<br>allocate 800 MB of memory in one step, even if it can allocate more
<br>memory afterwards. But 256 MB is much too low, presuming there is enough
<br>physical+virtual memory.
<p>In Unix, I can allocate more than 1 GB:
<p>--> idl
<br>IDL Version 5.2.1 (sunos sparc). (c) 1999, Research Systems, Inc.
<br>Installation number: 80864-0.
<br>Licensed for use by: MPI Forschung
<p>IDL> b=bytarr(1024,1024,1200)
<br>IDL> help, /mem
<br>heap memory used: 1258497902, max: 1258497930, gets:
227,
<br> frees: & ;nbsp; 89
<p>The limit command tells you about the datasize you are allowed to
<br>allocate, maybe this is set too low for you. And maybe there's a similar
<br>setting for Windows?
<p> Alex
<br>--
<br> Alex Schuster Wonko@weird.cologne.de
PGP Key available
<br>
alex@pet.mpin-koeln.mpg.de</blockquote>
</html>
|
|
|
|
| Re: How to make large array [message #24464 is a reply to message #24458] |
Wed, 28 March 2001 08:31 |
Alex Schuster
Messages: 124
Registered: February 1997
|
Senior Member |
|
|
renjie wrote:
> I tried to make large array, however, in Windows,
> the largest I can made is 256MB, and in Unix is 1 GB,
> it's supposed that the limitation is 2GB, do you have any
> idea about the largest array in IDL?
We habe encountered a problem with IDL and NT, where IDL is only able to
allocate 800 MB of memory in one step, even if it can allocate more
memory afterwards. But 256 MB is much too low, presuming there is enough
physical+virtual memory.
In Unix, I can allocate more than 1 GB:
--> idl
IDL Version 5.2.1 (sunos sparc). (c) 1999, Research Systems, Inc.
Installation number: 80864-0.
Licensed for use by: MPI Forschung
IDL> b=bytarr(1024,1024,1200)
IDL> help, /mem
heap memory used: 1258497902, max: 1258497930, gets: 227,
frees: 89
The limit command tells you about the datasize you are allowed to
allocate, maybe this is set too low for you. And maybe there's a similar
setting for Windows?
Alex
--
Alex Schuster Wonko@weird.cologne.de PGP Key available
alex@pet.mpin-koeln.mpg.de
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
