| Re: Large Arrays in IDL [message #5533] |
Mon, 15 January 1996 00:00 |
hahn
Messages: 108
Registered: November 1993
|
Senior Member |
|
|
Eric Deutsch <deutsch@astro.washington.edu> wrote:
[several paragraphs deleted]
> IRAF was designed to be able
> to work with large image arrays without much memory. You can write IDL
> routines to do this, too, but it is usually much simpler to give your
> computer a huge chunk of virtual memory, with 1 GB disks costing well under
> $300 these days...
IDL can work with arrays directly on disk by using the Associate
Variable. You open a file as you would for normal input or output
and call the IDL function ASSOC to combine the i/o unit number,
an array of needed type and size and get a variable as a result
of the function call, the associate variable. Each time the associate
variable appears on the right side of an IDL expression or as a
parameter for TV a chunk of the array is read from disk and processed.
Each time the associate variable appears on the left side of an
expression the file on disk will be written.
The was some discussion on memory usage. If I'm not mistaken
the amount of virtual storage you can use with any MS Windows 3.1
application is 2.5 times the RAM of your computer. This may no longer
be true for Windows 95, I couldn't check it.
> Eric
> --
> ------------------------------------------------------------ ----------------
> Eric Deutsch email: deutsch@astro.washington.edu
> Department of Astronomy Voice: (206) 616-2788
> University of Washington FAX: (206) 685-0403
> Box 351580 WWW: http://www.astro.washington.edu/deutsch
> Seattle, WA 98195-1580 Physics/Astronomy Bldg., Room B356F
Norbert Hahn
|
|
|
|
| Re: Large Arrays in IDL [message #5539 is a reply to message #5533] |
Sat, 13 January 1996 00:00  |
Eric Deutsch
Messages: 11
Registered: May 1995
|
Junior Member |
|
|
Saeid Zoonematkermani wrote:
>
> This is a question that concerns astronomers once in a while. I have
> wondered what the limit is to the size of the arrays that IDL can create.
> Some times IDL fails to read a FITS image complaining:
>
> % Unable to allocate memory: to make array.
> not enough core
>
> This is not so uncommon if one has to deal with even moderate size cubes.
> I have tried to use .SIZE to increase the default value for data area but
> it doesn't seem to work. So the question is if there is a way to open a
> very large image by IDL. IRAF seems to have no problem with this. Does
> this problem arise from lack of RAM on the computer?
The "not enought core" error appears when you try to create an array for
which there is not enough room in all of the memory space of the machine
you're working on. For most machines, this is the total virtual memory
available, which is not usually related to the amount of physical RAM
installed in the box. You don't mention what platform you use, so I
can't tell you how to check it exactly. On SunOS machines, try 'pstat -T'
to get 421092/601888 swap which indicates 421MB of possible 601MB in use.
On Solaris, try vmstat. On a VMS machine, you can try 'show mem'. There
is an added complication on VMS that individual processes are usually given
a certain working set limit, so even if the machine has plenty of virtual
memory, you may be limited to using only a certain portion. To use large
arrays in IDL, you'll need to increase the amount of virtual memory
available to you (either your working set or the whole machine's available
memory). This usually just involves setting aside more disk space for
virtual memory purposes, not buying more RAM chips. It usually isn't
very hard.
More efficient programming can help out, too. You can make use of the
IDL TEMPORARY() function or other tricks to avoid many copies of large
arrays. Or, you can do things the way IRAF does things: IRAF never really
holds image arrays in memory. Instead of loading image A and loading image
B and adding them and then writing out image C (which requires memory space
for three arrays plus temporary storage space), IRAF usually does a piece-
by-piece approach, i.e. in a loop, read small chunks of A and B,
add, and then write out the result to C. It never reads the whole
multi-megabyte images at once, but rather makes heavy use of temporary
disk images and works on parts of files. IRAF was designed to be able
to work with large image arrays without much memory. You can write IDL
routines to do this, too, but it is usually much simpler to give your
computer a huge chunk of virtual memory, with 1 GB disks costing well under
$300 these days...
Eric
--
------------------------------------------------------------ ----------------
Eric Deutsch email: deutsch@astro.washington.edu
Department of Astronomy Voice: (206) 616-2788
University of Washington FAX: (206) 685-0403
Box 351580 WWW: http://www.astro.washington.edu/deutsch
Seattle, WA 98195-1580 Physics/Astronomy Bldg., Room B356F
|
|
|
|
| Re: Large Arrays in IDL [message #5541 is a reply to message #5539] |
Sat, 13 January 1996 00:00 |
thompson
Messages: 584
Registered: August 1991
|
Senior Member |
|
|
szoonem@astro.sunysb.edu (Saeid Zoonematkermani) writes:
> Hi;
> This is a question that concerns astronomers once in a while. I have
> wondered what the limit is to the size of the arrays that IDL can create.
> Some times IDL fails to read a FITS image complaining:
> % Unable to allocate memory: to make array.
> not enough core
> This is not so uncommon if one has to deal with even moderate size cubes.
> I have tried to use .SIZE to increase the default value for data area but
> it doesn't seem to work. So the question is if there is a way to open a
> very large image by IDL. IRAF seems to have no problem with this. Does
> this problem arise from lack of RAM on the computer?
The .SIZE command doesn't have anything to do with the size of arrays that you
can handle in IDL. It's more related to how many variables (regardless of
size) that you have, and how many routines you have.
The size of arrays is controlled by how much RAM you have. On VMS computers,
it's also controlled by various system and process quotas--talk to your system
manager. If you can't allocate space for arrays, try deleting any arrays you
may already have in memory.
Another reason why you may be having trouble allocating array space is because
of memory fragmentation. For more information, read the FAQ.
As I understand it, IRAF avoids this problem by doing everything on disk. That
way, it only needs to have enough memory for one array at a time.
William Thompson
|
|
|
|
| Re: Large Arrays in IDL [message #5546 is a reply to message #5539] |
Fri, 12 January 1996 00:00 |
kspencer
Messages: 21
Registered: December 1993
|
Junior Member |
|
|
szoonem@astro.sunysb.edu (Saeid Zoonematkermani) writes:
> Hi;
> This is a question that concerns astronomers once in a while. I have
> wondered what the limit is to the size of the arrays that IDL can create.
> Some times IDL fails to read a FITS image complaining:
> % Unable to allocate memory: to make array.
> not enough core
> This is not so uncommon if one has to deal with even moderate size cubes.
> I have tried to use .SIZE to increase the default value for data area but
> it doesn't seem to work. So the question is if there is a way to open a
> very large image by IDL. IRAF seems to have no problem with this. Does
> this problem arise from lack of RAM on the computer?
I don't do astronomy and don't know about the FITS format, but the only
memory limitation for IDL is the total amount of RAM (physical and virtual)
on your system. Of course, if that memory space becomes fragmented, it will
seem like you have a lot less.
Kevin
-----------------------------------------------------------
Kevin Spencer
Cognitive Psychophysiology Laboratory and Beckman Institute
University of Illinois at Urbana-Champaign
kspencer@p300.cpl.uiuc.edu / kspencer@psych.uiuc.edu
-----------------------------------------------------------
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
