| Re: VARRAY, memory & extracting subarrays [message #25597] |
Mon, 02 July 2001 17:23 |
Kristine Hensel
Messages: 26
Registered: June 1999
|
Junior Member |
|
|
"Richard G. French" wrote:
>
> OOPS! Yet another typo - that's what I get for coding on the fly....
That's okay - I'll just wait here patiently until you get it right! ;)
Thanks for your suggestion - I'll certainly consider keeping my images
separate. We were heading that direction anyway.
Kristine
--
Kristine Hensel
Environmental Systems & Services Phone: +61-3-9835-7901
20 Council St.
Hawthorn East, VIC 3123 Australia e-mail: kristine@esands.com
|
|
|
|
| Re: VARRAY, memory & extracting subarrays [message #25601 is a reply to message #25597] |
Mon, 02 July 2001 05:29  |
Richard French
Messages: 173
Registered: December 2000
|
Senior Member |
|
|
OOPS! Yet another typo - that's what I get for coding on the fly....
>
> That should be:
>
>>
>> openr,lun,/GET_LUN,image_file
>> image=ASSOC(lun,fltarr(500,600))
> ^^^^
>>
>> sub_array=fltarr(201,201,4)
>>
>> for i=11,14 do begin
>> sub_array[0,0,i-11]=(image[i])[100:300,200:400]
^^^^
>> endfor
|
|
|
|
| Re: VARRAY, memory & extracting subarrays [message #25602 is a reply to message #25601] |
Mon, 02 July 2001 05:28 |
Richard French
Messages: 173
Registered: December 2000
|
Senior Member |
|
|
That should be:
>
> openr,lun,/GET_LUN,image_file
> image=ASSOC(lun,fltarr(500,600))
^^^^
>
> sub_array=fltarr(201,201,4)
>
> for i=11,14 do begin
> sub_array[0,0,i]=(image[i])[100:300,200:400]
> endfor
|
|
|
|
| Re: VARRAY, memory & extracting subarrays [message #25603 is a reply to message #25602] |
Mon, 02 July 2001 05:27 |
Richard French
Messages: 173
Registered: December 2000
|
Senior Member |
|
|
You've probably thought about this, but it appears that your images
array is a stack of images (a 3-D array) - if you can get away with
processing a single image at a time, or extracting a subsection of
each image in succession, you can always use the ASSOC command to
set up an associated variable. This is very handy when you are working
with a huge number of individual images. For example, let's say
you had 20 images of size 500 x 600:
images = fltarr(500,600,20)
and you wanted to make a sub=array of (100:300, 200:400, 11:14) of
this hunk. You could do:
openr,lun,/GET_LUN,image_file
image=ASSOC(fltarr(500,600))
sub_array=fltarr(201,201,4)
for i=11,14 do begin
sub_array[0,0,i]=(image[i])[100:300,200:400]
endfor
THis way, you never need to have the entire large image cube in
memory at a given time.
I do this all the time for sequences of astronomical images which
are stored in time order in an image cube. For your application, it
may or may not be a time-saver.
Hope this helps,
Dick French
|
|
|
|
| Re: VARRAY, memory & extracting subarrays [message #25607 is a reply to message #25603] |
Mon, 02 July 2001 00:38 |
Kristine Hensel
Messages: 26
Registered: June 1999
|
Junior Member |
|
|
Craig Markwardt wrote:
> VARRAY is a pretty extreme measure for your needs.
Not to mention that it's too much for my very small brain ...
>
> 1. Can you increase your swap space?
> 2. Have you checked your process limits (ie, "limit" or "ulimit"
> command before running IDL)?
We've tried our best with these. We're working with HP-UX's, and there
isn't really a resident expert around. The swap space and process
limits both seem to be very big numbers.
> 3. Big problem is that you say "sector_images = images", which deletes
> the old mapping of sector_images. You can do this instead:
> sector_images(*,*,*) = images ; or even better,
> sector_images(0,0,0) = images ; which is faster but sneaky
I *knew* it'd be something simple like that - thanks! (And thanks for
being up so late!)
Kristine
--
Kristine Hensel
Environmental Systems & Services Phone: +61-3-9835-7901
20 Council St.
Hawthorn East, VIC 3123 Australia e-mail: kristine@esands.com
|
|
|
|
| Re: VARRAY, memory & extracting subarrays [message #25608 is a reply to message #25607] |
Sun, 01 July 2001 22:10 |
Craig Markwardt
Messages: 1869
Registered: November 1996
|
Senior Member |
|
|
Kristine Hensel <kristine@esands.com> writes:
> I'm processing large (105 MB) arrays of images, and I've been running
> into memory problems. (Not surprisingly, right?) I've started using
> Eric Korpela's VARRAY routine, which has helped, but I still can't
> manage to extract a subarray without using all of the available memory.
> Theoretically I have 1 GB of memory, and we've tried maximizing every
> system variable that we can, but I'm still crashing ("Unable to allocate
> memory to create array") when I try to run my image processing program.
...
VARRAY is a pretty extreme measure for your needs. Avoid it if you
can. [ Although I admit Korpela's routine is *very* cool! ]
1. Can you increase your swap space?
2. Have you checked your process limits (ie, "limit" or "ulimit"
command before running IDL)?
3. Big problem is that you say "sector_images = images", which deletes
the old mapping of sector_images. You can do this instead:
sector_images(*,*,*) = images ; or even better,
sector_images(0,0,0) = images ; which is faster but sneaky
4. Investigate chunking or banding.
Hope these help! I would really bet on number 2 though.
Craig
--
------------------------------------------------------------ --------------
Craig B. Markwardt, Ph.D. EMAIL: craigmnet@cow.physics.wisc.edu
Astrophysics, IDL, Finance, Derivatives | Remove "net" for better response
------------------------------------------------------------ --------------
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
