comp.lang.idl-pvwave archive: archive » Re: Dealing with Large data arrays, reducing memory and ASSOC

Home » Public Forums » archive » Re: Dealing with Large data arrays, reducing memory and ASSOC

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: Dealing with Large data arrays, reducing memory and ASSOC [message #54459 is a reply to message #54458]

Thu, 14 June 2007 11:52

Kenneth Bowman
Messages: 86
Registered: November 2006

Member

In article <1181828486.257277.182530@q19g2000prn.googlegroups.com>,
bill.dman@gmail.com wrote:

> On Jun 14, 8:33 am, Ambrosia_Everlovely
> <ambrosia_everlov...@hotmail.com> wrote:
>> Hi,
>> I have a fairly large datacube, DC(x,y,t)=DC(512,512,2048) and I want
>> to perform an FFT in the t direction. Now I can do,
>> FFTDC=fft(DC,-1,dim=3) which takes an excessive amount of memory (19 G
>> + 50 G virtual) and slows the whole system down.
>> Since this must be a fairly common practice amongst astronomers, can
>> anyone provide - or link to - a small IDL algorithm which will allow
>> me to use ASSOC or reduce the memory in some way? I have also tried
>> TEMPORARY, but this doesn't seem to help at all.
>>
>> Thankyou!!!!
>
> Assuming you are using single precision, you can limit memory needed
> to about 6GB with
>
> fftdc = complexarr(512,512,2048)
> for i=0,511 do for j=0,511 do fftdc[i,j,0] = fft(dc[i,j,*],-1)
>
> this should help if your machine has more than 6GB for you to use.

I don't think this will work as written. The trick of zero-subscripting
on the LHS of an assignment works for the leading dimensions only.

IDL> x = findgen(4,4)
IDL> print, x
0.00000 1.00000 2.00000 3.00000
4.00000 5.00000 6.00000 7.00000
8.00000 9.00000 10.0000 11.0000
12.0000 13.0000 14.0000 15.0000
IDL> x[0,2] = replicate(99.0, 4)
IDL> print, x
0.00000 1.00000 2.00000 3.00000
4.00000 5.00000 6.00000 7.00000
99.0000 99.0000 99.0000 99.0000
12.0000 13.0000 14.0000 15.0000

If you try this with a trailing dimension you get this

IDL> x = findgen(4,4)
IDL> x[2,0] = replicate(99.0, 4)
% Out of range subscript encountered: X.
% Execution halted at: $MAIN$

To make your expression work, you would have to write

fftdc[i,j,*] = fft(dc[i,j,*],-1)

which results in some performance penalty.

Ken Bowman

Report message to a moderator

[Message index]

		Re: Dealing with Large data arrays, reducing memory and ASSOC By: Kenneth Bowman on Fri, 15 June 2007 12:54
		Re: Dealing with Large data arrays, reducing memory and ASSOC By: bill.dman on Fri, 15 June 2007 11:40
		Re: Dealing with Large data arrays, reducing memory and ASSOC By: Ambrosia_Everlovely on Fri, 15 June 2007 00:26
		Re: Dealing with Large data arrays, reducing memory and ASSOC By: Kenneth Bowman on Thu, 14 June 2007 11:52
		Re: Dealing with Large data arrays, reducing memory and ASSOC By: Kenneth Bowman on Thu, 14 June 2007 11:50
		Re: Dealing with Large data arrays, reducing memory and ASSOC By: JD Smith on Thu, 14 June 2007 09:32
		Re: Dealing with Large data arrays, reducing memory and ASSOC By: Haje Korth on Thu, 14 June 2007 09:18
		Re: Dealing with Large data arrays, reducing memory and ASSOC By: bill.dman on Thu, 14 June 2007 06:41
		Re: Dealing with Large data arrays, reducing memory and ASSOC By: Kenneth Bowman on Thu, 14 June 2007 06:08

Previous Topic:	Read particular band from air temperature image among MOD07 air products
Next Topic:	Yikes! Time for a blow-out party!

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Wed Oct 08 19:21:27 PDT 2025

Total time taken to generate the page: 0.00428 seconds