| Big arrays, reducing data [message #53095] |
Wed, 21 March 2007 10:15  |
Eric Hudson
Messages: 19
Registered: June 2006
|
Junior Member |
|
|
Hi,
I have what I hope is an easy question and then probably a hard one.
1) I need to make some big arrays (ideally 16000^2 elements or more)
but find I often get "unable to allocate memory" errors. Is there
some way of determining (at run time) the largest array that I can
make? In C, for example, I'd try to allocate the memory and check for
whether it was allocated, then cut the array size if it wasn't. Is
there an equivalent technique in IDL?
2) The reason I want to make these big arrays is that I have sets of
on the order of 20,000 data curves (50-200 pts each). I'd like to
reduce these to a set of "common curves" -- around 100 averaged/
extracted/smoothed curves which are representative of the larger set.
The curves are complex -- I don't have anything to fit to them -- and
they are noisy. But I get the feeling that if I handed someone a
stack of 20,000 of them and said "sort these into groups which are
similar" that they'd be able to do it. The question is, is there a
good way to do this programmatically?
My original thought was to calculate an rms difference between each
pair of curves and then reduce by partnering up the curves which are
good matches to each other. Repeat. To do this easily seems to
require big arrays and a pretty large amount of computation time. I'd
appreciate any thoughts.
Thanks,
Eric
|
|
|
|
| Re: Big array [message #83062 is a reply to message #53095] |
Tue, 05 February 2013 04:35  |
xqinshan
Messages: 21
Registered: November 2008
|
Junior Member |
|
|
在 2013年2月4日星期一UTC+8下午10时31分36秒，Matthew 写道：
>> like 'clear' in Matlab?
>
>
>
> "delvar" works but only from the command line or from within main-level programs. If you are executing a main level program, execution will halt when you hit the delvar statement. Use "IDL> help" to see which variables are "in the workspace".
Yes,delvar could stop the program. If you know the array to be swap out such as big array A, it's better set A=0. I'd like to swap out all the memory before a program start to run. I usually use 'clear' at the beginning of my matlab program.
|
|
|
|
| Re: Big array [message #83063 is a reply to message #53095] |
Tue, 05 February 2013 02:15 |
d.poreh
Messages: 406
Registered: October 2007
|
Senior Member |
|
|
On Tuesday, February 5, 2013 9:45:44 AM UTC+1, Yngvar Larsen wrote:
> On Monday, 4 February 2013 15:34:41 UTC+1, alx wrote:
>
>> Le lundi 4 février 2013 15:11:19 UTC+1, David Fanning a écrit :
>
>>
>
>>>
>
>>
>
>>> You could try A = scale * Temporary(A).
>
>>
>
>> An equivalent shorthand since IDL8 is:
>
>>
>
>> A *= scale
>
>
>
> Since IDL 6.0, actually.
>
>
>
> --
>
> Yngvar
What about CONGRID with a big scale like 100 or more...
|
|
|
|
| Re: Big array [message #83064 is a reply to message #53095] |
Tue, 05 February 2013 00:45 |
Yngvar Larsen
Messages: 134
Registered: January 2010
|
Senior Member |
|
|
On Monday, 4 February 2013 15:34:41 UTC+1, alx wrote:
> Le lundi 4 février 2013 15:11:19 UTC+1, David Fanning a écrit :
>
>>
>
>> You could try A = scale * Temporary(A).
>
> An equivalent shorthand since IDL8 is:
>
> A *= scale
Since IDL 6.0, actually.
--
Yngvar
|
|
|
|
| Re: Big array [message #83070 is a reply to message #53095] |
Mon, 04 February 2013 15:57 |
xqinshan
Messages: 21
Registered: November 2008
|
Junior Member |
|
|
This is the first time I see the message 'Array has too many elements. ' A=scale*Temporary(A) seems to work for 'unable to allocate ...'.
I have to divide the array into several segments to be sav file for later use. What is the maximum elements in a IDL array? Is that related to the system (32bit 64bit)?
|
|
|
|
| Re: Big array [message #83078 is a reply to message #53095] |
Mon, 04 February 2013 06:34 |
lecacheux.alain
Messages: 325
Registered: January 2008
|
Senior Member |
|
|
Le lundi 4 février 2013 15:11:19 UTC+1, David Fanning a écrit :
>
> You could try A = scale * Temporary(A).
>
An equivalent shorthand since IDL8 is:
A *= scale
alain.
|
|
|
|
| Re: Big array [message #83079 is a reply to message #53095] |
Mon, 04 February 2013 06:31 |
Matthew
Messages: 18
Registered: February 2006
|
Junior Member |
|
|
> like 'clear' in Matlab?
"delvar" works but only from the command line or from within main-level programs. If you are executing a main level program, execution will halt when you hit the delvar statement. Use "IDL> help" to see which variables are "in the workspace".
|
|
|
|
| Re: Big array [message #83081 is a reply to message #53095] |
Mon, 04 February 2013 06:11 |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
xqinshan@gmail.com writes:
> Hi, I download a netcdf file and NCDF_VARGET a big array A into my program. When I calculate it as A=scale*A, a message comes out: Array has too many elements.
> I cannot do anything with the data. How to handle this problem?
You could try A = scale * Temporary(A).
> By the way, are there any IDL statements to swap out workspace like
'clear' in Matlab?
You would try using a .reset command.
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.idlcoyote.com/
Sepore ma de ni thue. ("Perhaps thou speakest truth.")
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
