| Array concatenation [message #48962] |
Wed, 07 June 2006 03:41  |
maye
Messages: 29
Registered: June 2006
|
Junior Member |
|
|
Hi folks,
I was searching for an effective way to do this, but can't find anybody
writing about what (as usual) seems to be an obvious problem/task to
me. :)
I don't know how many elements I will collect while scanning a bunch of
images, so I want to use array concatenation to collect the values like
this:
means = [means, currMean]
But for this to compile/run properly, I need mean to exist before.
If I do a
means = 0.
before the loop, I will always have a zero-element in the beginning
that I don't want at plotting time. Of course I could remove it with
means = means[1:*]
but not only does this waste resources, it also becomes cumbersome if I
collect 20 different data values, for that I ALL have to remove the
first value?
Surely there must be a better way?
Please help me to program IDL efficiently! :)
Best regards,
Michael
|
|
|
|
| Re: Array concatenation [message #54848 is a reply to message #48962] |
Tue, 17 July 2007 09:25  |
Vince Hradil
Messages: 574
Registered: December 1999
|
Senior Member |
|
|
On Jul 17, 2:28 am, Paolo_Grigis <pgri...@astro.phys.ethz.ch> wrote:
> Julio wrote:
>> Hi there,
>
>> I have a problem with array concatenation. Suppose:
>
>> A = bytarr(10, 10)
>> B = bytar(10, 5)
>
>> C=[A,B]
>
>> Sure, it doesn't work, because dimensions don't agree. But, I need to
>> join A and B. One possibility is to change the dimensions of array B
>> and fill missing lines with zeros. Is it possible??
>
>> comments welcome,
>> Best,
>> Julio
>
> Just use C=[[A],[B]]
>
> IDL> help,a,b,c
> A BYTE = Array[10, 10]
> B BYTE = Array[10, 5]
> C BYTE = Array[10, 15]
>
> Ciao,
> Paolo
or maybe you want transpose([a,transpose(b)]) ?
|
|
|
|
|
|
| Re: Array concatenation [message #54859 is a reply to message #48962] |
Tue, 17 July 2007 05:11 |
Conor
Messages: 138
Registered: February 2007
|
Senior Member |
|
|
On Jul 17, 3:28 am, Paolo_Grigis <pgri...@astro.phys.ethz.ch> wrote:
> Julio wrote:
>> Hi there,
>
>> I have a problem with array concatenation. Suppose:
>
>> A = bytarr(10, 10)
>> B = bytar(10, 5)
>
>> C=[A,B]
>
>> Sure, it doesn't work, because dimensions don't agree. But, I need to
>> join A and B. One possibility is to change the dimensions of array B
>> and fill missing lines with zeros. Is it possible??
>
>> comments welcome,
>> Best,
>> Julio
>
> Just use C=[[A],[B]]
>
> IDL> help,a,b,c
> A BYTE = Array[10, 10]
> B BYTE = Array[10, 5]
> C BYTE = Array[10, 15]
>
> Ciao,
> Paolo
I would go with Paolo's suggestion, unless you absolutely have to join
in left right, instead of up down. In that case, you could try
something like:
c = [[B],[bytarr(10,5)]]
D = [A,C]
|
|
|
|
| Re: Array concatenation [message #54863 is a reply to message #48962] |
Tue, 17 July 2007 00:28 |
Paolo Grigis
Messages: 171
Registered: December 2003
|
Senior Member |
|
|
Julio wrote:
> Hi there,
>
> I have a problem with array concatenation. Suppose:
>
> A = bytarr(10, 10)
> B = bytar(10, 5)
>
> C=[A,B]
>
> Sure, it doesn't work, because dimensions don't agree. But, I need to
> join A and B. One possibility is to change the dimensions of array B
> and fill missing lines with zeros. Is it possible??
>
> comments welcome,
> Best,
> Julio
>
Just use C=[[A],[B]]
IDL> help,a,b,c
A BYTE = Array[10, 10]
B BYTE = Array[10, 5]
C BYTE = Array[10, 15]
Ciao,
Paolo
|
|
|
|
| Re: Array concatenation [message #54864 is a reply to message #48962] |
Tue, 17 July 2007 00:24 |
Peter Clinch
Messages: 98
Registered: April 1996
|
Member |
|
|
Julio wrote:
> Hi there,
>
> I have a problem with array concatenation. Suppose:
>
> A = bytarr(10, 10)
> B = bytar(10, 5)
>
> C=[A,B]
>
> Sure, it doesn't work, because dimensions don't agree. But, I need to
> join A and B. One possibility is to change the dimensions of array B
> and fill missing lines with zeros. Is it possible??
Not so elegant, but workable, is:
c=bytarr(10,15)
c[*,0:9] = a
c[*,10:14] = b
Pete.
--
Peter Clinch Medical Physics IT Officer
Tel 44 1382 660111 ext. 33637 Univ. of Dundee, Ninewells Hospital
Fax 44 1382 640177 Dundee DD1 9SY Scotland UK
net p.j.clinch@dundee.ac.uk http://www.dundee.ac.uk/~pjclinch/
|
|
|
|
| Re: array concatenation [message #58487 is a reply to message #48962] |
Thu, 31 January 2008 00:43 |
Wox
Messages: 184
Registered: August 2006
|
Senior Member |
|
|
On Wed, 30 Jan 2008 07:56:51 -0800 (PST), "Ryan."
<rchughes@brutus.uwaterloo.ca> wrote:
> Hi All,
>
> I'm trying to add a 1x2 array to the end of an Nx2 array to obtain a (N
> +1)x2 array using the square bracket concatenation trick, but based on
> my limited understanding of it, I can't get it to work. I've looked
> at JD's tutorial on David's website but I still don't get it.
>
> Here is an example of what I want to do:
> IDL> array = [[1,2,3,4],[1,2,3,4]]
> IDL> print, array
> 1 2 3 4
> 1 2 3 4
>
> I want to add another pair to the array to get this:
> 1 2 3 4 5
> 1 2 3 4 5
>
> I want to execute a command similar to this one but I can't figure out
> the correct number of brackets:
> array = [[array], [[5,5]]]
>
> Thanks,
> Ryan.
Yet another possibility:
IDL> array = [[1,2,3,4],[1,2,3,4]]
IDL> array = [array,[[5],[5]]]
IDL> print,array
1 2 3 4 5
1 2 3 4 5
|
|
|
|
| Re: array concatenation [message #58501 is a reply to message #48962] |
Wed, 30 January 2008 08:38 |
Spon
Messages: 178
Registered: September 2007
|
Senior Member |
|
|
On Jan 30, 3:56 pm, "Ryan." <rchug...@brutus.uwaterloo.ca> wrote:
> Hi All,
>
> I'm trying to add a 1x2 array to the end of an Nx2 array to obtain a (N
> +1)x2 array using the square bracket concatenation trick, but based on
> my limited understanding of it, I can't get it to work. I've looked
> at JD's tutorial on David's website but I still don't get it.
>
> Here is an example of what I want to do:
> IDL> array = [[1,2,3,4],[1,2,3,4]]
> IDL> print, array
> 1 2 3 4
> 1 2 3 4
>
> I want to add another pair to the array to get this:
> 1 2 3 4 5
> 1 2 3 4 5
>
> I want to execute a command similar to this one but I can't figure out
> the correct number of brackets:
> array = [[array], [[5,5]]]
>
> Thanks,
> Ryan.
You want:
array = [array, rebin([5,5],1,2)]
It's time to invoke that dimension juggling tutorial again! It's one
of JD's other tutorials on David's site.
http://www.dfanning.com/tips/rebin_magic.html
Thanks,
Chris
|
|
|
|
| Re: array concatenation [message #58502 is a reply to message #48962] |
Wed, 30 January 2008 08:21 |
Allan Whiteford
Messages: 117
Registered: June 2006
|
Senior Member |
|
|
Ryan. wrote:
> Hi All,
>
> I'm trying to add a 1x2 array to the end of an Nx2 array to obtain a (N
> +1)x2 array using the square bracket concatenation trick, but based on
> my limited understanding of it, I can't get it to work. I've looked
> at JD's tutorial on David's website but I still don't get it.
>
> Here is an example of what I want to do:
> IDL> array = [[1,2,3,4],[1,2,3,4]]
> IDL> print, array
> 1 2 3 4
> 1 2 3 4
>
> I want to add another pair to the array to get this:
> 1 2 3 4 5
> 1 2 3 4 5
>
> I want to execute a command similar to this one but I can't figure out
> the correct number of brackets:
> array = [[array], [[5,5]]]
>
> Thanks,
> Ryan.
Ryan,
IDL> array = [[1,2,3,4],[1,2,3,4]]
IDL> array=transpose([[transpose(array)],[[5,5]]])
IDL> print,array
1 2 3 4 5
1 2 3 4 5
but probably there is a smarter way.
Thanks,
Allan
|
|
|
|
| Re: array concatenation [message #62806 is a reply to message #48962] |
Fri, 03 October 2008 18:11 |
Karl[1]
Messages: 79
Registered: October 2005
|
Member |
|
|
On Oct 3, 3:16 pm, lecacheux.al...@wanadoo.fr wrote:
> On 3 oct, 17:15, Joost Aan de Brugh <joost...@gmail.com> wrote:
>
>
>
>> On Oct 3, 1:19 pm, lecacheux.al...@wanadoo.fr wrote:
>
>> Hello,
>
>> Maybe it has something to do with the array descriptor. Anyway, in a
>> large group concatenation is not the most elegant way. In Matlab (a
>> language similar to IDL), you get a warning if you use such a
>> construction. It has to do with the fact that if your array grows, you
>> ask your system for more space. A safer way is to ask for enough space
>> at once.
>
>> afh = a few 100
>
>> b = BytArr(afh*1000) ; Here is where you ask for a lot of space.
>> for i=0,999 do begin
>> ... compute a = array of bytes (a few 100) ...
>> b[i*afh:(i+1)*afh-1] = a ; Now b does not grow in the loop
>> end
>
>> Or use a 2D array
>
>> b = BytArr(afh,1000) ; Here, you ask for the space again.
>> for i=0,999 do begin
>> ... compute a = array of bytes (a few 100) ...
>> b[*,i] = a ; Now b does not grow in the loop
>> end
>
>> Cheers,
>> Joost
>
>> b = Reform(b,afh*1000) ; Or b = Reform(b,N_Elements(b))
>
>> It is a bit harder if you have different 'a few 100's for each
>> iteration
>
> Thanks for your reply. I agree with you that such a programming style
> is far from ideal.
> My point is that it likely can produce some not obvious array boundary
> error (and subsequent IDL crash),
> while largest used array sizes remain far below the maximum authorized
> one.
> Or I missed something ?
> alx.
I dunno, I think that there's something else going on. IDL should
fail gracefully if it runs out of memory allocating that array over
and over, even if you are chewing up space and causing a lot of
fragmentation. How big is the array actually getting when you crash?
I just tried it on linux with an array size of 500 and looping 25,000
times with no issue at all. Are you pushing up against the max virt
mem in your machine? If you are indeed paging, Windows can get a
little unstable under excessive paging.
The COMPILE_OPT IDL2 note is interesting too. Recall that if you
don't specify this, integers are 16-bit, and 32-bit if you do. What
does your code specified by "... compute a = array of bytes (a few
100) ... " do? Does it call a custom DLM? Is there something that
would break or overflow if ints are 16-bit? Still, IDL checks array
accesses at run time, so a bad array index shouldn't cause a crash.
It may be worth taking a closer look at why IDL2 makes a difference.
|
|
|
|
| Re: array concatenation [message #62810 is a reply to message #48962] |
Fri, 03 October 2008 14:20 |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
lecacheux.alain@wanadoo.fr writes:
> Thanks for your reply. I agree with you that such a programming style
> is far from ideal.
> My point is that it likely can produce some not obvious array boundary
> error (and subsequent IDL crash),
> while largest used array sizes remain far below the maximum authorized
> one.
> Or I missed something ?
I think you are missing the extent to which memory
fragmentation reduces the "maximum authorized size"
of an array. In other words, programming with memory
fragmentation issues in mind will, in the end, save
you an enormous amount of unexplained grief.
Cheers,
David
--
David Fanning, Ph.D.
Coyote's Guide to IDL Programming (www.dfanning.com)
Sepore ma de ni thui. ("Perhaps thou speakest truth.")
|
|
|
|
| Re: array concatenation [message #62811 is a reply to message #48962] |
Fri, 03 October 2008 14:16 |
lecacheux.alain
Messages: 325
Registered: January 2008
|
Senior Member |
|
|
On 3 oct, 17:15, Joost Aan de Brugh <joost...@gmail.com> wrote:
> On Oct 3, 1:19 pm, lecacheux.al...@wanadoo.fr wrote:
>
> Hello,
>
> Maybe it has something to do with the array descriptor. Anyway, in a
> large group concatenation is not the most elegant way. In Matlab (a
> language similar to IDL), you get a warning if you use such a
> construction. It has to do with the fact that if your array grows, you
> ask your system for more space. A safer way is to ask for enough space
> at once.
>
> afh = a few 100
>
> b = BytArr(afh*1000) ; Here is where you ask for a lot of space.
> for i=0,999 do begin
> ... compute a = array of bytes (a few 100) ...
> b[i*afh:(i+1)*afh-1] = a ; Now b does not grow in the loop
> end
>
> Or use a 2D array
>
> b = BytArr(afh,1000) ; Here, you ask for the space again.
> for i=0,999 do begin
> ... compute a = array of bytes (a few 100) ...
> b[*,i] = a ; Now b does not grow in the loop
> end
>
> Cheers,
> Joost
>
> b = Reform(b,afh*1000) ; Or b = Reform(b,N_Elements(b))
>
> It is a bit harder if you have different 'a few 100's for each
> iteration
Thanks for your reply. I agree with you that such a programming style
is far from ideal.
My point is that it likely can produce some not obvious array boundary
error (and subsequent IDL crash),
while largest used array sizes remain far below the maximum authorized
one.
Or I missed something ?
alx.
|
|
|
|
| Re: array concatenation [message #62815 is a reply to message #48962] |
Fri, 03 October 2008 08:15 |
Joost Aan de Brugh
Messages: 16
Registered: July 2008
|
Junior Member |
|
|
On Oct 3, 1:19 pm, lecacheux.al...@wanadoo.fr wrote:
> On 3 oct, 13:12, lecacheux.al...@wanadoo.fr wrote:
> I got a strange error when using the array concatenation construct
> within a large loop (a few thousands).
> something like:
> b = 0B
> for i=0,999 do begin
> ... compute a = array of bytes (a few 100) ...
> b = [b, a]
> endfor
> This could crash IDL and even, randomly, crash the system (Win2K).
> Nothing found by catching errors.
> Is there some limits in the implicit addressing of such arrays ?
> The error disappeared when compiling with COMPILE_OPT IDL2.
Hello,
Maybe it has something to do with the array descriptor. Anyway, in a
large group concatenation is not the most elegant way. In Matlab (a
language similar to IDL), you get a warning if you use such a
construction. It has to do with the fact that if your array grows, you
ask your system for more space. A safer way is to ask for enough space
at once.
afh = a few 100
b = BytArr(afh*1000) ; Here is where you ask for a lot of space.
for i=0,999 do begin
... compute a = array of bytes (a few 100) ...
b[i*afh:(i+1)*afh-1] = a ; Now b does not grow in the loop
end
Or use a 2D array
b = BytArr(afh,1000) ; Here, you ask for the space again.
for i=0,999 do begin
... compute a = array of bytes (a few 100) ...
b[*,i] = a ; Now b does not grow in the loop
end
Cheers,
Joost
b = Reform(b,afh*1000) ; Or b = Reform(b,N_Elements(b))
It is a bit harder if you have different 'a few 100's for each
iteration
|
|
|
|
| Re: array concatenation [message #62818 is a reply to message #48962] |
Fri, 03 October 2008 04:19 |
lecacheux.alain
Messages: 325
Registered: January 2008
|
Senior Member |
|
|
On 3 oct, 13:12, lecacheux.al...@wanadoo.fr wrote:
I got a strange error when using the array concatenation construct
within a large loop (a few thousands).
something like:
b = 0B
for i=0,999 do begin
... compute a = array of bytes (a few 100) ...
b = [b, a]
endfor
This could crash IDL and even, randomly, crash the system (Win2K).
Nothing found by catching errors.
Is there some limits in the implicit addressing of such arrays ?
The error disappeared when compiling with COMPILE_OPT IDL2.
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
