comp.lang.idl-pvwave archive - RDF feed
type: channel
description: Messages from Usenet group comp.lang.idl-pvwave, compiled by Paulo Penteado
items: Adding x,y events to a 2d array (quickly)
items: Re: Adding x,y events to a 2d array (quickly)

Adding x,y events to a 2d array (quickly)
type: item
description: Hi This may be a much answered question, but searching for an answer has failed me. I have 3 (very large) arrays giving x values, y values and energy values. I wish to create two 2d arrays - one of total (summed) energy for a particular x,y value, and one of total counts per x,y value. An example of what I tried is below: x=[1,1,2] y=[1,1,2] e=[10,10,10] To create the 'counts' value, i used the following: counts=fltarr(5,5) counts(x,y)++ This works. You end up with a value of 2 at position(1,1) and a value of 1 at position (2,2). I hoped to get the 'total energy' value by doing the following: totalenergy=fltarr(5,5) totalenergy(x,y)+=e However, this does not work. The final array only contains the last energy value added at each point. Is there an IDL trick I'm missing that allows you to incrementally add values to an array quickly? Thanks Oliver
creator: oliver[1]
date: 2013-11-07T12:45:37-00:00

Re: Adding x,y events to a 2d array (quickly)
type: item
description: I read your post several times, and I guess I'm not sure what you're after. That said, I've got a good hunch that you're going to need histogram and the reverse_indices output. If so, then you're actually gonna want hist_nd by JD Smith. -Russell
creator: Russell Ryan
date: 2013-11-07T14:10:10-00:00

Re: Adding x,y events to a 2d array (quickly)
type: item
description: Oliver, You have a good question, and I think this code illustrates it a little more plainly, starting each time with an array of zero values: counts=fltarr(3,3) counts[[1,1,2],[1,1,2]] ++ Print, 'counts[[1,1,2],[1,1,2]] ++:' Print, counts counts=fltarr(3,3) counts[[1,1,2],[1,1,2]] += 1 Print, 'counts[[1,1,2],[1,1,2]] += 1:' Print, counts counts=fltarr(3,3) counts[[1,1,2],[1,1,2]] += [1,1,1] Print, 'counts[[1,1,2],[1,1,2]] += [1,1,1]:' Print, counts counts=fltarr(3,3) counts[[1,1,2],[1,1,2]] += [10,20,30] Print, 'counts[[1,1,2],[1,1,2]] += [10,20,30]:' Print, counts The result of this is: counts[[1,1,2],[1,1,2]] ++: 0.000000 0.000000 0.000000 0.000000 2.00000 0.000000 0.000000 0.000000 1.00000 counts[[1,1,2],[1,1,2]] += 1: 0.000000 0.000000 0.000000 0.000000 1.00000 0.000000 0.000000 0.000000 1.00000 counts[[1,1,2],[1,1,2]] += [1,1,1]: 0.000000 0.000000 0.000000 0.000000 1.00000 0.000000 0.000000 0.000000 1.00000 counts[[1,1,2],[1,1,2]] += [10,20,30]: 0.000000 0.000000 0.000000 0.000000 20.0000 0.000000 0.000000 0.000000 30.0000 It seems that ++ increments for each (x,y) pair as you expect. However, the += operation seems to be creating a set of result values by adding the set of original values to the given scalar or vector, and then copying the results into the array. In this way, when [1,1] is assigned values twice by this copying, only the last value persists. I seem to recall someone explaining this behaviour before, and thanks to Russell, I realize one good way of getting *part* of what you (reasonably!) want to do. If all of your 'e' values were equal, then you can find how many counts of each (x,y) pair exist by using Hist_ND: (http://tir.astro.utoledo.edu/idl/hist_nd.pro) IDL> Print, Hist_ND(Transpose([[1,1,2],[1,1,2]]), 1, Min=0) 0 0 0 0 2 0 0 0 1 But, in general, to add a varying set of 'e' values to those (x,y) locations... I have to think a bit... Cheers, -Dick Dick Jackson Software Consulting Victoria, BC, Canada www.d-jackson.com oliver wrote, On 2013…
creator: Dick Jackson
date: 2013-11-07T19:27:00-00:00

Re: Adding x,y events to a 2d array (quickly)
type: item
description: On Thursday, November 7, 2013 1:27:00 PM UTC-6, Dick Jackson wrote: > I seem to recall someone explaining this behaviour before, and thanks to > > Russell, I realize one good way of getting *part* of what you (reasonably!) want > > to do. If all of your 'e' values were equal, then you can find how many counts > > of each (x,y) pair exist by using Hist_ND: > > (http://tir.astro.utoledo.edu/idl/hist_nd.pro) > > IDL> Print, Hist_ND(Transpose([[1,1,2],[1,1,2]]), 1, Min=0) > > But, in general, to add a varying set of 'e' values to those (x,y) locations... > > I have to think a bit... > I've got you covered.... Oliver, reverse indices are your friend here, as Russell alluded to. Get the two-dimensional histogram, slightly modified from Dick's version: h = HIST_ND( [ TRANSPOSE(x), TRANSPOSE(y) ], 1, MIN=0, REVERSE_INDICES=ri ) Since you said you have large arrays, I transpose each individually, and then concatenate. Now, go through the reverse indices: totalE = FLTARR(SIZE(h, /DIM)) FOR i=0, N_ELEMENTS(h)-1 do if h[i] GT 0 THEN totalE[i]= TOTAL( e[ri[ri[i]:ri[i+1]-1]]) print, totalE 0.00000 0.00000 0.00000 0.00000 20.0000 0.00000 0.00000 0.00000 10.0000 This is the basic idea. It can be sped up by only looping over the elements of h with non-zero counts (as opposed to "skipping" them as I did here). Here's some highly recommended reading on histograms: http://www.idlcoyote.com/tips/histogram_tutorial.html
creator: Phillip Bitzer
date: 2013-11-07T20:16:48-00:00

Re: Adding x,y events to a 2d array (quickly)
type: item
description: Phillip Bitzer wrote, On 2013-11-07, 12:16pm: > On Thursday, November 7, 2013 1:27:00 PM UTC-6, Dick Jackson wrote: > >> I seem to recall someone explaining this behaviour before, and thanks to >> >> Russell, I realize one good way of getting *part* of what you (reasonably!) want >> >> to do. If all of your 'e' values were equal, then you can find how many counts >> >> of each (x,y) pair exist by using Hist_ND: >> >> (http://tir.astro.utoledo.edu/idl/hist_nd.pro) >> >> IDL> Print, Hist_ND(Transpose([[1,1,2],[1,1,2]]), 1, Min=0) >> >> But, in general, to add a varying set of 'e' values to those (x,y) locations... >> >> I have to think a bit... >> > > I've got you covered.... > > Oliver, reverse indices are your friend here, as Russell alluded to. Get the two-dimensional histogram, slightly modified from Dick's version: > > h = HIST_ND( [ TRANSPOSE(x), TRANSPOSE(y) ], 1, MIN=0, REVERSE_INDICES=ri ) > > Since you said you have large arrays, I transpose each individually, and then concatenate. > > Now, go through the reverse indices: > > totalE = FLTARR(SIZE(h, /DIM)) > FOR i=0, N_ELEMENTS(h)-1 do if h[i] GT 0 THEN totalE[i]= TOTAL( e[ri[ri[i]:ri[i+1]-1]]) > > print, totalE > 0.00000 0.00000 0.00000 > 0.00000 20.0000 0.00000 > 0.00000 0.00000 10.0000 > > This is the basic idea. It can be sped up by only looping over the elements of h with non-zero counts (as opposed to "skipping" them as I did here). > > Here's some highly recommended reading on histograms: http://www.idlcoyote.com/tips/histogram_tutorial.html Histograms and reverse-indices are amazingly powerful and the right way to go in many tough problems, but I think Oliver is looking for a solution avoiding loops (I am too!). If a loop solution were OK, the last block here would be more direct, with no need for histograms: x=[1,1,2] y=[1,1,2] e=[10,11,12] counts=fltarr(3,3) counts(x,y)++ Print, 'counts:' Print, counts ; Shows that three increments by 1 were done totalenergy=fltarr(3,3) totalenergy(x,y)+=e Print, '…
creator: Dick Jackson
date: 2013-11-07T22:20:15-00:00

Re: Adding x,y events to a 2d array (quickly)
type: item
description: On 11/7/13, 3:20 PM, Dick Jackson wrote: > Phillip Bitzer wrote, On 2013-11-07, 12:16pm: >> On Thursday, November 7, 2013 1:27:00 PM UTC-6, Dick Jackson wrote: >> >>> I seem to recall someone explaining this behaviour before, and thanks to >>> >>> Russell, I realize one good way of getting *part* of what you >>> (reasonably!) want >>> >>> to do. If all of your 'e' values were equal, then you can find how >>> many counts >>> >>> of each (x,y) pair exist by using Hist_ND: >>> >>> (http://tir.astro.utoledo.edu/idl/hist_nd.pro) >>> >>> IDL> Print, Hist_ND(Transpose([[1,1,2],[1,1,2]]), 1, Min=0) >>> >>> But, in general, to add a varying set of 'e' values to those (x,y) >>> locations... >>> >>> I have to think a bit... >>> >> >> I've got you covered.... >> >> Oliver, reverse indices are your friend here, as Russell alluded to. >> Get the two-dimensional histogram, slightly modified from Dick's version: >> >> h = HIST_ND( [ TRANSPOSE(x), TRANSPOSE(y) ], 1, MIN=0, >> REVERSE_INDICES=ri ) >> >> Since you said you have large arrays, I transpose each individually, >> and then concatenate. >> >> Now, go through the reverse indices: >> >> totalE = FLTARR(SIZE(h, /DIM)) >> FOR i=0, N_ELEMENTS(h)-1 do if h[i] GT 0 THEN totalE[i]= TOTAL( >> e[ri[ri[i]:ri[i+1]-1]]) >> >> print, totalE >> 0.00000 0.00000 0.00000 >> 0.00000 20.0000 0.00000 >> 0.00000 0.00000 10.0000 >> >> This is the basic idea. It can be sped up by only looping over the >> elements of h with non-zero counts (as opposed to "skipping" them as I >> did here). >> >> Here's some highly recommended reading on histograms: >> http://www.idlcoyote.com/tips/histogram_tutorial.html > > Histograms and reverse-indices are amazingly powerful and the right way > to go in many tough problems, but I think Oliver is looking for a > solution avoiding loops (I am too!). If a loop solution were OK, the > last block here would be more direct, with no need for histograms: > > x=[1,1,2] > y=[1,1,2] > e=[10,11,12] > > counts=fltarr(3,3) > co…
creator: Michael Galloy
date: 2013-11-07T23:18:19-00:00

Re: Adding x,y events to a 2d array (quickly)
type: item
description: Michael Galloy wrote, On 2013-11-07, 3:18pm: > On 11/7/13, 3:20 PM, Dick Jackson wrote: >> Phillip Bitzer wrote, On 2013-11-07, 12:16pm: >>> On Thursday, November 7, 2013 1:27:00 PM UTC-6, Dick Jackson wrote: >>> >>>> I seem to recall someone explaining this behaviour before, and thanks to >>>> >>>> Russell, I realize one good way of getting *part* of what you >>>> (reasonably!) want >>>> >>>> to do. If all of your 'e' values were equal, then you can find how >>>> many counts >>>> >>>> of each (x,y) pair exist by using Hist_ND: >>>> >>>> (http://tir.astro.utoledo.edu/idl/hist_nd.pro) >>>> >>>> IDL> Print, Hist_ND(Transpose([[1,1,2],[1,1,2]]), 1, Min=0) >>>> >>>> But, in general, to add a varying set of 'e' values to those (x,y) >>>> locations... >>>> >>>> I have to think a bit... >>>> >>> >>> I've got you covered.... >>> >>> Oliver, reverse indices are your friend here, as Russell alluded to. >>> Get the two-dimensional histogram, slightly modified from Dick's version: >>> >>> h = HIST_ND( [ TRANSPOSE(x), TRANSPOSE(y) ], 1, MIN=0, >>> REVERSE_INDICES=ri ) >>> >>> Since you said you have large arrays, I transpose each individually, >>> and then concatenate. >>> >>> Now, go through the reverse indices: >>> >>> totalE = FLTARR(SIZE(h, /DIM)) >>> FOR i=0, N_ELEMENTS(h)-1 do if h[i] GT 0 THEN totalE[i]= TOTAL( >>> e[ri[ri[i]:ri[i+1]-1]]) >>> >>> print, totalE >>> 0.00000 0.00000 0.00000 >>> 0.00000 20.0000 0.00000 >>> 0.00000 0.00000 10.0000 >>> >>> This is the basic idea. It can be sped up by only looping over the >>> elements of h with non-zero counts (as opposed to "skipping" them as I >>> did here). >>> >>> Here's some highly recommended reading on histograms: >>> http://www.idlcoyote.com/tips/histogram_tutorial.html >> >> Histograms and reverse-indices are amazingly powerful and the right way >> to go in many tough problems, but I think Oliver is looking for a >> solution avoiding loops (I am too!). If a loop solution were OK, the >> last block here would be more di…
creator: Dick Jackson
date: 2013-11-08T08:22:22-00:00

Re: Adding x,y events to a 2d array (quickly)
type: item
description: Thanks for all the replies - need time to digest last one, thanks! Dick was right in that I was trying to avoid loops. I get the required result using foreach element,EnergyArray,index DO Image(XArray(index ),YArray(index))+=element but it takes ~30 seconds for the array size I am using. Image(XArray,YArray)+=EnergyArray takes only 1 second to run, but doesn't give the result I expected. As I said, I haven't fully gone through Dick's last message - I just wanted to say thanks for efforts! Oliver
creator: oliver[1]
date: 2013-11-08T09:19:11-00:00

Re: Adding x,y events to a 2d array (quickly)
type: item
description: Hi again - thanks for all replies especially Dick's last with all the timings, which allows me to sheepishly admit that I solved the speed problem, but not how I expected to! Looking at the timings, which I could match in the test program but not in my main data program, it turns out that the single loop over the array contents was taking much longer as the array itself was buried in a structure. Creating a temporary array and looping over that increased the speed from ~30 seconds to ~2 seconds The red herring was that using the non looping method, the fact that it was in a structure hadn't affected the speed... (Although I stand by original message that the += operator doesn't work as you might expect with arrays!) Thanks again Oliver
creator: oliver[1]
date: 2013-11-08T14:18:26-00:00

Re: Adding x,y events to a 2d array (quickly)
type: item
description: Oliver wrote, On 2013-11-08, 6:18am: > Hi again - thanks for all replies especially Dick's last with all the timings, which allows me to sheepishly admit that I solved the speed problem, but not how I expected to! > > Looking at the timings, which I could match in the test program but not in my main data program, it turns out that the single loop over the array contents was taking much longer as the array itself was buried in a structure. > > Creating a temporary array and looping over that increased the speed from ~30 seconds to ~2 seconds > > The red herring was that using the non looping method, the fact that it was in a structure hadn't affected the speed... > > (Although I stand by original message that the += operator doesn't work as you might expect with arrays!) > > Thanks again > > Oliver You're most welcome. It is indeed counterintuitive that array[indicesWithDuplicates] ++ is not equivalent to array[indicesWithDuplicates] += 1 -- Cheers, -Dick Dick Jackson Software Consulting Victoria, BC, Canada www.d-jackson.com
creator: Dick Jackson
date: 2013-11-08T16:11:43-00:00