| Re: Is there a standard 'null' array? [message #34579] |
Tue, 01 April 2003 11:55 |
JD Smith
Messages: 850
Registered: December 1999
|
Senior Member |
|
|
On Tue, 01 Apr 2003 11:52:40 -0700, James Kuyper wrote:
> JD Smith wrote:
> ...
>> Hmmm, it really wants those lines together. OK, I'll try one more
>> time:
>>
>> nacc=1000 ; or whatever
>>
>> vec=fltarr(nacc,/NOZERO)
>
> All three times you've posted it, the 'whatever' and the 'vec' are
> seperated by two newlines on my system. Maybe it's just your newsreader?
Aha. Right. Silly newsreader.
JD
|
|
|
|
| Re: Is there a standard 'null' array? [message #34580 is a reply to message #34579] |
Tue, 01 April 2003 10:52  |
James Kuyper
Messages: 425
Registered: March 2000
|
Senior Member |
|
|
JD Smith wrote:
...
> Hmmm, it really wants those lines together. OK, I'll try one more time:
>
> nacc=1000 ; or whatever
>
> vec=fltarr(nacc,/NOZERO)
All three times you've posted it, the 'whatever' and the 'vec' are
seperated by two newlines on my system. Maybe it's just your newsreader?
|
|
|
|
| Re: Is there a standard 'null' array? [message #34581 is a reply to message #34580] |
Tue, 01 April 2003 10:18 |
JD Smith
Messages: 850
Registered: December 1999
|
Senior Member |
|
|
On Tue, 01 Apr 2003 11:17:22 -0700, JD Smith wrote:
> On Fri, 28 Mar 2003 09:32:09 -0700, JD Smith wrote:
>
>
>> 2. Be smart about array concatenation. Guess how big your array will
>> be, pre-allocate, and fill it in. If you run out of room, grow the
>> array with concatenation:
>>
>> nacc=1000 ; or whatever vec=fltarr(nacc,/NOZERO)
>
> Sorry, should have been
>
> nacc=1000 ; or whatever vec=fltarr(nacc,/NOZERO)
>
> Silly mailer.
Hmmm, it really wants those lines together. OK, I'll try one more time:
nacc=1000 ; or whatever
vec=fltarr(nacc,/NOZERO)
|
|
|
|
| Re: Is there a standard 'null' array? [message #34582 is a reply to message #34581] |
Tue, 01 April 2003 10:17 |
JD Smith
Messages: 850
Registered: December 1999
|
Senior Member |
|
|
On Fri, 28 Mar 2003 09:32:09 -0700, JD Smith wrote:
>
> 2. Be smart about array concatenation. Guess how big your array will
> be, pre-allocate, and fill it in. If you run out of room, grow the
> array with concatenation:
>
> nacc=1000 ; or whatever vec=fltarr(nacc,/NOZERO)
Sorry, should have been
nacc=1000 ; or whatever
vec=fltarr(nacc,/NOZERO)
Silly mailer.
JD
|
|
|
|
| Re: Is there a standard 'null' array? [message #34584 is a reply to message #34582] |
Tue, 01 April 2003 09:20 |
Matt Feinstein
Messages: 33
Registered: July 2002
|
Member |
|
|
On Mon, 31 Mar 2003 17:40:27 +0900, Marc <m_schellens@hotmail.com>
wrote:
> I don't know Matt's actual need, but another simple method I
> use for small arrays is just to set the first element and later
> delete it.
This is more or less what I ended up doing... The program I was
writing looked for outliers in large data files-- this meant running a
series of tests on the data (using the histogram() function, if you're
curious), where each test was composed of several conditions that
might add up to a small number of 'guilty parties' or may not yield
any result. So I had to keep track of possible null results for each
condition of each test in the series-- At some point I got irritated
about continually checking for null results and posted the query.
Matt Feinstein
--
The Law of Polarity: The probability of wiring a battery with
the correct polarity is (1/2)^N, where N is the number of times
you try to connect it.
|
|
|
|
| Re: Is there a standard 'null' array? [message #34590 is a reply to message #34584] |
Mon, 31 March 2003 08:15 |
JD Smith
Messages: 850
Registered: December 1999
|
Senior Member |
|
|
On Mon, 31 Mar 2003 01:40:27 -0700, Marc wrote:
>> P.S. There was talk of adding a zero-length-array type a few years ago,
>> but the consensus was that it's probably too late to fit it easily into
>> a 25 year old program.
>
> So what? If its now introduced, old code would not have to change.
>
I think it turned out that lots of code *would* have to change. For
instance, a nice use of zero-length arrays would be as a return from
WHERE:
b=indgen(10)
b[where(b lt 0)]=5
should work without fuss, by returning a ZLA, and permitting it in
indexing operations. But what about all the code that explicitly tests
for the value "-1" returned from WHERE? The only possibility is using
tons of /ZLA keywords to turn on that feature. Ugly.
There were many other instances of this type of problem. I think a
complete re-write of IDL which explicitly does *not* preserve full
backwards-compatibility might be sensible. This IDL++ could fix this and
hundreds of other problems tied to old mistakes and conventions.
Parallel release of IDL and IDL++ for a few major versions would ease the
transition. I'm sure the sheer anticipation of people howling and
screaming is enough to stop this plan in its tracks.
JD
|
|
|
|
| Re: Is there a standard 'null' array? [message #34593 is a reply to message #34590] |
Mon, 31 March 2003 00:40 |
marc schellens[1]
Messages: 183
Registered: January 2000
|
Senior Member |
|
|
> P.S. There was talk of adding a zero-length-array type a few years
> ago, but the consensus was that it's probably too late to fit it
> easily into a 25 year old program.
So what? If its now introduced, old code would not have to change.
I don't know Matt's actual need, but another simple method I
use for small arrays is just to set the first element and later
delete it. The code looks nicer than with this if n_elements()...
and for less than somthing 100 elements (and not deep inside some loops)
its fine (even quicker than with if n_elements() within the loop):
vec=0
for i=0,cowscomehome do begin
if some_test then do vec=[vec,complicated_function(i)]
endfor
if n_elements(vec) gt 1 then vec=[1:*] else tmp=temporary(vec)
cheers,
marc
|
|
|
|
| Re: Is there a standard 'null' array? [message #34606 is a reply to message #34593] |
Fri, 28 March 2003 08:32 |
JD Smith
Messages: 850
Registered: December 1999
|
Senior Member |
|
|
On Fri, 28 Mar 2003 06:47:01 -0700, Matt Feinstein wrote:
> This is an embarassingly elementary question, and I'm -sure- that it has
> a simple, elegant, and obvious answer, but...
>
> I often find myself wanting to do things kinda like
>
> Data_I_want_vec = Init
> ...
> for ix = 0 , (size(data_array))[1]-1 do $
> if condition(data_array[ix]) then $
> Data_I_want_vec = [Data_I_want_vec, data_array[ix]]
>
> I realize that there is analogous code for a vectorized version of my
> question, but I think the for-loop version is clearer.
>
> The questions that arise are things like: What to use for 'Init' ? What
> do I do to test for a null result? What if there's no data?
>
> Some notes:
> a) Matlab has an empty vector = [] that serves these purposes. It's
> possible that my desire to program in this fashion is simply a
> Matlab-ism, and I need instruction in the IDL way. b) I do not, really
> and truly, need instruction in the use of the WHERE function or in the
> use of vectorization. Believe me, I know all about that. The analog of
> my question in the vectorized case is that it looks like I have to
> surround the vectorized expression with boring and error-prone tests for
> null inputs and outputs.
>
> Is there a 'standard' way of doing this sort of thing?
Yes. By vectorizing it ;). Despite your awareness of vector
techniques, you might be unaware of the following: in the above code
you've combined two of the absolute least efficient operations in all of
IDL: long loops (assuming your data is substantial), and array
concatenation. Loops are slow because each trip around a for loop takes
a spin through the full IDL interpreter loop, which is necessarily
pokey, thanks to all the wonderful conveniences it provides. With
concatenation, every time you append an element, you're allocating space
for a new larger array, copying the entire original array into it, and
then copying the appended element(s). With time and experience, you'll
find that vectorized versions are more terse and quickly understandable.
It's not the cleanest syntax, but it does become second-nature in time.
Of course, there are plenty of times where "just vectorize it" is not an
option, and you'd really like to build dynamic-sized arrays efficiently.
You have (at least) two choices:
1. Just eat the large overheads and test/concatenate on each round.
This is the standard recipe:
for i=0,cowscomehome do begin
if some_test then do $
if n_elements(vec) eq 0 then vec=[complicated_function(i)] else $
vec=[vec,complicated_function(i)]
endfor
2. Be smart about array concatenation. Guess how big your array will
be, pre-allocate, and fill it in. If you run out of room, grow the
array with concatenation:
nacc=1000 ; or whatever
vec=fltarr(nacc,/NOZERO)
pos=0
for i=0,cowscomehome do begin
if NOT some_test(i) then continue
new_vec=complicated_function(i)
nnv=n_elements(new_vec)
if pos+nnv ge nacc then begin ; grow as necessary
vec=[vec,fltarr(nacc,/NOZERO)]
nacc=2*nacc
endif
vec[pos]=new_vec
pos=pos+nnv
endfor
vec=vec[0:pos-1] ; trim the empty bits
Here I double the array size whenever new space is needed, and trim to
size afterwards. Typically this will result in far fewer
concatenations, but the details depend on your initial guess and array
growing prescription compared to the problem at hand. Sometimes it's
not so easy to guess.
I for one would love to have true efficient array concatenation in IDL,
ala:
push,vec,append_vec
unshift,vec,prepend_vec
These functions from Perl are efficient because Perl does all the ugly
work typified in #2 above for you. Each PUSH does not necessarily copy
the entire array. Instead, arrays are pre-allocated to be larger than
you need, and grown in intelligently-sized chunks, shifted, unshifted,
spliced, etc., but only re-allocated and copied if it's absolutely
necessary. This would come with a tradeoff though: arrays would need to
be smarter, and hence, in more pedestrian usage (e.g. a=b+c) would be
slower. Perhaps two kinds of interoperable arrays would be in order:
the fancy kind that grow/shrink/prepend/append/pre-allocate
automagically, and those that are built purely for speed, and never take
up more space than the data they contain.
By the way, if you don't like to type all the repetitive "if
n_elements()" bit, you can use a routine to do it for you (albeit even
slightly more inefficiently thanks to the routine-call overhead):
pro push, array, append
if n_elements(array) eq 0 then array=[append] else array=[array,append]
end
Good luck,
JD
P.S. There was talk of adding a zero-length-array type a few years
ago, but the consensus was that it's probably too late to fit it
easily into a 25 year old program.
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
