| Re: Where vs Histogram vs ?? [message #32607 is a reply to message #32526] |
Fri, 18 October 2002 13:49   |
Pavel A. Romashkin
Messages: 531
Registered: November 2000
|
Senior Member |
|
|
Allright, so we need a solution in IDL.
At this array size the slowest portion of the process is not the WHERE
function as far as I can tell. It is memory reallocation for the main
array and for the temporary index arrays that IDL creates. Therefore I
can suggest trying the following approach.
Allocate it all only once:
ind = ptrarr(n_tags(data_St)
for i = 0, n_tags(data_st) do ind[i] = ptr_new(data_st.(i))
This does take a little time to execute.
Now you have a static index of all fields. Of course, you have used
twice the memory but given the relatively small data volume it seems ok.
On my machine the RAM used by both structure and pointer index barely
reaches 1010 Mb, so I have room for further calculations.
Now, you can search the pointer array elements using WHERE, and it is
fairly fast. I tested it on my machine; the same WHERE statement you
show took 56 s for the structure array, but only 6 s using the index
pointer array. Further speed increase will be achievable if you merged
timestamps into one field, as others recommended; some flexibility in
querying would be, however, lost. And of course you can use the
resulting INDEX to subscript your original structire array.
Hope this helps,
Pavel
**********
index = Where(* ind[0] GE 596 AND $
* ind[0] LE 2000 AND $ ; yr
* ind[1] GE 15 AND $ ; day
* ind[1] LE 52 AND $
* ind[6] GE 6 AND $ ;beam
* ind[6] LE 5 AND $
* ind[2] GE 15 AND $ ;half hr
* ind[2] LE 5 AND $
* ind[4] EQ 5 AND $ ;WRF
* ind[5] EQ 5 AND $ ;Freq
(* ind[8])[0] NE -555)
**********
Andrew Cool wrote:
>
> "Pavel A. Romashkin" wrote:
>>
>> I can definitely echo Bob's suggestion to use index for searching. Don't
>> use structure fields. Using a database would like ly be better yet; I
>> think MS Access Jet should be reasonably fast with 15 mln records.
>> Good luck,
>> Pavel
>
> Hi Pavel,
>
> I'm confined to running this under OpenVMS, so MS Access probably
> ain't the cure here. ;-)
>
> Andrew
>
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
