| Re: Correlate and NAN [message #23155] |
Mon, 08 January 2001 10:24  |
btt
Messages: 345
Registered: December 2000
|
Senior Member |
|
|
Thanks Pavel and Andy,
I have deNANed the data as you suggested.
Ben
Andy Loughe wrote:
> Why not simply perform the correlation on a subset of the larger arrays,
> that portion with the NaNs removed? Something like:
>
> indices = [ where( finite(dataset1) eq 1 ), where( finite(dataset2) eq 2
> ) ]
>
> common_indices = indices( UNIQ(indices, sort(indices)) )
>
> Result = CORRELATE( dataset1(common_indices), dataset2(common_indices)
> )
>
> Ben Tupper wrote:
>>
>> Hello,
>>
>> I have two datasets that I would like to correlate using the CORRELATE
>> function. Each dataset has some members flagged as NANs; the NANs are
>> not neccessarily coincident. The online documentation makes no mention
>> of NAN-handling, but the procedure in the lib directory indicates (see
>> modifications history) that it handles NANs (although there is no
>> keyword for it). It doesn't really handle NANs the way I expect it
>> to. For example, repeated calls to the TOTAL function don't set the
>> keyword NAN, so TOTAL doesn't check for NANs.
>>
>> I'm not sure if it is reasonable to involve NANs in a correlation... but
>> it seems reasonable to request that the routine ignore NANs in the input
>> arguments.
>>
>> Is there a simple solution to this NAN-jam?
>>
>> Thanks,
>>
>> Ben
>>
>> --
>> Ben Tupper
>> Bigelow Laboratory for Ocean Sciences
>> 180 McKown Point Rd.
>> W. Boothbay Harbor, ME 04575
>> btupper@bigelow.org
>
> --
> Andrew Loughe =====================================================
> NOAA/OAR/FSL/AD R/FS5 | email: loughe@fsl.noaa.gov
> 325 Broadway | wwweb: www-ad.fsl.noaa.gov/users/loughe
> Boulder, CO 80305-3328 | phone: 303-497-6211 fax: 303-497-6301
--
Ben Tupper
Bigelow Laboratory for Ocean Sciences
180 McKown Point Rd.
W. Boothbay Harbor, ME 04575
btupper@bigelow.org
|
|
|
|
| Re: Correlate and NAN [message #23156 is a reply to message #23155] |
Mon, 08 January 2001 09:08  |
Andy Loughe
Messages: 174
Registered: November 1995
|
Senior Member |
|
|
Why not simply perform the correlation on a subset of the larger arrays,
that portion with the NaNs removed? Something like:
indices = [ where( finite(dataset1) eq 1 ), where( finite(dataset2) eq 2
) ]
common_indices = indices( UNIQ(indices, sort(indices)) )
Result = CORRELATE( dataset1(common_indices), dataset2(common_indices)
)
Ben Tupper wrote:
>
> Hello,
>
> I have two datasets that I would like to correlate using the CORRELATE
> function. Each dataset has some members flagged as NANs; the NANs are
> not neccessarily coincident. The online documentation makes no mention
> of NAN-handling, but the procedure in the lib directory indicates (see
> modifications history) that it handles NANs (although there is no
> keyword for it). It doesn't really handle NANs the way I expect it
> to. For example, repeated calls to the TOTAL function don't set the
> keyword NAN, so TOTAL doesn't check for NANs.
>
> I'm not sure if it is reasonable to involve NANs in a correlation... but
> it seems reasonable to request that the routine ignore NANs in the input
> arguments.
>
> Is there a simple solution to this NAN-jam?
>
> Thanks,
>
> Ben
>
> --
> Ben Tupper
> Bigelow Laboratory for Ocean Sciences
> 180 McKown Point Rd.
> W. Boothbay Harbor, ME 04575
> btupper@bigelow.org
--
Andrew Loughe =====================================================
NOAA/OAR/FSL/AD R/FS5 | email: loughe@fsl.noaa.gov
325 Broadway | wwweb: www-ad.fsl.noaa.gov/users/loughe
Boulder, CO 80305-3328 | phone: 303-497-6211 fax: 303-497-6301
|
|
|
|
| Re: Correlate and NAN [message #23157 is a reply to message #23156] |
Mon, 08 January 2001 08:55 |
Pavel A. Romashkin
Messages: 531
Registered: November 2000
|
Senior Member |
|
|
How about using just the coincident valid data pairs that you can get
using FINITE and GET_INTERSECTION functions?
ind_1 = where(finite(data_set_1))
ind_2 = where(finite(data_set_2))
index = Setintersection(ind_1, ind_2)
result = correlate(data_set_1[index], data_set_2[index])
"Setintersection" is posted on David's home page,
http://www.dfanning.com/tips/set_operations.html. Although it is called
there *Setintersection*, although it does not *set* anything, it returns
what you need :-)
Cheers,
Pavel
Ben Tupper wrote:
>
> Hello,
>
> I have two datasets that I would like to correlate using the CORRELATE
> function. Each dataset has some members flagged as NANs; the NANs are
> not neccessarily coincident. The online documentation makes no mention
> of NAN-handling, but the procedure in the lib directory indicates (see
> modifications history) that it handles NANs (although there is no
> keyword for it). It doesn't really handle NANs the way I expect it
> to. For example, repeated calls to the TOTAL function don't set the
> keyword NAN, so TOTAL doesn't check for NANs.
>
> I'm not sure if it is reasonable to involve NANs in a correlation... but
> it seems reasonable to request that the routine ignore NANs in the input
> arguments.
>
> Is there a simple solution to this NAN-jam?
>
> Thanks,
>
> Ben
>
> --
> Ben Tupper
> Bigelow Laboratory for Ocean Sciences
> 180 McKown Point Rd.
> W. Boothbay Harbor, ME 04575
> btupper@bigelow.org
|
|
|
|
| Re: Correlate and NAN [message #90155 is a reply to message #23155] |
Tue, 03 February 2015 07:06 |
atmospheric physics
Messages: 121
Registered: June 2010
|
Senior Member |
|
|
Hello,
Following the earlier posts, I have one query: How about applying this method for a data matrix with m columns & n rows? Finding common indices for two arrays is easy. How can this be done for a data matrix with m x n dimensions so that CORRELATE function can be used automatically to obtain m x m dimensional correlation matrix.
Any suggestions???
Thanks in advance,
Madhavan
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
