| Re: matching fields in ascii or text files. [message #26083] |
Fri, 03 August 2001 17:36 |
Craig Markwardt
Messages: 1869
Registered: November 1996
|
Senior Member |
|
|
patrick@es.ucsc.edu (Patrick McEnaney) writes:
> I'm writing an idl gui script to cycle through fairly long files of
> data that are collected from a ctd and compare them with chlorophyll
> measurements from insitu sampling. I want to select the data from a
> specific depth for chlorophyll and and write all values for that
> depth in another file. Just putting the data into excel files would be
> very cumbersome because the data are collected over a month long
> cruise and there are alot of measurements. Ultimately I'll use the
> script whenever I need to compare fields from cruise data. Can you
> suggest a way to construct such a matching routine in idl?
The solution is even easier in that case.
wh = where(depth_list EQ depth)
Now WH are the indices of depths in DEPTH_LIST which have the same
depth as DEPTH. Wow, I hope that was intelligible.
The problem with this approach is that if you want to select out many
many DEPTHs in a loop, you will start to slow down because you are
essentially doing an N^2 operation (ie, searching the entire
DEPTH_LIST every time). In that case you may care to revert to
something else.
Craig
--
------------------------------------------------------------ --------------
Craig B. Markwardt, Ph.D. EMAIL: craigmnet@cow.physics.wisc.edu
Astrophysics, IDL, Finance, Derivatives | Remove "net" for better response
------------------------------------------------------------ --------------
|
|
|
|
| Re: matching fields in ascii or text files. [message #26088 is a reply to message #26083] |
Fri, 03 August 2001 15:35  |
Paul van Delst
Messages: 364
Registered: March 1997
|
Senior Member |
|
|
Patrick McEnaney wrote:
>
> "Pavel A. Romashkin" <pavel.romashkin@noaa.gov> wrote in message news:<3B6AD991.5669DE2B@noaa.gov>...
>> Are you attempting this in IDL? Or is it a general data handling
>> question? It seems to me that TXT file and ASCII is the same kind of
>> file. To compare the columns by scrolling them in a window, Excel will do.
>> Cheers,
>> Pavel
>>
> Greetings Pavel-
>
> I'm writing an idl gui script to cycle through fairly long files of
> data that are collected from a ctd and compare them with chlorophyll
> measurements from insitu sampling. I want to select the data from a
> specific depth for chlorophyll and and write all values for that
> depth in another file. Just putting the data into excel files would be
> very cumbersome because the data are collected over a month long
> cruise and there are alot of measurements. Ultimately I'll use the
> script whenever I need to compare fields from cruise data. Can you
> suggest a way to construct such a matching routine in idl?
The first question that came to my mind (apart from "aren't TXT and ASCII the same thing?"
:o) was: what do you mean by compare? Do you have a depth tolerance? e.g. given a depth
value from one file is +/- 10m from another file considered the "same" depth?
Also when you said:
"I want to select the data from a specific depth for chlorophyll and and write all values
for that depth in another file."
did you mean:
"I want to select the data from a specific depth for chlorophyll and write all **the ctd**
values for that depth in another file."
??
My first cut at something like this would be to pick the dataset with the least number of
depth values, say the chlorophyll stuff - you can loop through those depths. Then you can
use where to find the corresponding depths for the ctd data, like:
IDL> ctd_depth = findgen(10000)/100. & chlorophyll_depth = 20.0
IDL> help, ctd_depth, chlorophyll_depth
CTD_DEPTH FLOAT = Array[10000]
CHLOROPHYLL_DEPTH
FLOAT = 20.0000
IDL> depth_tolerance=0.1 ; metres, for example
IDL> loc = where( abs(ctd_depth-chlorophyll_depth) lt depth_tolerance, count )
and loc should give those ctd_depths that are within your tolerance for matching
chlorophyll depth.
IDL> print, ctd_depth[loc]
19.9100 19.9200 19.9300 19.9400 19.9500
19.9600 19.9700 19.9800 19.9900 20.0000
20.0100 20.0200 20.0300 20.0400 20.0500
20.0600 20.0700 20.0800 20.0900
adn then write all the data asociated with those ctd depths (using loc) to another file.
Is this the sort of thing you mean?? If so, why would you need a gui?
paulv
p.s. If you have to read in simple columnar ASCII data files (that just contain numbers),
you might want to have a look at DDREAD.PRO - it's a piece of IDL code (written by a
feller called Fred Knight) that I find indispensible for simply reading in column ASCII
numbers.
--
Paul van Delst A little learning is a dangerous thing;
CIMSS @ NOAA/NCEP Drink deep, or taste not the Pierian spring;
Ph: (301)763-8000 x7274 There shallow draughts intoxicate the brain,
Fax:(301)763-8545 And drinking largely sobers us again.
Alexander Pope.
|
|
|
|
| Re: matching fields in ascii or text files. [message #26090 is a reply to message #26088] |
Fri, 03 August 2001 15:25 |
Craig Markwardt
Messages: 1869
Registered: November 1996
|
Senior Member |
|
|
patrick@es.ucsc.edu (Patrick McEnaney) writes:
> Folks, suggestions needed for going about the following problem:
>
> I have columnar oceanographic data in two different data files. One is
> a txt file, the other ascii. I want to compare depth values in a
> column from one file to depth values in a column from a second file,
> then write all values associated with a specific depth in other
> columns such as temperature, conductivity, nutrients, etc, to another
> file. Should I attempt some sort of boolean expression to search
> through the two files to match values or is there a more efficient
> method that someone can suggest?
Hi Patrick--
I think what you are trying to do is match up the rows of two files
based on a shared column. If you really need to match them exactly,
then perhaps the best route is to SORT both arrays by depth, and then
step through the lists in tandem.
depth1 = ...
depth2 = ...
ds1 = sort(depth1) & ds2 = sort(depth2)
j = 0L
for i = 0, n_elements(ds1)-1 do begin ;; Step through list 1
while depth2(ds2(j)) LT depth1(ds1(i)) do j = j + 1 ;; Find in list 2
; ... process
endfor
You don't say whether you want to match them exactly however. Another
really good solution is often spline interpolation. The is especially
useful when you have two files sampled on two different time grids (or
depth grids in your case). Then you would do,
y2 = spl_interp(depth1, y1, spl_init(depth1, y1), depth2)
which interpolates the values of Y1, measured on the DEPTH1 grid, onto
the DEPTH2 grid. Whew!
Craig
--
------------------------------------------------------------ --------------
Craig B. Markwardt, Ph.D. EMAIL: craigmnet@cow.physics.wisc.edu
Astrophysics, IDL, Finance, Derivatives | Remove "net" for better response
------------------------------------------------------------ --------------
|
|
|
|
| Re: matching fields in ascii or text files. [message #26094 is a reply to message #26090] |
Fri, 03 August 2001 14:59 |
patrick
Messages: 15
Registered: July 2001
|
Junior Member |
|
|
"Pavel A. Romashkin" <pavel.romashkin@noaa.gov> wrote in message news:<3B6AD991.5669DE2B@noaa.gov>...
> Are you attempting this in IDL? Or is it a general data handling
> question? It seems to me that TXT file and ASCII is the same kind of
> file. To compare the columns by scrolling them in a window, Excel will do.
> Cheers,
> Pavel
>
Greetings Pavel-
I'm writing an idl gui script to cycle through fairly long files of
data that are collected from a ctd and compare them with chlorophyll
measurements from insitu sampling. I want to select the data from a
specific depth for chlorophyll and and write all values for that
depth in another file. Just putting the data into excel files would be
very cumbersome because the data are collected over a month long
cruise and there are alot of measurements. Ultimately I'll use the
script whenever I need to compare fields from cruise data. Can you
suggest a way to construct such a matching routine in idl?
Cheers,
Patrick
|
|
|
|
| Re: matching fields in ascii or text files. [message #26107 is a reply to message #26094] |
Fri, 03 August 2001 10:04 |
Pavel A. Romashkin
Messages: 531
Registered: November 2000
|
Senior Member |
|
|
Are you attempting this in IDL? Or is it a general data handling
question? It seems to me that TXT file and ASCII is the same kind of
file. To compare the columns by scrolling them in a window, Excel will do.
Cheers,
Pavel
Patrick McEnaney wrote:
>
> Folks, suggestions needed for going about the following problem:
>
> I have columnar oceanographic data in two different data files. One is
> a txt file, the other ascii. I want to compare depth values in a
> column from one file to depth values in a column from a second file,
> then write all values associated with a specific depth in other
> columns such as temperature, conductivity, nutrients, etc, to another
> file. Should I attempt some sort of boolean expression to search
> through the two files to match values or is there a more efficient
> method that someone can suggest?
>
> Thanks,
>
> Patrick
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
