| Re: Bad data in structure (NaN HowTo?) [message #22581] |
Thu, 23 November 2000 00:00  |
Martin Schultz
Messages: 515
Registered: August 1997
|
Senior Member |
|
|
Randall Skelton wrote:
>
> I have an array of large structures which occasionally is filled (via an
> external C program) and passed into IDL with -12345 signifying the data
> for that element is lost. I would like to convert the occurances of
> -12345 to NaN's in IDL but I am a little perplexed on how to do this.
>
> I had hoped that since this is technically an 'array' (albeit an array of
> structures) I would just be able to use the 'where' command; alas, it
> appears that structures are not allowed in the where command:
>
> B = where(atrl eq -12345, count)
> % Struct expression not allowed in this context: ATRL.
>
> Each structure has about 450 elements in it and is comprised itself of
> strings, ints, floats and doubles and arrays of each of the above. My
> initial thought is to make an array of strings which represent the
> elements of the structure and loop over that array, searching for '-12345'
> in each element or array as I go. This seems rather inefficient. It
> would be much nicer if I could directly assign the structure element to
> NaN in C and pass it into IDL (and have IDL interpret the missing data as
> as NaN)? Oh yes, it would be nice if I could use the same C code on
> intel, PowerPC, and sparc architechures... Does anyone know how this
> might be done?
>
> Thanks for the help.
>
> Randall
How about including if ( value == -12345 ) value = value/0. in your
C program. With the appropriate compiler flags you might be able to
catch the math error and then you surely have a NaN as result. I just
don't know if IDL uses the same representation for NaN as the C
compiler on all platforms. (if NaN is consistent on one platform but
different between platforms this shouldn't hurt).
Playing dirty ;-),
Martin
--
[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[ [[[[[[[
[[ Dr. Martin Schultz Max-Planck-Institut fuer Meteorologie [[
[[ Bundesstr. 55, 20146 Hamburg [[
[[ phone: +49 40 41173-308 [[
[[ fax: +49 40 41173-298 [[
[[ martin.schultz@dkrz.de [[
[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[ [[[[[[[
|
|
|
|
| Re: Bad data in structure (NaN HowTo?) [message #22582 is a reply to message #22581] |
Wed, 22 November 2000 16:03  |
Randall Skelton
Messages: 169
Registered: October 2000
|
Senior Member |
|
|
Thanks to both Pavel and Craig for their help.
> In your case, I'd automate the search for numerical fields, because I
> wouldn't be able to go through 450 fields myself. None of my business,
> but are you sure this is the best way to organize the data?
I am certainly open to suggestions on how to organize this data... It is
spectroscopic remote sensing data and during measurements I get a dump of
all of the sensor data (every 2 seconds) and one high resolution spectrum
of the atmosphere. The 400+ items in the structure correspond to
instrument readings (i.e. exact GMT time, spacecraft orientation data,
latitude, longitude, etc...) and different data-streams (i.e. point arrays
for each useful spectral window, an oversampled pointing mirror array,
gyro arrays, etc...). The real kicker is that I am grouping a series of
these measurements together as an array of structures as it is then much
easier to plot and manipulate this beastly data with 'plot,
atcalc.ind_time, atrl.latitude'. My real problem here is that
occasionally I get data drop out in key areas and I have to fiddle to fill
my structure in C. At the moment, if a calculated string is garbled I
fill with an empty string which is no problem. For all others (shorts,
longs, floats and doubles) I am using '-12345' to indicate bad data.
Although I can try and use data thresholds to flag this as bad data when
plotting, it tends to make a mess of things when I write files or do
analyses. What would people suggest I do for bad ints, shorts, longs, and
floats? Should I try to handle this better in C so as to avoid the
problem in IDL?
All of my structures are created in ANSI C from data which originates in
SQL databases. I currently interface to IDL via call_external (I am
looking at the alternatives to call_external but for the moment this
works).
All of this begs the question, how much data can one person plot and
interpret at one time? The answer is: you'd be surprised! but you do need a
*big* monitor :) and an 11x17 printer.
Cheers,
Randall
|
|
|
|
| Re: Bad data in structure (NaN HowTo?) [message #22589 is a reply to message #22582] |
Wed, 22 November 2000 00:00 |
Craig Markwardt
Messages: 1869
Registered: November 1996
|
Senior Member |
|
|
Randall Skelton <rhskelto@atm.ox.ac.uk> writes:
> I have an array of large structures which occasionally is filled (via an
> external C program) and passed into IDL with -12345 signifying the data
> for that element is lost. I would like to convert the occurances of
> -12345 to NaN's in IDL but I am a little perplexed on how to do this.
>
> I had hoped that since this is technically an 'array' (albeit an array of
> structures) I would just be able to use the 'where' command; alas, it
> appears that structures are not allowed in the where command:
>
> B = where(atrl eq -12345, count)
> % Struct expression not allowed in this context: ATRL.
Hi Randall--
You have the 500-pound gorilla approach from Pavel. However, if your
-12345's appear in a few well defined fields in the structure, then
it's much easier and more efficent to do the following:
wh = where(atrl.field1 EQ -12345, ct)
if ct GT 0 then atrl(wh).field1 = -12345
However, if there are lots of fields then the 500-pound gorilla may be
what you need. You mentioned that integers and strings are the same
way. How can this be? There is no such thing as a NaN string or
integer.
If you do it in C, I would propose that you pass in the value of NaN
into your C procedure and then just use it there. That should be the
most portable. Then from the IDL side you would call your routine
with !VALUES.NAN
Cheers,
Craig
--
------------------------------------------------------------ --------------
Craig B. Markwardt, Ph.D. EMAIL: craigmnet@cow.physics.wisc.edu
Astrophysics, IDL, Finance, Derivatives | Remove "net" for better response
------------------------------------------------------------ --------------
|
|
|
|
| Re: Bad data in structure (NaN HowTo?) [message #22591 is a reply to message #22589] |
Wed, 22 November 2000 00:00 |
Pavel A. Romashkin
Messages: 531
Registered: November 2000
|
Senior Member |
|
|
Randall Skelton wrote:
>
> B = where(atrl eq -12345, count)
> % Struct expression not allowed in this context: ATRL.
I would try something like
;Loop for each field.
FOR I = 0, N_TAGS(atrl) - 1 DO BEGIN
type = size(atrl.(i), /type)
if (type lt 6) or (type gt 11) then begin
; Assume no undefined tags are present.
; If they are present, exclude type=0.
B = where(atrl.(i) eq -12345, count)
if count gt 0 then atrl.(i)[B] = !values.f_nan
endif
ENDFOR
For multiple numerical fields, I'd use a loop. Given relatively small
number of fields, this would not require vectorizing.
> Each structure has about 450 elements in it and is comprised itself of
> strings, ints, floats and doubles and arrays of each of the above.
In your case, I'd automate the search for numerical fields, because I
wouldn't be able to go through 450 fields myself. None of my business,
but are you sure this is the best way to organize the data?
Cheers,
Pavel
|
|
|
|
| Re: Bad data in structure (NaN HowTo?) [message #22663 is a reply to message #22582] |
Tue, 28 November 2000 00:00 |
Craig Markwardt
Messages: 1869
Registered: November 1996
|
Senior Member |
|
|
Randall Skelton <rhskelto@atm.ox.ac.uk> writes:
> Thanks to both Pavel and Craig for their help.
>
>> In your case, I'd automate the search for numerical fields, because I
>> wouldn't be able to go through 450 fields myself. None of my business,
>> but are you sure this is the best way to organize the data?
>
> I am certainly open to suggestions on how to organize this data... It is
> spectroscopic remote sensing data and during measurements I get a dump of
> all of the sensor data (every 2 seconds) and one high resolution spectrum
Randall, is it true that you are filling the -12345 values within your
CALL_EXTERNAL procedure? If so, then perhaps you can try the second
thing I suggested. Namely, in IDL create whatever "null" variables
you want, either NaN or whatever. Then pass those variables into the
CALL_EXTERNAL procedure. Then your C code can easily subtitute those
values wherever needed. However there is no such thing as NaN for
integers so I'm not sure what you can do there.
Craig
--
------------------------------------------------------------ --------------
Craig B. Markwardt, Ph.D. EMAIL: craigmnet@cow.physics.wisc.edu
Astrophysics, IDL, Finance, Derivatives | Remove "net" for better response
------------------------------------------------------------ --------------
|
|
|
|
| Re: Bad data in structure (NaN HowTo?) [message #22672 is a reply to message #22582] |
Mon, 27 November 2000 00:00 |
Pavel A. Romashkin
Messages: 531
Registered: November 2000
|
Senior Member |
|
|
Randall Skelton wrote:
> I am certainly open to suggestions on how to organize this data... It is
> spectroscopic remote sensing data and during measurements I get a dump of
> all of the sensor data (every 2 seconds) and one high resolution spectrum
> of the atmosphere. The 400+ items in the structure correspond to
> instrument readings (i.e. exact GMT time, spacecraft orientation data,
> latitude, longitude, etc...) and different data-streams (i.e. point arrays
> for each useful spectral window, an oversampled pointing mirror array,
> gyro arrays, etc...).
On the second though... maybe its allright as it is? :-(
I see now that the amount of data you have is huge. Besides, it keeps
coming, huh. Maybe then just make an interface to shield yourself from
the 450 fields (which send chill down my spine) and let IDL handle the
mess internally.
By the way, I thought you have and array of those structures. Replacing
your -12345 can be done on all of the structures in the array in one
step with the "500 lbs gorilla" method. IDL is smart enough to index all
arrays in a field of a structure array.
Cheers,
Pavel
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
