Messages from Usenet group comp.lang.idl-pvwave, compiled by Paulo Penteado
I just tried to plot a number of ncdf data sets using idl. Now I want to find where the missing values and also want to plot the values and missing values in the given time. Can u help me to solve this problem.
Krishnapriya M writes: > I just tried to plot a number of ncdf data sets using idl. Now I want to find where the missing values and also want to plot the values and missing values in the given time. Can u help me to solve this problem. If you were to read your netCDF files with NCDF_File object from the Coyote Library, you will be able to get this information directly from the GetVarData method that reads the data out of the file. The FILLVALUE keyword returns the missing data value, and the MISSINGINDICES keyword returns the indices of the missing values in your data set. Note that the method also applies whatever offset and scale factor are present in the file to the data when it is read from the file. http://www.idlcoyote.com/fileio_tips/ncdf_browser.html Cheers, David -- David Fanning, Ph.D. Fanning Software Consulting, Inc. Coyote's Guide to IDL Programming: http://www.idlcoyote.com/ Sepore ma de ni thue. ("Perhaps thou speakest truth.")
Howdy, Is a netCDF4-capable version of your netcdf browser/ncdf_data program on the cards? (Refiled under "stuff I want" :o) What's a $ estimate for the upgrade? cheers, paulv p.s. disclaimer: I do not speak for my employer (and they do not speak for me) On 12/03/13 08:01, David Fanning wrote: > Krishnapriya M writes: > >> I just tried to plot a number of ncdf data sets using idl. Now I >> want to find where the missing values and also want to plot the values and missing values in the given time. Can u help me to solve this problem. > > If you were to read your netCDF files with NCDF_File object from the > Coyote Library, you will be able to get this information directly from > the GetVarData method that reads the data out of the file. The FILLVALUE > keyword returns the missing data value, and the MISSINGINDICES keyword > returns the indices of the missing values in your data set. Note that > the method also applies whatever offset and scale factor are present in > the file to the data when it is read from the file. > > http://www.idlcoyote.com/fileio_tips/ncdf_browser.html > > Cheers, > > David >
Paul van Delst writes: > Is a netCDF4-capable version of your netcdf browser/ncdf_data program on > the cards? Well, I've made changes over time that people who "say" they using netCDF4 files have been happy with. I'm not currently working with netCDF4 files, so I don't know for sure whether more is needed. > What's a $ estimate for the upgrade? I'd have to have a look. Can you send me a file of the type you think doesn't work well with the software? I'll have a look and see how much time I think it will take to make the software compatible with the file. Cheers, David -- David Fanning, Ph.D. Fanning Software Consulting, Inc. Coyote's Guide to IDL Programming: http://www.idlcoyote.com/ Sepore ma de ni thue. ("Perhaps thou speakest truth.")
On 04.12.2013 02:00, David Fanning wrote: > Well, I've made changes over time that people who "say" they using > netCDF4 files have been happy with. I'm not currently working with > netCDF4 files, so I don't know for sure whether more is needed. NCDF_FILE is compatible with netCDF4 features (read, create, compression options). I just tried NCDF_BROWSER and it seems to work on my NetCDF4 files too.
Hello, On 12/04/13 03:57, Fabien wrote: > On 04.12.2013 02:00, David Fanning wrote: >> Well, I've made changes over time that people who "say" they using >> netCDF4 files have been happy with. I'm not currently working with >> netCDF4 files, so I don't know for sure whether more is needed. > > NCDF_FILE is compatible with netCDF4 features (read, create, compression > options). I just tried NCDF_BROWSER and it seems to work on my NetCDF4 > files too. My netcdf4 files are replete with groups. Each group is self contained and written by a separate object method - either as a subgroup (or subsubgroup), or as a single file in its own right (all the processing and writing code is Fortran2003, not IDL). This flexibility is needed since all the subgroups are optional (e.g. microwave sensors won't have non-LTE data, and infrared sensors won't have antenna correction data). All the datafile viewing code is IDL - but my simple netCDF reader doesn't grok netcdf4. Here is a (heavily truncated) ncdump of one of my files: -----%<----- netcdf amsua_metop-a.SpcCoeff { // global attributes: :write_module_history = "$Id: ..." ; :creation_date_and_time = "2013/10/16, ..." ; :Release = 9 ; :Version = 1 ; :Title = "Spectral coefficients for MetOp-A AMSU-A" ; :History = "$Id: ..." ; :Comment = "Converted from ..." ; group: spccoeff { group: sensorinfo { } // group sensorinfo group: accoeff { group: sensorinfo { } // group sensorinfo } // group accoeff } // group spccoeff } -----%<----- Only the global attributes are read in with ncdf_data. When I click the "Read Entire File" button I get: IDL> ncdf_browser, 'amsua_metop-a.SpcCoeff.nc' IDL> A variable named "data" has been created at the main IDL level. IDL> help, data ** Structure <1d6dc28>, 2 tags, length=120, data length=120, refs=1: _FILENAME STRING 'amsua_metop-a.SpcCoeff.nc' _GLOBAL_ATTR STRUCT -> <Anonymous> Array[1] IDL> help, data._global_attr ** Structure <2013138>, 8 tags, length=104, data length=104, refs=2: NCDF_FILENAME STRING 'amsua_metop-a.SpcCoeff.nc' WRITE_MODULE_HISTORY STRING '$Id: SpcCoeff_netCDF_IO.f90 '... CREATION_DATE_AND_TIME STRING '2013/10/16, 15:05:07 -0400UTC' RELEASE LONG 9 VERSION LONG 1 TITLE STRING 'Spectral coefficients for '... HISTORY STRING '$Id: SpcCoeff_Convert.f90 32636'... COMMENT STRING 'Converted from Release-8 format'... cheers, paulv
On 12/03/13 20:00, David Fanning wrote: > Paul van Delst writes: > >> Is a netCDF4-capable version of your netcdf browser/ncdf_data program on >> the cards? > > Well, I've made changes over time that people who "say" they using > netCDF4 files have been happy with. I'm not currently working with > netCDF4 files, so I don't know for sure whether more is needed. > >> What's a $ estimate for the upgrade? > > I'd have to have a look. Can you send me a file of the type you think > doesn't work well with the software? I'll have a look and see how much > time I think it will take to make the software compatible with the file. Done. cheers, paulv
Paul van Delst writes: > My netcdf4 files are replete with groups. Each group is self contained > and written by a separate object method - either as a subgroup (or > subsubgroup), or as a single file in its own right (all the processing > and writing code is Fortran2003, not IDL). > > This flexibility is needed since all the subgroups are optional (e.g. > microwave sensors won't have non-LTE data, and infrared sensors won't > have antenna correction data). > > All the datafile viewing code is IDL - but my simple netCDF reader > doesn't grok netcdf4. As you might expect, the documentation for netCDF4 is basically non- existent in the IDL documentation, beyond some new routines for handling groups. Does anyone know if there is a simple test I can perform in IDL to tell if the file is a netCDF3 as opposed to a netCDF4 file? Cheers, David -- David Fanning, Ph.D. Fanning Software Consulting, Inc. Coyote's Guide to IDL Programming: http://www.idlcoyote.com/ Sepore ma de ni thue. ("Perhaps thou speakest truth.")
Hi David, On 04.12.2013 16:41, David Fanning wrote: > Does anyone know if there is a simple test I can perform in IDL > to tell if the file is a netCDF3 as opposed to a netCDF4 file? I had a support request on this a couple of months ago. It will be included in IDL 8.3, I've been told. In the mean time they gave me following code, which works: ;+ ; :Description: ; Checks the format of a NCDF file without opening it ; (CLASSIC, 64BIT or NETCDF4) ; ; :Returns: ; A string:: ; - 'FORMAT_CLASSIC' ; - 'FORMAT_64BIT' ; - 'FORMAT_NETCDF4' ; - 'Unknown' ; ; :Params: ; filename: in, string ; the path to the file to check ; ; :Author: Anonymous Developer from Exelis ;- function w_ncdf_format, filename ON_ERROR, 2 openr, unit, filename, /get_lun data = bytarr(4) readu, unit, data free_lun, unit case string(data) of string([67b, 68b, 70b, 1b]): return, 'FORMAT_CLASSIC' string([67b, 68b, 70b, 2b]): return, 'FORMAT_64BIT' string([137b, 72b, 68b, 70b]): return, 'FORMAT_NETCDF4' else: return, 'UNKNOWN' endcase end
Paul van Delst writes: >> I'd have to have a look. Can you send me a file of the type you think >> doesn't work well with the software? I'll have a look and see how much >> time I think it will take to make the software compatible with the file. > > Done. The netCDF4 file you sent me can be browsed with the H5_Browser supplied with IDL. If the good folks at ExelisVis would give me the code for that browser, I'd fix it so that it worked more intuitively. ;-) So, I guess the question is, do we need a better HDF5/netCDF4 browser or do we just need a better way to get at the data in these files? I haven't yet looked at Mike Galloy's routines, but these are generally extremely reliable for these kinds of things. Cheers, David -- David Fanning, Ph.D. Fanning Software Consulting, Inc. Coyote's Guide to IDL Programming: http://www.idlcoyote.com/ Sepore ma de ni thue. ("Perhaps thou speakest truth.")
David Fanning writes: > The netCDF4 file you sent me can be browsed with the H5_Browser supplied > with IDL. If the good folks at ExelisVis would give me the code for that > browser, I'd fix it so that it worked more intuitively. ;-) Well, maybe I do have this code. I'll have a look. :-) Cheers, David -- David Fanning, Ph.D. Fanning Software Consulting, Inc. Coyote's Guide to IDL Programming: http://www.idlcoyote.com/ Sepore ma de ni thue. ("Perhaps thou speakest truth.")
On Wednesday, December 4, 2013 9:22:28 AM UTC-7, David Fanning wrote: > Paul van Delst writes: > > > >>> I'd have to have a look. Can you send me a file of the type you think > >>> doesn't work well with the software? I'll have a look and see how much > >>> time I think it will take to make the software compatible with the file. > >> > >> Done. > > > > The netCDF4 file you sent me can be browsed with the H5_Browser supplied > > with IDL. If the good folks at ExelisVis would give me the code for that > > browser, I'd fix it so that it worked more intuitively. ;-) > > > > So, I guess the question is, do we need a better HDF5/netCDF4 browser or > > do we just need a better way to get at the data in these files? I > > haven't yet looked at Mike Galloy's routines, but these are generally > > extremely reliable for these kinds of things. > > > > Cheers, > > > > David > > -- > > David Fanning, Ph.D. > > Fanning Software Consulting, Inc. > > Coyote's Guide to IDL Programming: http://www.idlcoyote.com/ > > Sepore ma de ni thue. ("Perhaps thou speakest truth.") Hi David, The h5_browser.pro should be in your lib directory. Feel free to hack away! -Chris p.s. the hdf_browser is shipped as a save file. Not sure why...
Hello, On 12/04/13 11:22, David Fanning wrote: > Paul van Delst writes: > >>> I'd have to have a look. Can you send me a file of the type you think >>> doesn't work well with the software? I'll have a look and see how much >>> time I think it will take to make the software compatible with the file. >> >> Done. > > The netCDF4 file you sent me can be browsed with the H5_Browser supplied > with IDL. If the good folks at ExelisVis would give me the code for that > browser, I'd fix it so that it worked more intuitively. ;-) > > So, I guess the question is, do we need a better HDF5/netCDF4 browser or > do we just need a better way to get at the data in these files? I The latter. Browsing is nice, but being able to read the data simply, over and over again, in a production-like setting is the goal. Typically, I know what the variables names are. Once the data has been read into a structure/object, getting the particular variable is trivial (e.g. via a tag name match, or get_property method). > haven't yet looked at Mike Galloy's routines, but these are generally > extremely reliable for these kinds of things. ? Mike has netCDF4 readers? Crikey, I can't keep up! cheers, paulv
Hello, On 12/04/13 11:22, David Fanning wrote: > Paul van Delst writes: > >>> I'd have to have a look. Can you send me a file of the type you think >>> doesn't work well with the software? I'll have a look and see how much >>> time I think it will take to make the software compatible with the file. >> >> Done. > > The netCDF4 file you sent me can be browsed with the H5_Browser supplied > with IDL. If the good folks at ExelisVis would give me the code for that > browser, I'd fix it so that it worked more intuitively. ;-) > > So, I guess the question is, do we need a better HDF5/netCDF4 browser or > do we just need a better way to get at the data in these files? I > haven't yet looked at Mike Galloy's routines, but these are generally > extremely reliable for these kinds of things. Based on the to-and-fro about HDF5, I tried the following: IDL> result=h5_parse('amsua_metop-a.SpcCoeff.nc',/read_data) and bugger me if it didn't work, making all the groups accessible: IDL> help, result ** Structure <1e95a58>, 14 tags, length=33952, data length=33548, refs=1: _NAME STRING 'amsua_metop-a.SpcCoeff.nc' _ICONTYPE STRING 'hdf' _TYPE STRING 'GROUP' _FILE STRING 'amsua_metop-a.SpcCoeff.nc' _PATH STRING '/' _COMMENT STRING '' SPCCOEFF STRUCT -> <Anonymous> Array[1] WRITE_MODULE_HISTORY STRUCT -> <Anonymous> Array[1] CREATION_DATE_AND_TIME STRUCT -> <Anonymous> Array[1] RELEASE STRUCT -> <Anonymous> Array[1] VERSION STRUCT -> <Anonymous> Array[1] TITLE STRUCT -> <Anonymous> Array[1] HISTORY STRUCT -> <Anonymous> Array[1] COMMENT STRUCT -> <Anonymous> Array[1] IDL> help, result.spccoeff ** Structure <208a1e8>, 19 tags, length=33032, data length=32628, refs=2: _NAME STRING 'spccoeff' _ICONTYPE STRING '' _TYPE STRING 'GROUP' _FILE STRING 'amsua_metop-a.SpcCoeff.nc' _PATH STRING '/' _COMMENT STRING '' BAND_C1 STRUCT -> <Anonymous> Array[1] BAND_C2 STRUCT -> <Anonymous> Array[1] CHANNEL_FLAG STRUCT -> <Anonymous> Array[1] COSMIC_BACKGROUND_RADIANCE STRUCT -> <Anonymous> Array[1] FREQUENCY STRUCT -> <Anonymous> Array[1] PLANCK_C1 STRUCT -> <Anonymous> Array[1] PLANCK_C2 STRUCT -> <Anonymous> Array[1] SENSOR_CHANNEL STRUCT -> <Anonymous> Array[1] SOLAR_IRRADIANCE STRUCT -> <Anonymous> Array[1] WAVENUMBER STRUCT -> <Anonymous> Array[1] ACCOEFF STRUCT -> <Anonymous> Array[1] N_CHANNELS STRUCT -> <Anonymous> Array[1] SENSORINFO STRUCT -> <Anonymous> Array[1] IDL> help, result.spccoeff.accoeff ** Structure <1d538e8>, 13 tags, length=14608, data length=14424, refs=2: _NAME STRING 'accoeff' _ICONTYPE STRING '' _TYPE STRING 'GROUP' _FILE STRING 'amsua_metop-a.SpcCoeff.nc' _PATH STRING '/spccoeff' _COMMENT STRING '' A_EARTH STRUCT -> <Anonymous> Array[1] A_PLATFORM STRUCT -> <Anonymous> Array[1] A_SPACE STRUCT -> <Anonymous> Array[1] SENSOR_CHANNEL STRUCT -> <Anonymous> Array[1] N_CHANNELS STRUCT -> <Anonymous> Array[1] N_FOVS STRUCT -> <Anonymous> Array[1] SENSORINFO STRUCT -> <Anonymous> Array[1] IDL> help, result.spccoeff.accoeff.sensorinfo ** Structure <1e82768>, 20 tags, length=9976, data length=9876, refs=2: _NAME STRING 'sensorinfo' _ICONTYPE STRING '' _TYPE STRING 'GROUP' _FILE STRING 'amsua_metop-a.SpcCoeff.nc' _PATH STRING '/spccoeff/accoeff' _COMMENT STRING '' NOISE STRUCT -> <Anonymous> Array[1] POLARIZATION_TYPE STRUCT -> <Anonymous> Array[1] SATELLITE_NAME STRUCT -> <Anonymous> Array[1] SENSOR_CHANNEL STRUCT -> <Anonymous> Array[1] SENSOR_ID STRUCT -> <Anonymous> Array[1] SENSOR_NAME STRUCT -> <Anonymous> Array[1] SENSOR_TYPE STRUCT -> <Anonymous> Array[1] USE_FLAG STRUCT -> <Anonymous> Array[1] WMO_SATELLITE_ID STRUCT -> <Anonymous> Array[1] WMO_SENSOR_ID STRUCT -> <Anonymous> Array[1] DL_STRLEN STRUCT -> <Anonymous> Array[1] N_CHANNELS STRUCT -> <Anonymous> Array[1] N_FOVS STRUCT -> <Anonymous> Array[1] SL_STRLEN STRUCT -> <Anonymous> Array[1] And the data is there: IDL> help, result.spccoeff.frequency ** Structure <1ea1388>, 18 tags, length=856, data length=848, refs=2: _NAME STRING 'Frequency' _ICONTYPE STRING 'binary' _TYPE STRING 'DATASET' _FILE STRING 'amsua_metop-a.SpcCoeff.nc' _PATH STRING '/spccoeff' _DATA DOUBLE Array[15] _NDIMENSIONS LONG 1 _DIMENSIONS ULONG64 Array[1] _NELEMENTS ULONG64 15 _DATATYPE STRING 'H5T_FLOAT' _STORAGESIZE ULONG 8 _PRECISION LONG 64 _SIGN STRING '' LONG_NAME STRUCT -> <Anonymous> Array[1] DESCRIPTION STRUCT -> <Anonymous> Array[1] UNITS STRUCT -> <Anonymous> Array[1] _FILLVALUE STRUCT -> <Anonymous> Array[1] DIMENSION_LIST STRUCT -> <Anonymous> Array[1] IDL> print, result.spccoeff.frequency._data 23.800904 31.400728 50.300069 52.799890 53.596155 54.400633 54.940002 55.499802 57.290327 57.290327 57.290327 57.290327 57.290327 57.290327 88.997000 Yay. So I guess netcdf4 files are just hdf5 files with a fancier name? cheers, paulv
Paul van Delst writes: > Typically, I know what the variables names are. Once the data has been > read into a structure/object, getting the particular variable is trivial > (e.g. via a tag name match, or get_property method). It looks to me like H5_Parse can read this data file into a structure in a blink of an eye. ;-) Cheers, David -- David Fanning, Ph.D. Fanning Software Consulting, Inc. Coyote's Guide to IDL Programming: http://www.idlcoyote.com/ Sepore ma de ni thue. ("Perhaps thou speakest truth.")
Paul van Delst writes: > Based on the to-and-fro about HDF5, I tried the following: > > IDL> result=h5_parse('amsua_metop-a.SpcCoeff.nc',/read_data) > > and bugger me if it didn't work, making all the groups accessible: There you go! Job done. Two hours spent on research. Total economic appreciation for the week so far: $10. Cheers, David -- David Fanning, Ph.D. Fanning Software Consulting, Inc. Coyote's Guide to IDL Programming: http://www.idlcoyote.com/ Sepore ma de ni thue. ("Perhaps thou speakest truth.")
On 12/04/13 12:12, David Fanning wrote: > Paul van Delst writes: > >> Based on the to-and-fro about HDF5, I tried the following: >> >> IDL> result=h5_parse('amsua_metop-a.SpcCoeff.nc',/read_data) >> >> and bugger me if it didn't work, making all the groups accessible: > > There you go! Job done. Two hours spent on research. Total economic > appreciation for the week so far: $10. Jeez mate, that's not even minimum wage!
Paul van Delst writes: > Jeez mate, that's not even minimum wage! No, MacDonald's is going to be a step up. :-) Cheers, David -- David Fanning, Ph.D. Fanning Software Consulting, Inc. Coyote's Guide to IDL Programming: http://www.idlcoyote.com/ Sepore ma de ni thue. ("Perhaps thou speakest truth.")
Paul van Delst writes: > So I guess netcdf4 files are just hdf5 files with a fancier name? Guess so. Even H5F_IS_HDF5 returns true when passed the name of your netCDF4 file! Cheers, David -- David Fanning, Ph.D. Fanning Software Consulting, Inc. Coyote's Guide to IDL Programming: http://www.idlcoyote.com/ Sepore ma de ni thue. ("Perhaps thou speakest truth.")
On 12/4/13, 5:51 pm, David Fanning wrote: > Paul van Delst writes: > >> So I guess netcdf4 files are just hdf5 files with a fancier name? > > Guess so. Even H5F_IS_HDF5 returns true when passed the name of your > netCDF4 file! > > Cheers, > > David > > Yes, my understanding is netCDF 4 files are just HDF 5 files with some naming conventions. netCDF 4 is not a file type; it is an API. Mike -- Michael Galloy www.michaelgalloy.com Modern IDL: A Guide to IDL Programming (http://modernidl.idldev.com) Research Mathematician Tech-X Corporation