| Re: NetCDF and empty variables [message #66124] |
Mon, 20 April 2009 05:18  |
renaud.dussurget
Messages: 8
Registered: April 2008
|
Junior Member |
|
|
On 20 avr, 01:08, Mark <mark.h...@gmail.com> wrote:
> On Apr 18, 7:34 am, renaud.dussur...@gmail.com wrote:
>
>> Well, as I've said, it is much nastier than this.
>> varid returned by NCDF_VARID is correct
>> (the variable exists in the netcdf file and this
>> is its ID). The return of NCDF_VARINQ also suppose
>> that this variable should have 3 dimensions
>> (as, I think,the creator of this file
>> intended it...) But there is simply NO DATA
>> within this variable, and there is no way
>> to check this before trying to get the
>> data from the file.
>
> To the best of my knowledge the only way you can get an "empty"
> variable in a netCDF file is for that variable to depend on the
> unlimited dimension, and for the unilimited dimension to have 0
> records. You can check for the ID of the unlimited dimension with
> NCDF_INQUIRE and then determine its size with NCDF_DIMINQ. You can
> determine the dimensions associated with each variable with
> NCDF_VARINQ. And you can use the ncdump utility (outside IDL) to give
> you a user-friendly listing of the file structure and contents.
NCDF_INQUIRE and NCDF_DIMINQ just give what the properties of the
variable should be, and not what it really is in the file (i.e. no
data)
|
|
|
|
| Re: NetCDF and empty variables [message #66125 is a reply to message #66124] |
Mon, 20 April 2009 05:11  |
renaud.dussurget
Messages: 8
Registered: April 2008
|
Junior Member |
|
|
Here is the file:
ftp://ftp.ifremer.fr/ifremer/argo/dac/coriolis/69003/69003_p rof.nc
Renaud
On 20 avr, 11:25, Reimar Bauer <R.Ba...@fz-juelich.de> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> renaud.dussur...@gmail.com schrieb:
>
>> Hi,
>> I've been trying reading a NC file with David Fanning's (very handy!)
>> NCDF_DATA object, and it failed reading a file while parsing it.
>> After trying to understand where it was crashing, I've realised that
>> this variable was empty (though it is supposed to have 3 dimensions)
>> and that IDL NCDF_VARGET function was not able to deal with this
>> problem...
>
> Can you share an example file? I am interested if my read_ncdf routine
> crashes too.
>
> cheers
> Reimar
>
>
>
>
>
>> An example :
>
>> IDL>ncid = NCDF_OPEN('myFile.nc')
>> IDL>varId = NCDF_VARID(ncid,'HISTORY_INSTITUTION')
>> IDL>out=NCDF_VARINQ(ncid, VarId)
>> IDL>help, out, /str
>> ** Structure <a9c098>, 5 tags, length=56, data length=52, refs=1:
>> NAME STRING 'HISTORY_INSTITUTION'
>> DATATYPE STRING 'CHAR'
>> NDIMS LONG 3
>> NATTS LONG 3
>> DIM LONG Array[3]
>
>> IDL>print, out.dim
>> 6 8 12
>
>> IDL>NCDF_VARGET, ncid, varId, data, COUNT=cnt
>> % Array dimensions must be greater than 0.
>
>> I have found a simple way to deal with this, however I don't really
>> like it...
>> IDL>cmd = 'NCDF_VarGet, ncid, varId, data
>> IDL>res= EXECUTE(cmd)
>> % Array dimensions must be greater than 0.
>
>> And then in the code, write something like this
>> IF (res EQ 0) THEN data = -1
>
>> Does anyone has a better solution???
>
>> Thanks
>> Bye
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.9 (GNU/Linux)
> Comment: Using GnuPG with SUSE -http://enigmail.mozdev.org
>
> iEYEARECAAYFAknsP3IACgkQ5aOc3Q9hk/kaowCggAblq8UPEa2u5UpvS4Gk Ym39
> kjwAnihW7lT92JMA/JDNyvNOExlRfGI/
> =7Ip4
> -----END PGP SIGNATURE-----
|
|
|
|
| Re: NetCDF and empty variables [message #66128 is a reply to message #66125] |
Mon, 20 April 2009 02:25 |
R.Bauer
Messages: 1424
Registered: November 1998
|
Senior Member |
|
|
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
renaud.dussurget@gmail.com schrieb:
> Hi,
> I've been trying reading a NC file with David Fanning's (very handy!)
> NCDF_DATA object, and it failed reading a file while parsing it.
> After trying to understand where it was crashing, I've realised that
> this variable was empty (though it is supposed to have 3 dimensions)
> and that IDL NCDF_VARGET function was not able to deal with this
> problem...
Can you share an example file? I am interested if my read_ncdf routine
crashes too.
cheers
Reimar
>
> An example :
>
> IDL>ncid = NCDF_OPEN('myFile.nc')
> IDL>varId = NCDF_VARID(ncid,'HISTORY_INSTITUTION')
> IDL>out=NCDF_VARINQ(ncid, VarId)
> IDL>help, out, /str
> ** Structure <a9c098>, 5 tags, length=56, data length=52, refs=1:
> NAME STRING 'HISTORY_INSTITUTION'
> DATATYPE STRING 'CHAR'
> NDIMS LONG 3
> NATTS LONG 3
> DIM LONG Array[3]
>
> IDL>print, out.dim
> 6 8 12
>
> IDL>NCDF_VARGET, ncid, varId, data, COUNT=cnt
> % Array dimensions must be greater than 0.
>
> I have found a simple way to deal with this, however I don't really
> like it...
> IDL>cmd = 'NCDF_VarGet, ncid, varId, data
> IDL>res= EXECUTE(cmd)
> % Array dimensions must be greater than 0.
>
> And then in the code, write something like this
> IF (res EQ 0) THEN data = -1
>
> Does anyone has a better solution???
>
> Thanks
> Bye
>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org
iEYEARECAAYFAknsP3IACgkQ5aOc3Q9hk/kaowCggAblq8UPEa2u5UpvS4Gk Ym39
kjwAnihW7lT92JMA/JDNyvNOExlRfGI/
=7Ip4
-----END PGP SIGNATURE-----
|
|
|
|
| Re: NetCDF and empty variables [message #66129 is a reply to message #66128] |
Sun, 19 April 2009 16:08 |
Mark[1]
Messages: 66
Registered: February 2008
|
Member |
|
|
On Apr 18, 7:34 am, renaud.dussur...@gmail.com wrote:
> Well, as I've said, it is much nastier than this.
> varid returned by NCDF_VARID is correct
> (the variable exists in the netcdf file and this
> is its ID). The return of NCDF_VARINQ also suppose
> that this variable should have 3 dimensions
> (as, I think,the creator of this file
> intended it...) But there is simply NO DATA
> within this variable, and there is no way
> to check this before trying to get the
> data from the file.
To the best of my knowledge the only way you can get an "empty"
variable in a netCDF file is for that variable to depend on the
unlimited dimension, and for the unilimited dimension to have 0
records. You can check for the ID of the unlimited dimension with
NCDF_INQUIRE and then determine its size with NCDF_DIMINQ. You can
determine the dimensions associated with each variable with
NCDF_VARINQ. And you can use the ncdump utility (outside IDL) to give
you a user-friendly listing of the file structure and contents.
Yes, it's a pity IDL doesn't support empty arrays, isn't it?
|
|
|
|
| Re: NetCDF and empty variables [message #66134 is a reply to message #66129] |
Fri, 17 April 2009 12:34 |
renaud.dussurget
Messages: 8
Registered: April 2008
|
Junior Member |
|
|
On 17 avr, 17:55, liamgum...@gmail.com wrote:
> On Apr 17, 8:18 am, renaud.dussur...@gmail.com wrote:
>
>
>
>> IDL>ncid = NCDF_OPEN('myFile.nc')
>> IDL>varId = NCDF_VARID(ncid,'HISTORY_INSTITUTION')
>> IDL>out=NCDF_VARINQ(ncid, VarId)
>> IDL>help, out, /str
>> ** Structure <a9c098>, 5 tags, length=56, data length=52, refs=1:
>> NAME STRING 'HISTORY_INSTITUTION'
>> DATATYPE STRING 'CHAR'
>> NDIMS LONG 3
>> NATTS LONG 3
>> DIM LONG Array[3]
>
>> IDL>print, out.dim
>> 6 8 12
>
>> IDL>NCDF_VARGET, ncid, varId, data, COUNT=cnt
>> % Array dimensions must be greater than 0.
>
> Where is the array CNT defined?
>
> Have you tried something like this?
>
> PRO TEST
> ncid = ncdf_open('myfile.nc')
> varid = ncdf_varid(ncid, 'HISTORY_INSTITUTION')
> if (varid eq -1) then message, 'Variable does not exist'
> ncdf_varget, ncid, varid, value
> help, value
> END
>
> If the code gets past the check for (varid eq -1) and it still
> crashes, then something very odd is going on, and I think you have a
> reason to contact ITTVIS technical support.
>
> Liam.
> Practical IDL Programminghttp://www.gumley.com/
Hello
Well, as I've said, it is much nastier than this.
varid returned by NCDF_VARID is correct (the variable exists in the
netcdf file and this is its ID)
The return of NCDF_VARINQ also suppose that this variable should have
3 dimensions (as, I think,the creator of this file intended it...)
But there is simply NO DATA within this variable, and there is no way
to check this before trying to get the data from the file.
When I do a ncdump of this file I get this
netcdf 69003_prof {
dimensions:
DATE_TIME = 14 ;
STRING256 = 256 ;
STRING64 = 64 ;
STRING32 = 32 ;
STRING16 = 16 ;
STRING8 = 8 ;
STRING4 = 4 ;
STRING2 = 2 ;
N_PROF = 16 ;
N_PARAM = 2 ;
N_LEVELS = 94 ;
N_CALIB = 1 ;
N_HISTORY = UNLIMITED ; // (0 currently)
variables:
[...]
char HISTORY_INSTITUTION(N_HISTORY, N_PROF, STRING4) ;
HISTORY_INSTITUTION:long_name = "Institution which
performed action" ;
HISTORY_INSTITUTION:conventions = "Argo reference
table 4" ;
HISTORY_INSTITUTION:_FillValue = " " ;
[...]
data:
}
-> so there is no data within this variable.
It seems that all "HISTORY_*" variables within this file have been
forgotten... We cannot think that NetCDF file will always be perfectly
defined as they should be, so here is our problem...
If I repeat this operation with Matlab, using the ncload command, I
have this:
>> ncload 'myFile.nc'
>> whos
[...]
HISTORY_INSTITUTION 0x0 0 double
[...]
So Matlab is able to detect that the variable exists,even if it is
empty, and does not crash reading it.
But, anyway, the EXECUTE() trick works well, as I've been able to read
my whole data set without any problem.
I think I'll write a mail to ITT tech support to work around this bug.
Bye
|
|
|
|
| Re: NetCDF and empty variables [message #66138 is a reply to message #66134] |
Fri, 17 April 2009 08:55 |
liamgumley
Messages: 74
Registered: June 2005
|
Member |
|
|
On Apr 17, 8:18 am, renaud.dussur...@gmail.com wrote:
> IDL>ncid = NCDF_OPEN('myFile.nc')
> IDL>varId = NCDF_VARID(ncid,'HISTORY_INSTITUTION')
> IDL>out=NCDF_VARINQ(ncid, VarId)
> IDL>help, out, /str
> ** Structure <a9c098>, 5 tags, length=56, data length=52, refs=1:
> NAME STRING 'HISTORY_INSTITUTION'
> DATATYPE STRING 'CHAR'
> NDIMS LONG 3
> NATTS LONG 3
> DIM LONG Array[3]
>
> IDL>print, out.dim
> 6 8 12
>
> IDL>NCDF_VARGET, ncid, varId, data, COUNT=cnt
> % Array dimensions must be greater than 0.
Where is the array CNT defined?
Have you tried something like this?
PRO TEST
ncid = ncdf_open('myfile.nc')
varid = ncdf_varid(ncid, 'HISTORY_INSTITUTION')
if (varid eq -1) then message, 'Variable does not exist'
ncdf_varget, ncid, varid, value
help, value
END
If the code gets past the check for (varid eq -1) and it still
crashes, then something very odd is going on, and I think you have a
reason to contact ITTVIS technical support.
Liam.
Practical IDL Programming
http://www.gumley.com/
|
|
|
|
|
|
| Re: NetCDF and empty variables [message #66214 is a reply to message #66124] |
Mon, 20 April 2009 17:11 |
Mark[1]
Messages: 66
Registered: February 2008
|
Member |
|
|
On Apr 21, 12:18 am, renaud.dussur...@gmail.com wrote:
> NCDF_INQUIRE and NCDF_DIMINQ just give what the
> properties of the variable should be, and not what
> it really is in the file (i.e. no
> data)
To misquote a very wise man who wasn't talking about NetCDF files,
what is is, what isn't isn't, there is no should.
Perhaps you could put a copy of the file on an FTP server, so we could
look at it. Or if it's small, maybe you could convert it to CDL (text)
format and post that on the group.
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
