| Re: Can't get out of define mode in ncdf, not specific to variable. [message #82058] |
Fri, 16 November 2012 19:35  |
Fabzi
Messages: 305
Registered: July 2010
|
Senior Member |
|
|
On 11/17/2012 12:25 AM, Paul van Delst wrote:
> How many variables are you writing? What version of IDL? What version of
> netCDF?
Also, before NCDF4 (7.1 patched and IDL8+), files could not be larger
than 2Gb. If you attempt to define to many variables that will make the
file larger than 2Gb (NCDF knows how big a file is going to be since you
define the size of your variables first) this could also cause such an
error...
> If you are writing lots of variables, there used to be a limit to the
> total number (in older netcdf's at least. I think it was 300).
>
> Complete shot in the dark.....
>
>
> On 11/15/12 17:57, neleh.mac wrote:
>> So I am using IDL to read in ncdf data, and write out (to a new
>> file)
>> the processed data. So I go to define mode, set up the variable, add
>> some attributes, switch to data mode, stick in the data, and then go
>> back to define for the next one.
>>
>> e.g.
>> ncid = ncdf_create(file_name, /CLOBBER)
>> londim = ncdf_dimdef(ncid, 'longitude_dim', 144)
>> latdim = ncdf_dimdef(ncid, 'latitude_dim', 91)
>>
>> vid = ncdf_vardef(ncid, 'emission', [londim,latdim], /float)
>> ncdf_attput, ncid, vid, 'long_name', "Some emissions"
>> ncdf_control, ncid, /ENDEF
>> ncdf_varput, ncid, vid, emiss_dat
>> ncdf_control, ncid, /REDEF
>>
>> vid = ncdf_vardef(ncid, 'emission2', [londim,latdim], /float)
>> ncdf_attput, ncid, vid, 'long_name', "Some emissions 2"
>> ncdf_control, ncid, /ENDEF
>> ncdf_varput, ncid, vid, emiss_dat2
>> ncdf_control, ncid, /REDEF
>>
>> ... etc.
>>
>> This works fine for several variables, over and over, until a certain
>> point in the code and then at it will not get out of define mode. I
>> commented out that particular part of the code where it was falling
>> over, and got it to work fine (but my output file now has one less
>> variable).
>>
>> Now the weird part.
>>
>> If I take the chunk of code for the variable that it falls at, and
>> copy it back in further up the code, it works through it fine. It
>> only fails when it gets to that line again, this time on a variable
>> it wrote fine first time round.
>>
>> It's like it's failing after going into define mode a certain number
>> of times, it's not specific to any variable I'm trying to write, it
>> just always fails on the nth one, regardless of what it is. I've
>> swopped round the order it writes them to the file, and they all get
>> written, just not if they happen to be at a certain point in the
>> order. I have tried removing attributes from the variables but this
>> doesn't seem to have any effect. I am very confused by this, please
>> can you help?
|
|
|
|
| Re: Can't get out of define mode in ncdf, not specific to variable. [message #82059 is a reply to message #82058] |
Fri, 16 November 2012 15:25  |
Paul Van Delst[1]
Messages: 1157
Registered: April 2002
|
Senior Member |
|
|
How many variables are you writing? What version of IDL? What version of
netCDF?
If you are writing lots of variables, there used to be a limit to the
total number (in older netcdf's at least. I think it was 300).
Complete shot in the dark.....
On 11/15/12 17:57, neleh.mac wrote:
> So I am using IDL to read in ncdf data, and write out (to a new
> file)
> the processed data. So I go to define mode, set up the variable, add
> some attributes, switch to data mode, stick in the data, and then go
> back to define for the next one.
>
> e.g.
> ncid = ncdf_create(file_name, /CLOBBER)
> londim = ncdf_dimdef(ncid, 'longitude_dim', 144)
> latdim = ncdf_dimdef(ncid, 'latitude_dim', 91)
>
> vid = ncdf_vardef(ncid, 'emission', [londim,latdim], /float)
> ncdf_attput, ncid, vid, 'long_name', "Some emissions"
> ncdf_control, ncid, /ENDEF
> ncdf_varput, ncid, vid, emiss_dat
> ncdf_control, ncid, /REDEF
>
> vid = ncdf_vardef(ncid, 'emission2', [londim,latdim], /float)
> ncdf_attput, ncid, vid, 'long_name', "Some emissions 2"
> ncdf_control, ncid, /ENDEF
> ncdf_varput, ncid, vid, emiss_dat2
> ncdf_control, ncid, /REDEF
>
> ... etc.
>
> This works fine for several variables, over and over, until a certain
> point in the code and then at it will not get out of define mode. I
> commented out that particular part of the code where it was falling
> over, and got it to work fine (but my output file now has one less
> variable).
>
> Now the weird part.
>
> If I take the chunk of code for the variable that it falls at, and
> copy it back in further up the code, it works through it fine. It
> only fails when it gets to that line again, this time on a variable
> it wrote fine first time round.
>
> It's like it's failing after going into define mode a certain number
> of times, it's not specific to any variable I'm trying to write, it
> just always fails on the nth one, regardless of what it is. I've
> swopped round the order it writes them to the file, and they all get
> written, just not if they happen to be at a certain point in the
> order. I have tried removing attributes from the variables but this
> doesn't seem to have any effect. I am very confused by this, please
> can you help?
|
|
|
|
| Re: Can't get out of define mode in ncdf, not specific to variable. [message #82061 is a reply to message #82059] |
Fri, 16 November 2012 10:46 |
Chip Helms
Messages: 24
Registered: November 2012
|
Junior Member |
|
|
This is a bit of a shot in the dark (I don't currently have easy access to a copy of IDL, hopefully that'll change in the next couple days) but this reminds me of when I forget to use free_lun when opening files in a loop. Could it be possible that IDL is storing the history of the modes (i.e. def and data) and simply runs out of reference space?
You might try using 'ncdf_control, cdfid, /sync' to update the file on the disk before you use 'ncdf_control, cdfid, /redef'. When I've made netcdf files in the past I've always defined everything before moving on to store the data, so I've never run into this sort of situation myself. I look forward to hearing Exelis' explaination.
Cheers,
Chip
On Thursday, November 15, 2012 5:57:17 PM UTC-5, neleh.mac wrote:
> So I am using IDL to read in ncdf data, and write out (to a new file) the processed data. So I go to define mode, set up the variable, add some attributes, switch to data mode, stick in the data, and then go back to define for the next one.
>
>
>
> e.g.
>
> ncid = ncdf_create(file_name, /CLOBBER)
>
> londim = ncdf_dimdef(ncid, 'longitude_dim', 144)
>
> latdim = ncdf_dimdef(ncid, 'latitude_dim', 91)
>
>
>
> vid = ncdf_vardef(ncid, 'emission', [londim,latdim], /float)
>
> ncdf_attput, ncid, vid, 'long_name', "Some emissions"
>
> ncdf_control, ncid, /ENDEF
>
> ncdf_varput, ncid, vid, emiss_dat
>
> ncdf_control, ncid, /REDEF
>
>
>
> vid = ncdf_vardef(ncid, 'emission2', [londim,latdim], /float)
>
> ncdf_attput, ncid, vid, 'long_name', "Some emissions 2"
>
> ncdf_control, ncid, /ENDEF
>
> ncdf_varput, ncid, vid, emiss_dat2
>
> ncdf_control, ncid, /REDEF
>
>
>
> ... etc.
>
>
>
> This works fine for several variables, over and over, until a certain point in the code and then at it will not get out of define mode. I commented out that particular part of the code where it was falling over, and got it to work fine (but my output file now has one less variable).
>
>
>
> Now the weird part.
>
>
>
> If I take the chunk of code for the variable that it falls at, and copy it back in further up the code, it works through it fine. It only fails when it gets to that line again, this time on a variable it wrote fine first time round.
>
>
>
> It's like it's failing after going into define mode a certain number of times, it's not specific to any variable I'm trying to write, it just always fails on the nth one, regardless of what it is. I've swopped round the order it writes them to the file, and they all get written, just not if they happen to be at a certain point in the order. I have tried removing attributes from the variables but this doesn't seem to have any effect. I am very confused by this, please can you help?
|
|
|
|
|
|
| Re: Can't get out of define mode in ncdf, not specific to variable. [message #82132 is a reply to message #82058] |
Mon, 19 November 2012 11:46 |
neleh.mac
Messages: 3
Registered: November 2012
|
Junior Member |
|
|
Thanks for your suggestions.
Ok, I tried Chip's suggestion of syncing, but I still run into the problem.
Paul, I am writing around 30 variables and have around 20 dimensions defined, so I don't think this is the problem.
However, the file I am writing will be just larger than 2Gb, so Fab this looks like the most likely candidate for the problem. However, I have read that if I am using netcdf 3.6 or higher it should be ok. (I am using netcdf 3.6.3 and IDL 8). Is there something else I'm missing with the setup?
FYI, I also tried writing and closing the file part way through so that it was opened up again and less than 2Gb was appended and written second time round, but I still encounter the error.
Still confused, if anyone has any more suggestions I'd really appreciate it :-)
On Friday, November 16, 2012 10:35:45 PM UTC-5, Fab wrote:
> On 11/17/2012 12:25 AM, Paul van Delst wrote:
>
>> How many variables are you writing? What version of IDL? What version of
>
>> netCDF?
>
>
>
>
>
> Also, before NCDF4 (7.1 patched and IDL8+), files could not be larger
>
> than 2Gb. If you attempt to define to many variables that will make the
>
> file larger than 2Gb (NCDF knows how big a file is going to be since you
>
> define the size of your variables first) this could also cause such an
>
> error...
>
>
>
>
>
>
>
>> If you are writing lots of variables, there used to be a limit to the
>
>> total number (in older netcdf's at least. I think it was 300).
>
>>
>
>> Complete shot in the dark.....
>
>>
>
>>
>
>> On 11/15/12 17:57, neleh.mac wrote:
>
>>> So I am using IDL to read in ncdf data, and write out (to a new
>
>>> file)
>
>>> the processed data. So I go to define mode, set up the variable, add
>
>>> some attributes, switch to data mode, stick in the data, and then go
>
>>> back to define for the next one.
>
>>>
>
>>> e.g.
>
>>> ncid = ncdf_create(file_name, /CLOBBER)
>
>>> londim = ncdf_dimdef(ncid, 'longitude_dim', 144)
>
>>> latdim = ncdf_dimdef(ncid, 'latitude_dim', 91)
>
>>>
>
>>> vid = ncdf_vardef(ncid, 'emission', [londim,latdim], /float)
>
>>> ncdf_attput, ncid, vid, 'long_name', "Some emissions"
>
>>> ncdf_control, ncid, /ENDEF
>
>>> ncdf_varput, ncid, vid, emiss_dat
>
>>> ncdf_control, ncid, /REDEF
>
>>>
>
>>> vid = ncdf_vardef(ncid, 'emission2', [londim,latdim], /float)
>
>>> ncdf_attput, ncid, vid, 'long_name', "Some emissions 2"
>
>>> ncdf_control, ncid, /ENDEF
>
>>> ncdf_varput, ncid, vid, emiss_dat2
>
>>> ncdf_control, ncid, /REDEF
>
>>>
>
>>> ... etc.
>
>>>
>
>>> This works fine for several variables, over and over, until a certain
>
>>> point in the code and then at it will not get out of define mode. I
>
>>> commented out that particular part of the code where it was falling
>
>>> over, and got it to work fine (but my output file now has one less
>
>>> variable).
>
>>>
>
>>> Now the weird part.
>
>>>
>
>>> If I take the chunk of code for the variable that it falls at, and
>
>>> copy it back in further up the code, it works through it fine. It
>
>>> only fails when it gets to that line again, this time on a variable
>
>>> it wrote fine first time round.
>
>>>
>
>>> It's like it's failing after going into define mode a certain number
>
>>> of times, it's not specific to any variable I'm trying to write, it
>
>>> just always fails on the nth one, regardless of what it is. I've
>
>>> swopped round the order it writes them to the file, and they all get
>
>>> written, just not if they happen to be at a certain point in the
>
>>> order. I have tried removing attributes from the variables but this
>
>>> doesn't seem to have any effect. I am very confused by this, please
>
>>> can you help?
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
