comp.lang.idl-pvwave archive
Messages from Usenet group comp.lang.idl-pvwave, compiled by Paulo Penteado

Home » Public Forums » archive » Error opening netCDF file
Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend 
Switch to threaded view of this topic Create a new topic Submit Reply
Error opening netCDF file [message #86932] Fri, 13 December 2013 14:55 Go to next message
dan.westervelt is currently offline  dan.westervelt
Messages: 4
Registered: December 2013
Junior Member
I am attempting to read in about ~100 or so netCDF files in a loop. I'm using ncdf_read.pro ( http://acmg.seas.harvard.edu/gamap/doc/by_alphabet/gamap_n.h tml#NCDF_READ). My code looks like this:

; read in met variables
ncdf_read, result_meteo, filename = dir1+sim4+dir4+dir3+'atmos.' $
+years(n)+'-'+string(fix(years(n))+4,format='(I4)')+'.ann.nc ', $
variables = ['netrad_toa']

; ; assign variables
net_rad_85(*,*)=result_meteo.netrad_toa
Undefine, result_meteo

I am getting this error at the assignment line above:
% Expression must be a structure in this context: RESULT_METEO.

Strangely, if I reset everything, and run again, the code gets a little further in the loop, and more of my 100 or so files are read in. If I repeat it enough times, I can get through the entire program without error, but it can take hours, which is not good.

Each file is about 18M, so I don't think I'm approaching a memory limit? My first guess was memory error or memory leak, but I've tried undefining everything, calling heap_gc, etc, with no luck. Any idea what I'm doing wrong here?

Thanks
Dan
Re: Error opening netCDF file [message #86949 is a reply to message #86932] Mon, 16 December 2013 15:58 Go to previous messageGo to next message
dan.westervelt is currently offline  dan.westervelt
Messages: 4
Registered: December 2013
Junior Member
To follow up, this is with IDL 8.0.1 and netcdf 4.2. I define the net_rad_85 array earlier in the program (not shown) to have the same dimensions as netrad_toa variable in the structure.

On Friday, December 13, 2013 5:55:53 PM UTC-5, Dan Westervelt wrote:
> I am attempting to read in about ~100 or so netCDF files in a loop. I'm using ncdf_read.pro ( http://acmg.seas.harvard.edu/gamap/doc/by_alphabet/gamap_n.h tml#NCDF_READ). My code looks like this:
>
>
>
> ; read in met variables
>
> ncdf_read, result_meteo, filename = dir1+sim4+dir4+dir3+'atmos.' $
>
> +years(n)+'-'+string(fix(years(n))+4,format='(I4)')+'.ann.nc ', $
>
> variables = ['netrad_toa']
>
>
>
> ; ; assign variables
>
> net_rad_85(*,*)=result_meteo.netrad_toa
>
> Undefine, result_meteo
>
>
>
> I am getting this error at the assignment line above:
>
> % Expression must be a structure in this context: RESULT_METEO.
>
>
>
> Strangely, if I reset everything, and run again, the code gets a little further in the loop, and more of my 100 or so files are read in. If I repeat it enough times, I can get through the entire program without error, but it can take hours, which is not good.
>
>
>
> Each file is about 18M, so I don't think I'm approaching a memory limit? My first guess was memory error or memory leak, but I've tried undefining everything, calling heap_gc, etc, with no luck. Any idea what I'm doing wrong here?
>
>
>
> Thanks
>
> Dan
Re: Error opening netCDF file [message #86950 is a reply to message #86932] Mon, 16 December 2013 16:47 Go to previous messageGo to next message
Phillip Bitzer is currently offline  Phillip Bitzer
Messages: 223
Registered: June 2006
Senior Member
Weird. I don't have ncdf_read (and am denied access when I try to clone the git repo), but do they close the file (NCDF_CLOSE) when finished reading it in? I seem to recall coming across something similar where this was the problem.

Perhaps setting some sort of debug "STOP" might help:
IF N_TAGS(result_meteo) EQ 0 THEN STOP ELSE net_rad_85=result_meteo.netrad_toa

(Notice I've gotten rid of the (*, *) on the LHS - see http://www.idlcoyote.com/code_tips/asterisk.html)

When it stops, you might be able to look around and find if something has gone screwy.
Re: Error opening netCDF file [message #86952 is a reply to message #86950] Mon, 16 December 2013 18:46 Go to previous messageGo to next message
dan.westervelt is currently offline  dan.westervelt
Messages: 4
Registered: December 2013
Junior Member
Thanks for your response!

Someone has posted the code here: http://coco.atmos.washington.edu/ion_script/tools/ncdf_read. pro

They do indeed call NCDF_CLOSE, it looks like.

I'm running in interactive mode (with the IDL prompt). Would the STOP really do anything in that case? I am able to print variables from the IDL prompt right after the code crashes. The only odd thing I found was the value of result_meteo = -1 and thus result_meteo.netrad_toa is non-existent.

Thanks for the tip on the asterisks - didn't know that!

On Monday, December 16, 2013 7:47:47 PM UTC-5, Phillip Bitzer wrote:
> Weird. I don't have ncdf_read (and am denied access when I try to clone the git repo), but do they close the file (NCDF_CLOSE) when finished reading it in? I seem to recall coming across something similar where this was the problem.
>
>
>
> Perhaps setting some sort of debug "STOP" might help:
>
> IF N_TAGS(result_meteo) EQ 0 THEN STOP ELSE net_rad_85=result_meteo.netrad_toa
>
>
>
> (Notice I've gotten rid of the (*, *) on the LHS - see http://www.idlcoyote.com/code_tips/asterisk.html)
>
>
>
> When it stops, you might be able to look around and find if something has gone screwy.
Re: Error opening netCDF file [message #86954 is a reply to message #86932] Mon, 16 December 2013 22:50 Go to previous messageGo to next message
Fabzi is currently offline  Fabzi
Messages: 305
Registered: July 2010
Senior Member
On 13.12.2013 23:55, dan.westervelt@gmail.com wrote:
> read in about ~100 or so netCDF files in a loop. I'm using ncdf_read.pro

have you tried David's NCDF_FILE object? It is not very efficient in
opening many files but it is stable.

http://www.idlcoyote.com/programs/ncdf_file__define.pro

Cheers

Fab
Re: Error opening netCDF file [message #86964 is a reply to message #86954] Tue, 17 December 2013 10:36 Go to previous messageGo to next message
dan.westervelt is currently offline  dan.westervelt
Messages: 4
Registered: December 2013
Junior Member
Thanks for the suggestion. Just tried this - code crashes at the same point as with NCDF_READ. Pops up a dialog box with NC_ERROR = -31.

Maybe the netCDF files themselves are corrupted somehow?

On Tuesday, December 17, 2013 1:50:23 AM UTC-5, Fabien wrote:
> On 13.12.2013 23:55, dan.westervelt@gmail.com wrote:
>
>> read in about ~100 or so netCDF files in a loop. I'm using ncdf_read.pro
>
>
>
> have you tried David's NCDF_FILE object? It is not very efficient in
>
> opening many files but it is stable.
>
>
>
> http://www.idlcoyote.com/programs/ncdf_file__define.pro
>
>
>
> Cheers
>
>
>
> Fab
Re: Error opening netCDF file [message #86968 is a reply to message #86964] Tue, 17 December 2013 13:38 Go to previous message
David Fanning is currently offline  David Fanning
Messages: 11724
Registered: August 2001
Senior Member
Dan Westervelt writes:

> Thanks for the suggestion. Just tried this - code crashes at the same point as with NCDF_READ. Pops up a dialog box with NC_ERROR = -31.
>
> Maybe the netCDF files themselves are corrupted somehow?

You can find nCDF_isValidFile here:

http://www.idlcoyote.com/programs/ncdf_isvalidfile.pro

It might shed some light.

; CALLING SEQUENCE:
;
; result = NCDF_IsValidFile(filename)
;
; INPUTS:
;
; filename: The name of a filename to open to see if it is a valid
netCDF file.
;
; RETURN VALUE:
;
; result: A 1 if the file can be opened as a netCDF file. A 0
otherwise.

Cheers,

David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.idlcoyote.com/
Sepore ma de ni thue. ("Perhaps thou speakest truth.")
  Switch to threaded view of this topic Create a new topic Submit Reply
Previous Topic: PNG output quality degraded after RHEL6 patch
Next Topic: Issue with long integer arrays

-=] Back to Top [=-
[ Syndicate this forum (XML) ] [ RSS ] [ PDF ]

Current Time: Wed Oct 08 15:11:20 PDT 2025

Total time taken to generate the page: 0.00517 seconds