comp.lang.idl-pvwave archive
Messages from Usenet group comp.lang.idl-pvwave, compiled by Paulo Penteado

Home » Public Forums » archive » Re: ncdf + x-connection + many files = crazy.
Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend 
Switch to threaded view of this topic Create a new topic Submit Reply
Re: ncdf + x-connection + many files = crazy. [message #74916] Wed, 09 February 2011 01:21 Go to next message
Maarten[1] is currently offline  Maarten[1]
Messages: 176
Registered: November 2005
Senior Member
On Feb 9, 9:56 am, Reimar Bauer <R.Ba...@fz-juelich.de> wrote:
>> crash_ncdf
>> % Loaded DLM: NCDF.
>> i =       819132
>> ncdf_fid =        65536
>
> strange it is for me always  "ncdf_fid =        65536"
>
> I can reproduce that number

The nc_open documentation just says it is an int. To me that means 32-
bit, but could they have compiled the library with 16-bit ints?

Maarten
Re: ncdf + x-connection + many files = crazy. [message #74917 is a reply to message #74916] Wed, 09 February 2011 00:56 Go to previous messageGo to next message
R.Bauer is currently offline  R.Bauer
Messages: 1424
Registered: November 1998
Senior Member
>
> Hi Matt
>
> I got it too, and I remember a similiar test case for a old fixed bug.
> I'll search for that. May be it is a regression.
>
>
> crash_ncdf
> % Loaded DLM: NCDF.
> i = 819132
> ncdf_fid = 65536


strange it is for me always "ncdf_fid = 65536"

I can reproduce that number

Reimar
Re: ncdf + x-connection + many files = crazy. [message #74918 is a reply to message #74917] Wed, 09 February 2011 00:39 Go to previous messageGo to next message
R.Bauer is currently offline  R.Bauer
Messages: 1424
Registered: November 1998
Senior Member
Am 08.02.2011 21:37, schrieb Matt:
> Hey all,
>
> I've got a great riddle for you today. (Long story short, can you run
> the test code included below to completion on your machine?) I've
> already filed a bug report, but I want to see if other people can
> reproduce these errors.

Hi Matt

I got it too, and I remember a similiar test case for a old fixed bug.
I'll search for that. May be it is a regression.


crash_ncdf
% Loaded DLM: NCDF.
i = 819132
ncdf_fid = 65536
% CRASH_NCDF: NCDF_OPEN: Unable to open the file "./sample.nc".
(NC_ERROR=-31)
% Execution halted at: CRASH_NCDF 41


** NCDF - IDL NetCDF support (loaded)
Version: 4.1.1, Build Date: JUN 18 2010, Source: ITT Visual
Information Solutions

IDL> help, !version,/str
** Structure !VERSION, 8 tags, length=104, data length=100:
ARCH STRING 'x86_64'
OS STRING 'linux'
OS_FAMILY STRING 'unix'
OS_NAME STRING 'linux'
RELEASE STRING '8.0'
BUILD_DATE STRING 'Jun 18 2010'
MEMORY_BITS INT 64
FILE_OFFSET_BITS
INT 64


Linux sunshine 2.6.32-28-generic #55-Ubuntu SMP Mon Jan 10 23:42:43 UTC
2011 x86_64 GNU/Linux

Reimar

>
> I'm experiencing machine/version-dependent, non-deterministic behavior
> that seems to present itself orthogonal to the triggering mechanism.
> In other words, I've got a crazy-making problem.
>
> I experienced this in working code, but through patient debugging,
> I've pared this down my until I've got a relatively small program that
> demonstrates the problem behavior.
>
> The undesirable behavior is a NCDF_OPEN error (NC_ERROR=-31) after
> some *random* number of open/close iterations on a netcdf file that
> was created with IDL (or possibly a segfault core dump on other
> machines.). The second feature is that I don't see this behavior
> until I make an x-connection of some sort (window, /free, or even a
> "device, depth=depth" will cause errors)
>
> Machine 1: (only has idl 7.0.1 64bit installed)
> savoie@snow:~/tmp> uname -a
> Linux snow 2.6.27.54-0.2-default #1 SMP 2010-10-19 18:40:07 +0200
> x86_64 x86_64 x86_64 GNU/Linux
>
> Machine 2 (fails with 32 bit idl7.0.1 & idl6.4):
> savoie@snowblower:~/tmp> uname -a
> Linux snowblower 2.6.34.7-0.5-default #1 SMP 2010-10-25 08:40:12 +0200
> i686 i686 i386 GNux
>
>
> All the code does is open and close a netcdf file, many, many times.
> But after an x-connection (via "window, /free"), the code errors out
> at a different spot in the iteration.
>
>
> Below is the output I get on the problem machines and below that, the
> source code for you to try at home. For extra credit, modify the code
> to open a netcdf file created with a different program than IDL, NCO
> for example.
>
>
> --OUTPUT--------------------------------------
> savoie@snow:~/tmp> idl70
> IDL Version 7.0.1 (linux x86_64 m64). (c) 2008, ITT Visual Information
> Solutions
> Installation number: 100-431.
> Licensed for use by: University of Colorado
>
> IDL> .run ./crash_ncdf.pro
> % Compiled module: CREATE_HAND_SAMPLE.
> % Compiled module: CRASH_NCDF.
> IDL> crash_ncdf
> % Loaded DLM: NCDF.
> Successfully completed first iterations
> i = 143017
> ncdf_fid = 5
> % CRASH_NCDF: NCDF_OPEN: Unable to open the file "/projects/NRTSI-G/
> tmp_crashing_ncdf/sample_h
> and.nc".
> (NC_ERROR=-31)
> % Execution halted at: CRASH_NCDF 44 /homes/snowblower/savoie/
> tmp/crash_ncdf.pro
> % $MAIN$
>
> ----------------------------------------
> Sample Test Code:
> ----------------------------------------
> ;+============================================
> ; :Author: Matt Savoie <savoie@nsidc.org>
> ; :Copyright: (C) <2011> University of Colorado.
> ; :Version: $Id:$
> ;
> ; Created 02/03/2011
> ; National Snow & Ice Data Center, University of Colorado, Boulder
> ;-============================================*/
>
> ;+
> ; Generate a very simple netcdf file.
> ;-
> pro create_hand_sample, file
> compile_opt idl2, logical_predicate
>
> sample_var = dindgen( 4, 3 )
> ncid = ncdf_create( file, /CLOBBER )
> dimidx = ncdf_dimdef( ncid, 'x', 4 )
> dimidy = ncdf_dimdef( ncid, 'y', 3 )
> varid = ncdf_vardef( ncid, 'variable', /double )
> ncdf_control, ncid, /ENDEF
> ncdf_varput, ncid, varid, sample_var
> ncdf_close, ncid
>
> end
>
>
> ;+
> ; This is just a sample looping program that shows a problem
> ; Opening and Closing a single netcdf file many times.
> ;-
> pro crash_ncdf
> compile_opt idl2, logical_predicate
>
> catch, theError
> if theError ne 0 then begin
> Catch, /cancel
> print, "i = ", i
> print, "ncdf_fid = ", ncdf_fid
> message, !ERROR_STATE.msg
> endif
>
>
> file = './sample.nc'
> create_hand_sample, file
>
> long_iteration = 1900000L
>
> for i = 0l, long_iteration do begin
> ncdf_fid = ncdf_open( file, write = 0 )
> ncdf_close, ncdf_fid
> endfor
>
> print, 'Successfully completed first iterations'
> window, /free
> wdelete
>
> for i = 0l, long_iteration do begin
> ncdf_fid = ncdf_open( file, write = 0 )
> ncdf_close, ncdf_fid
> endfor
>
> print, 'Successfully completed second iterations'
>
> end
Re: ncdf + x-connection + many files = crazy. [message #74919 is a reply to message #74918] Wed, 09 February 2011 00:32 Go to previous messageGo to next message
Fabzou is currently offline  Fabzou
Messages: 76
Registered: November 2010
Member
Hi,

On 02/08/2011 09:37 PM, Matt wrote:

> The undesirable behavior is a NCDF_OPEN error (NC_ERROR=-31) after
> some *random* number of open/close iterations on a netcdf file that
> was created with IDL (or possibly a segfault core dump on other
> machines.).

I reported a similar problem
( http://www.rhinocerus.net/forum/lang-idl-pvwave/657163-rando m-netcdf-i-o-error.html).
It happens by reading *or* writing of numerous netcdf files, randomly.

It appears to me that the error occurence is related to both the machine
load and the response time of the hard drives. I stopped using IDL
(7.1.1) with data stored on network drives because of this problem.

Maybe IDL8.0 is doing better here?

Fab
Re: ncdf + x-connection + many files = crazy. [message #74924 is a reply to message #74919] Tue, 08 February 2011 15:17 Go to previous messageGo to next message
Paul Van Delst[1] is currently offline  Paul Van Delst[1]
Messages: 1157
Registered: April 2002
Senior Member
Matt wrote:
> Hey all,
>
> I've got a great riddle for you today. (Long story short, can you run
> the test code included below to completion on your machine?)

Dunno if another couple datapoints will help or not, but:

#1:

IDL> crash_ncdf
% Compiled module: CRASH_NCDF.
% Loaded DLM: NCDF.
Successfully completed first iterations
Successfully completed second iterations

IDL> print, !version
{ x86 linux unix linux 8.0.1 Oct 6 2010 32 64}

IDL> help, 'ncdf', /dlm
** NCDF - IDL NetCDF support (loaded)
Version: 4.1.1, Build Date: JUN 18 2010, Source: ITT Visual Information Solutions
Path: /usr/local/idl/idl/idl80/bin/bin.linux.x86/idl_netcdf.so


#2:

IDL> crash_ncdf
% Compiled module: CRASH_NCDF.
% Loaded DLM: NCDF.
Successfully completed first iterations
Successfully completed second iterations

IDL> print, !version
{ ibmr2 AIX unix AIX 6.4 Mar 24 2007 64 64}

IDL> help, 'ncdf', /dlm
** NCDF - IDL NetCDF support (loaded)
Version: 3.5.0, Build Date: MAR 24 2007, Source: ITT Visual Information Solutions
Path: /usrx/local/rsi64/idl64/bin/bin.ibm.rs6000_64/idl_netcdf.a


cheers,

paulv
Re: ncdf + x-connection + many files = crazy. [message #74925 is a reply to message #74924] Tue, 08 February 2011 14:04 Go to previous messageGo to next message
Matt[2] is currently offline  Matt[2]
Messages: 69
Registered: March 2007
Member
On Feb 8, 2:44 pm, Michael Galloy <mgal...@gmail.com> wrote:
> On 2/8/11 2:36 PM, Michael Galloy wrote:
>
> Actually, I'm OK on 7.0 also.
>
> IDL> crash_ncdf
> % Compiled module: CRASH_NCDF.
> % Loaded DLM: NCDF.
> Successfully completed first iterations
> Successfully completed second iterations
> IDL> print, !version
> { i386 darwin unix Mac OS X 7.0 Oct 25 2007      32      64}
> IDL> help, 'ncdf', /dlm
> ** NCDF - IDL NetCDF support (loaded)
>      Version: 3.6.2, Build Date: OCT 25 2007, Source: ITT Visual
> Information Solutions
>      Path: /Applications/itt/idl70/bin/bin.darwin.i386/idl_netcdf.so


Yeah Mike,

I can't recreate this on 8.0x with a mac either. But at least one
other person sees this problem, and it means it's not just our system.

Thanks everyone.

Matt
Re: ncdf + x-connection + many files = crazy. [message #74926 is a reply to message #74925] Tue, 08 February 2011 14:02 Go to previous messageGo to next message
Matt[2] is currently offline  Matt[2]
Messages: 69
Registered: March 2007
Member
Ken,

On Feb 8, 2:29 pm, "Kenneth P. Bowman" <k-bow...@null.edu> wrote:
>
> We get error messages like this some times when reading or
> writing netcdf files to network volumes.  They seem to go
> away when reading/writing on the local machine.
>
> I assume it is a filesystem bug.  It usually happens with
> multiple large files.

The first time I got an NC_ERROR=-31 (previous problems), I was
trying to write a file to a location mounted with fuse-ssh and it was
definitely a problem with writing across the file systems and my
solution was to write that single file locally, and copy it later.

The funny thing in this case, is actually if I write to a different/
remote filesystem, I *don't* get the error. It only occurs when I
write to a local filesystem.

Thanks for the input,

Matt
Re: ncdf + x-connection + many files = crazy. [message #74927 is a reply to message #74926] Tue, 08 February 2011 13:58 Go to previous messageGo to next message
Matt[2] is currently offline  Matt[2]
Messages: 69
Registered: March 2007
Member
On Feb 8, 2:31 pm, FÖLDY Lajos <fo...@rmki.kfki.hu> wrote:
> On Tue, 8 Feb 2011, Matt wrote:

> ps: you are defining a scalar variable, not an array
>
>>   dimidx = ncdf_dimdef( ncid, 'x', 4 )
>>   dimidy = ncdf_dimdef( ncid, 'y', 3 )
>>   varid = ncdf_vardef( ncid,  'variable', /double )

Look at that, you're right. But the file is still created o.k. even as
a vector and since I'm just opening and closing it it shouldn't effect
it.....

Yup changing the offending line to this has no effect on my errors.
varid = ncdf_vardef( ncid, 'variable', [ dimidx, dimidy ], /double )

But thanks for checking this for me.

Matt
Re: ncdf + x-connection + many files = crazy. [message #74928 is a reply to message #74927] Tue, 08 February 2011 13:44 Go to previous messageGo to next message
Michael Galloy is currently offline  Michael Galloy
Messages: 1114
Registered: April 2006
Senior Member
On 2/8/11 2:36 PM, Michael Galloy wrote:
> On 2/8/11 1:37 PM, Matt wrote:
>> Hey all,
>>
>> I've got a great riddle for you today. (Long story short, can you run
>> the test code included below to completion on your machine?) I've
>> already filed a bug report, but I want to see if other people can
>> reproduce these errors.
>>
>> I'm experiencing machine/version-dependent, non-deterministic behavior
>> that seems to present itself orthogonal to the triggering mechanism.
>> In other words, I've got a crazy-making problem.
>>
>> I experienced this in working code, but through patient debugging,
>> I've pared this down my until I've got a relatively small program that
>> demonstrates the problem behavior.
>>
>> The undesirable behavior is a NCDF_OPEN error (NC_ERROR=-31) after
>> some *random* number of open/close iterations on a netcdf file that
>> was created with IDL (or possibly a segfault core dump on other
>> machines.). The second feature is that I don't see this behavior
>> until I make an x-connection of some sort (window, /free, or even a
>> "device, depth=depth" will cause errors)
>>
>> Machine 1: (only has idl 7.0.1 64bit installed)
>> savoie@snow:~/tmp> uname -a
>> Linux snow 2.6.27.54-0.2-default #1 SMP 2010-10-19 18:40:07 +0200
>> x86_64 x86_64 x86_64 GNU/Linux
>>
>> Machine 2 (fails with 32 bit idl7.0.1& idl6.4):
>> savoie@snowblower:~/tmp> uname -a
>> Linux snowblower 2.6.34.7-0.5-default #1 SMP 2010-10-25 08:40:12 +0200
>> i686 i686 i386 GNux
>>
>>
>> All the code does is open and close a netcdf file, many, many times.
>> But after an x-connection (via "window, /free"), the code errors out
>> at a different spot in the iteration.
>>
>>
>> Below is the output I get on the problem machines and below that, the
>> source code for you to try at home. For extra credit, modify the code
>> to open a netcdf file created with a different program than IDL, NCO
>> for example.
>>
>>
>> --OUTPUT--------------------------------------
>> savoie@snow:~/tmp> idl70
>> IDL Version 7.0.1 (linux x86_64 m64). (c) 2008, ITT Visual Information
>> Solutions
>> Installation number: 100-431.
>> Licensed for use by: University of Colorado
>>
>> IDL> .run ./crash_ncdf.pro
>> % Compiled module: CREATE_HAND_SAMPLE.
>> % Compiled module: CRASH_NCDF.
>> IDL> crash_ncdf
>> % Loaded DLM: NCDF.
>> Successfully completed first iterations
>> i = 143017
>> ncdf_fid = 5
>> % CRASH_NCDF: NCDF_OPEN: Unable to open the file "/projects/NRTSI-G/
>> tmp_crashing_ncdf/sample_h
>> and.nc".
>> (NC_ERROR=-31)
>> % Execution halted at: CRASH_NCDF 44 /homes/snowblower/savoie/
>> tmp/crash_ncdf.pro
>> % $MAIN$
>
> I made it through OK with IDL 8.0.1 (and netCDF 4.1 instead of the 3.6.2
> in IDL 7.0):
>
> IDL> crash_ncdf
> % Loaded DLM: NCDF.
> Successfully completed first iterations
> Successfully completed second iterations
> IDL> print, !version
> { x86_64 darwin unix Mac OS X 8.0.1 Oct 5 2010 64 64}
>
> Mike
> --
> www.michaelgalloy.com
> Research Mathematician
> Tech-X Corporation
>

Actually, I'm OK on 7.0 also.

IDL> crash_ncdf
% Compiled module: CRASH_NCDF.
% Loaded DLM: NCDF.
Successfully completed first iterations
Successfully completed second iterations
IDL> print, !version
{ i386 darwin unix Mac OS X 7.0 Oct 25 2007 32 64}
IDL> help, 'ncdf', /dlm
** NCDF - IDL NetCDF support (loaded)
Version: 3.6.2, Build Date: OCT 25 2007, Source: ITT Visual
Information Solutions
Path: /Applications/itt/idl70/bin/bin.darwin.i386/idl_netcdf.so

Mike
--
www.michaelgalloy.com
Research Mathematician
Tech-X Corporation
Re: ncdf + x-connection + many files = crazy. [message #74929 is a reply to message #74928] Tue, 08 February 2011 13:29 Go to previous messageGo to next message
Kenneth P. Bowman is currently offline  Kenneth P. Bowman
Messages: 585
Registered: May 2000
Senior Member
In article
<5011027c-85df-499f-b725-2e7af1965254@m27g2000prj.googlegroups.com>,
Matt <savoie@nsidc.org> wrote:

> Hey all,
>
> I've got a great riddle for you today. (Long story short, can you run
> the test code included below to completion on your machine?) I've
> already filed a bug report, but I want to see if other people can
> reproduce these errors.

We get error messages like this some times when reading or
writing netcdf files to network volumes. They seem to go
away when reading/writing on the local machine.

I assume it is a filesystem bug. It usually happens with
multiple large files.

Ken Bowman
Re: ncdf + x-connection + many files = crazy. [message #74930 is a reply to message #74929] Tue, 08 February 2011 13:36 Go to previous messageGo to next message
Michael Galloy is currently offline  Michael Galloy
Messages: 1114
Registered: April 2006
Senior Member
On 2/8/11 1:37 PM, Matt wrote:
> Hey all,
>
> I've got a great riddle for you today. (Long story short, can you run
> the test code included below to completion on your machine?) I've
> already filed a bug report, but I want to see if other people can
> reproduce these errors.
>
> I'm experiencing machine/version-dependent, non-deterministic behavior
> that seems to present itself orthogonal to the triggering mechanism.
> In other words, I've got a crazy-making problem.
>
> I experienced this in working code, but through patient debugging,
> I've pared this down my until I've got a relatively small program that
> demonstrates the problem behavior.
>
> The undesirable behavior is a NCDF_OPEN error (NC_ERROR=-31) after
> some *random* number of open/close iterations on a netcdf file that
> was created with IDL (or possibly a segfault core dump on other
> machines.). The second feature is that I don't see this behavior
> until I make an x-connection of some sort (window, /free, or even a
> "device, depth=depth" will cause errors)
>
> Machine 1: (only has idl 7.0.1 64bit installed)
> savoie@snow:~/tmp> uname -a
> Linux snow 2.6.27.54-0.2-default #1 SMP 2010-10-19 18:40:07 +0200
> x86_64 x86_64 x86_64 GNU/Linux
>
> Machine 2 (fails with 32 bit idl7.0.1& idl6.4):
> savoie@snowblower:~/tmp> uname -a
> Linux snowblower 2.6.34.7-0.5-default #1 SMP 2010-10-25 08:40:12 +0200
> i686 i686 i386 GNux
>
>
> All the code does is open and close a netcdf file, many, many times.
> But after an x-connection (via "window, /free"), the code errors out
> at a different spot in the iteration.
>
>
> Below is the output I get on the problem machines and below that, the
> source code for you to try at home. For extra credit, modify the code
> to open a netcdf file created with a different program than IDL, NCO
> for example.
>
>
> --OUTPUT--------------------------------------
> savoie@snow:~/tmp> idl70
> IDL Version 7.0.1 (linux x86_64 m64). (c) 2008, ITT Visual Information
> Solutions
> Installation number: 100-431.
> Licensed for use by: University of Colorado
>
> IDL> .run ./crash_ncdf.pro
> % Compiled module: CREATE_HAND_SAMPLE.
> % Compiled module: CRASH_NCDF.
> IDL> crash_ncdf
> % Loaded DLM: NCDF.
> Successfully completed first iterations
> i = 143017
> ncdf_fid = 5
> % CRASH_NCDF: NCDF_OPEN: Unable to open the file "/projects/NRTSI-G/
> tmp_crashing_ncdf/sample_h
> and.nc".
> (NC_ERROR=-31)
> % Execution halted at: CRASH_NCDF 44 /homes/snowblower/savoie/
> tmp/crash_ncdf.pro
> % $MAIN$

I made it through OK with IDL 8.0.1 (and netCDF 4.1 instead of the 3.6.2
in IDL 7.0):

IDL> crash_ncdf
% Loaded DLM: NCDF.
Successfully completed first iterations
Successfully completed second iterations
IDL> print, !version
{ x86_64 darwin unix Mac OS X 8.0.1 Oct 5 2010 64 64}

Mike
--
www.michaelgalloy.com
Research Mathematician
Tech-X Corporation
Re: ncdf + x-connection + many files = crazy. [message #74931 is a reply to message #74929] Tue, 08 February 2011 13:31 Go to previous messageGo to next message
Foldy Lajos is currently offline  Foldy Lajos
Messages: 268
Registered: October 2001
Senior Member
On Tue, 8 Feb 2011, Matt wrote:

> Hey all,
>
> I've got a great riddle for you today. (Long story short, can you run
> the test code included below to completion on your machine?) I've
> already filed a bug report, but I want to see if other people can
> reproduce these errors.
>
> I'm experiencing machine/version-dependent, non-deterministic behavior
> that seems to present itself orthogonal to the triggering mechanism.
> In other words, I've got a crazy-making problem.
>
> I experienced this in working code, but through patient debugging,
> I've pared this down my until I've got a relatively small program that
> demonstrates the problem behavior.
>
> The undesirable behavior is a NCDF_OPEN error (NC_ERROR=-31) after
> some *random* number of open/close iterations on a netcdf file that
> was created with IDL (or possibly a segfault core dump on other
> machines.). The second feature is that I don't see this behavior
> until I make an x-connection of some sort (window, /free, or even a
> "device, depth=depth" will cause errors)
>
> Machine 1: (only has idl 7.0.1 64bit installed)
> savoie@snow:~/tmp> uname -a
> Linux snow 2.6.27.54-0.2-default #1 SMP 2010-10-19 18:40:07 +0200
> x86_64 x86_64 x86_64 GNU/Linux
>
> Machine 2 (fails with 32 bit idl7.0.1 & idl6.4):
> savoie@snowblower:~/tmp> uname -a
> Linux snowblower 2.6.34.7-0.5-default #1 SMP 2010-10-25 08:40:12 +0200
> i686 i686 i386 GNux
>
>
> All the code does is open and close a netcdf file, many, many times.
> But after an x-connection (via "window, /free"), the code errors out
> at a different spot in the iteration.
>
>
> Below is the output I get on the problem machines and below that, the
> source code for you to try at home. For extra credit, modify the code
> to open a netcdf file created with a different program than IDL, NCO
> for example.
>
>
> --OUTPUT--------------------------------------
> savoie@snow:~/tmp> idl70
> IDL Version 7.0.1 (linux x86_64 m64). (c) 2008, ITT Visual Information
> Solutions
> Installation number: 100-431.
> Licensed for use by: University of Colorado
>
> IDL> .run ./crash_ncdf.pro
> % Compiled module: CREATE_HAND_SAMPLE.
> % Compiled module: CRASH_NCDF.
> IDL> crash_ncdf
> % Loaded DLM: NCDF.
> Successfully completed first iterations
> i = 143017
> ncdf_fid = 5
> % CRASH_NCDF: NCDF_OPEN: Unable to open the file "/projects/NRTSI-G/
> tmp_crashing_ncdf/sample_h
> and.nc".
> (NC_ERROR=-31)
> % Execution halted at: CRASH_NCDF 44 /homes/snowblower/savoie/
> tmp/crash_ncdf.pro
> % $MAIN$
>
> ----------------------------------------
> Sample Test Code:
> ----------------------------------------
> ;+============================================
> ; :Author: Matt Savoie <savoie@nsidc.org>
> ; :Copyright: (C) <2011> University of Colorado.
> ; :Version: $Id:$
> ;
> ; Created 02/03/2011
> ; National Snow & Ice Data Center, University of Colorado, Boulder
> ;-============================================*/
>
> ;+
> ; Generate a very simple netcdf file.
> ;-
> pro create_hand_sample, file
> compile_opt idl2, logical_predicate
>
> sample_var = dindgen( 4, 3 )
> ncid = ncdf_create( file, /CLOBBER )
> dimidx = ncdf_dimdef( ncid, 'x', 4 )
> dimidy = ncdf_dimdef( ncid, 'y', 3 )
> varid = ncdf_vardef( ncid, 'variable', /double )
> ncdf_control, ncid, /ENDEF
> ncdf_varput, ncid, varid, sample_var
> ncdf_close, ncid
>
> end
>
>
> ;+
> ; This is just a sample looping program that shows a problem
> ; Opening and Closing a single netcdf file many times.
> ;-
> pro crash_ncdf
> compile_opt idl2, logical_predicate
>
> catch, theError
> if theError ne 0 then begin
> Catch, /cancel
> print, "i = ", i
> print, "ncdf_fid = ", ncdf_fid
> message, !ERROR_STATE.msg
> endif
>
>
> file = './sample.nc'
> create_hand_sample, file
>
> long_iteration = 1900000L
>
> for i = 0l, long_iteration do begin
> ncdf_fid = ncdf_open( file, write = 0 )
> ncdf_close, ncdf_fid
> endfor
>
> print, 'Successfully completed first iterations'
> window, /free
> wdelete
>
> for i = 0l, long_iteration do begin
> ncdf_fid = ncdf_open( file, write = 0 )
> ncdf_close, ncdf_fid
> endfor
>
> print, 'Successfully completed second iterations'
>
> end
>

linux x86_64:

IDL 7.0 (64 bit) fails
IDL 6.4 (64 bit) succeeds

regards,
Lajos

ps: you are defining a scalar variable, not an array

> sample_var = dindgen( 4, 3 )
> ncid = ncdf_create( file, /CLOBBER )
> dimidx = ncdf_dimdef( ncid, 'x', 4 )
> dimidy = ncdf_dimdef( ncid, 'y', 3 )
> varid = ncdf_vardef( ncid, 'variable', /double )
> ncdf_control, ncid, /ENDEF
> ncdf_varput, ncid, varid, sample_var
> ncdf_close, ncid
Re: ncdf + x-connection + many files = crazy. [message #74996 is a reply to message #74916] Wed, 09 February 2011 14:27 Go to previous messageGo to next message
Matt[2] is currently offline  Matt[2]
Messages: 69
Registered: March 2007
Member
On Feb 9, 2:21 am, Maarten <maarten.sn...@knmi.nl> wrote:
> On Feb 9, 9:56 am, Reimar Bauer <R.Ba...@fz-juelich.de> wrote:
>
>>> crash_ncdf
>>> % Loaded DLM: NCDF.
>>> i =       819132
>>> ncdf_fid =        65536
>
>> strange it is for me always  "ncdf_fid =        65536"
>
>> I can reproduce that number
>
> The nc_open documentation just says it is an int. To me that means 32-
> bit, but could they have compiled the library with 16-bit ints?
>
> Maarten



Looks like I should have read all three results before posting. I
agree that Reimar's number is pretty fishy. My ncdf_fid was always
pretty low.

Thanks again everyone.

Matt
Re: ncdf + x-connection + many files = crazy. [message #74997 is a reply to message #74918] Wed, 09 February 2011 14:26 Go to previous messageGo to next message
Matt[2] is currently offline  Matt[2]
Messages: 69
Registered: March 2007
Member
On Feb 9, 1:39 am, Reimar Bauer <R.Ba...@fz-juelich.de> wrote:
>
> I got it too, and I remember a similiar test case for a old fixed bug.
> I'll search for that. May be it is a regression.
>
> crash_ncdf
> % Loaded DLM: NCDF.
> i =       819132
> ncdf_fid =        65536
> % CRASH_NCDF: NCDF_OPEN: Unable to open the file "./sample.nc".
> (NC_ERROR=-31)
> % Execution halted at: CRASH_NCDF         41
>

Remar,

It look like you ran out of file handles? Or at least it's quite a
coincidence if not. 2^16 = 65536. Thanks for the data point.

Matt
Re: ncdf + x-connection + many files = crazy. [message #75159 is a reply to message #74917] Fri, 18 February 2011 15:30 Go to previous messageGo to next message
Matt[2] is currently offline  Matt[2]
Messages: 69
Registered: March 2007
Member
Reimar,

So I might have spoken too soon. I just installed 8.0.1 and we are
reproducing the error, but now our ncdf_fid is also 65536. Can you
give me the version, and OS information for your machine?

Thanks,
Matt

On Feb 9, 1:56 am, Reimar Bauer <R.Ba...@fz-juelich.de> wrote:
>> Hi Matt
>
>> I got it too, and I remember a similiar test case for a old fixed bug.
>> I'll search for that. May be it is a regression.
>
>> crash_ncdf
>> % Loaded DLM:NCDF.
>> i =       819132
>> ncdf_fid =        65536
>
> strange it is for me always  "ncdf_fid =        65536"
>
> I can reproduce that number
>
> Reimar
Re: ncdf + x-connection + many files = crazy. [message #75237 is a reply to message #75159] Mon, 21 February 2011 02:38 Go to previous message
R.Bauer is currently offline  R.Bauer
Messages: 1424
Registered: November 1998
Senior Member
Am 19.02.2011 00:30, schrieb Matt:
> Reimar,
>
> So I might have spoken too soon. I just installed 8.0.1 and we are
> reproducing the error, but now our ncdf_fid is also 65536. Can you
> give me the version, and OS information for your machine?

IDL> help, !version,/str
** Structure !VERSION, 8 tags, length=104, data length=100:
ARCH STRING 'x86_64'
OS STRING 'linux'
OS_FAMILY STRING 'unix'
OS_NAME STRING 'linux'
RELEASE STRING '8.0'
BUILD_DATE STRING 'Jun 18 2010'
MEMORY_BITS INT 64
FILE_OFFSET_BITS
INT 64

OS: linux ubuntu 10.04 LTS



Reimar

>
> Thanks,
> Matt
>
> On Feb 9, 1:56 am, Reimar Bauer <R.Ba...@fz-juelich.de> wrote:
>>> Hi Matt
>>
>>> I got it too, and I remember a similiar test case for a old fixed bug.
>>> I'll search for that. May be it is a regression.
>>
>>> crash_ncdf
>>> % Loaded DLM:NCDF.
>>> i = 819132
>>> ncdf_fid = 65536
>>
>> strange it is for me always "ncdf_fid = 65536"
>>
>> I can reproduce that number
>>
>> Reimar
>
  Switch to threaded view of this topic Create a new topic Submit Reply
Previous Topic: Re: More cgplot, ps_start, ps_end problems
Next Topic: save axis

-=] Back to Top [=-
[ Syndicate this forum (XML) ] [ RSS ] [ PDF ]

Current Time: Wed Oct 08 15:16:23 PDT 2025

Total time taken to generate the page: 0.00484 seconds