Messages from Usenet group comp.lang.idl-pvwave, compiled by Paulo Penteado
I'm running IDL5.1 on an SGI (IRIX6.2). I'm using FindFile to create an array of data filenames that I can pass to some processing code. There are (as it turns out) 515 files in the directory I'm searching. The filenames are all of the form xxxxxxxxx.xx.yymmdd.hhmmss.cdf. I get the following from FindFile- IDL> files=FindFile('*.*', COUNT=nf) IDL> PRINT, nf 0 IDL> files=FindFile('*.cdf', COUNT=nf) IDL> PRINT, nf 0 IDL> files=FindFile('*98*.cdf', COUNT=nf) IDL> PRINT, nf 0 IDL> files=FindFile('*980*.cdf', COUNT=nf) IDL> PRINT, nf 425 IDL> files=FindFile('*981*.cdf', COUNT=nf) IDL> PRINT, nf 90 I don't see why cases 2 and (especially) 3 didn't result in nf=515. Can anyone explain this? Thanks. Barry Lesht bmlesht@anl.gov
In article <73holl$fts@milo.mcs.anl.gov>, lesht@ANLER.ER.ANL.GOV (Barry Lesht) writes: > I'm running IDL5.1 on an SGI (IRIX6.2). I'm using FindFile to create > an array of data filenames that I can pass to some processing code. > There are (as it turns out) 515 files in the directory I'm searching. > The filenames are all of the form xxxxxxxxx.xx.yymmdd.hhmmss.cdf. I > get the following from FindFile- > > IDL> files=FindFile('*.*', COUNT=nf) > IDL> PRINT, nf > 0 > IDL> files=FindFile('*.cdf', COUNT=nf) > IDL> PRINT, nf > 0 > IDL> files=FindFile('*98*.cdf', COUNT=nf) > IDL> PRINT, nf > 0 > IDL> files=FindFile('*980*.cdf', COUNT=nf) > IDL> PRINT, nf > 425 > IDL> files=FindFile('*981*.cdf', COUNT=nf) > IDL> PRINT, nf > 90 > > I don't see why cases 2 and (especially) 3 didn't result in nf=515. > Can anyone explain this? Thanks. This problem should really be addressed by RSI. What is happening, I believe, is that FINDFILE is spawning a shell to get the file list via something like "ls". However, the shell has limits on the byte length of command line arguments - for long lists, the command will fail. RSI should really be using another UNIX tool meant for this job - that is the "find" command. See the UNIX man pages exec(2), find(1), xargs(1) for more info. I have a program FINDFILES on my web page that should work for you, and also "fixes" searching for files recursively. See http://cspar.uah.edu/~mallozzir/idl/idl.html -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~ Robert S. Mallozzi 256-544-0887 Mail Code ES 84 Work: http://www.batse.msfc.nasa.gov/ Marshall Space Flight Center Play: http://cspar.uah.edu/~mallozzir/ Huntsville, AL 35812 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~
Barry Lesht wrote: :> I'm running IDL5.1 on an SGI (IRIX6.2). I'm using FindFile to create :> an array of data filenames that I can pass to some processing code. :> There are (as it turns out) 515 files in the directory I'm searching. :> The filenames are all of the form xxxxxxxxx.xx.yymmdd.hhmmss.cdf. I :> get the following from FindFile- :> :> IDL> files=FindFile('*.*', COUNT=nf) :> IDL> PRINT, nf :> 0 :> IDL> files=FindFile('*.cdf', COUNT=nf) :> IDL> PRINT, nf :> 0 :> IDL> files=FindFile('*98*.cdf', COUNT=nf) :> IDL> PRINT, nf :> 0 :> IDL> files=FindFile('*980*.cdf', COUNT=nf) ;> IDL> PRINT, nf :> 425 :> IDL> files=FindFile('*981*.cdf', COUNT=nf) :> IDL> PRINT, nf :> 90 :> :> I don't see why cases 2 and (especially) 3 didn't result in nf=515. :> Can anyone explain this? Thanks. :> :> Barry Lesht :> bmlesht@anl.gov We had a problem on our system when the filesystem holding /tmp was short of room. I discovered that findfile was creating a temporary file in /tmp containing the matched filenames. When this list was large and file filesystem containg /tmp didn't have room to hold it a truncated list of filenames was returned. There was no error indication. The number of matched files also varied with how much space was available in /tmp. Have a look and see how much space is left in /tmp. Or was it /var/tmp? Now I'm not sure which it was... Check both, it was one of the two. -- ----------------------------------------------------------- Nigel Wade, System Administrator, Space Plasma Physics Group, University of Leicester, Leicester, LE1 7RH, UK E-mail : nmw@ion.le.ac.uk Phone : +44 (0)116 2523568, Fax : +44 (0)116 2523555
Barry Lesht wrote: > I'm running IDL5.1 on an SGI (IRIX6.2). I'm using FindFile to create > an array of data filenames that I can pass to some processing code. > There are (as it turns out) 515 files in the directory I'm searching. > The filenames are all of the form xxxxxxxxx.xx.yymmdd.hhmmss.cdf. I > get the following from FindFile- > > IDL> files=FindFile('*.*', COUNT=nf) > IDL> PRINT, nf > 0 > [...] > > I don't see why cases 2 and (especially) 3 didn't result in nf=515. > Can anyone explain this? Thanks. > > Barry Lesht > bmlesht@anl.gov Yup, this seems to be an SGI related problem. I have never heard of other machines with this behaviour. I had once contacted RSI about a similar problem, and they replied it may have to do with the shell expansion (in my case I was getting into trouble using '~' identifiers even in small directories). It would probbaly help if you could forward this bug report to RSI -- maybe they will fix it at some point... Please find below a third variant of a routine that has been designed to overcome the problem you posted, Regards, Martin. -- ------------------------------------------------------------ ------- Dr. Martin Schultz Department for Engineering&Applied Sciences, Harvard University 109 Pierce Hall, 29 Oxford St., Cambridge, MA-02138, USA phone: (617)-496-8318 fax : (617)-495-4551 e-mail: mgs@io.harvard.edu Internet-homepage: http://www-as.harvard.edu/people/staff/mgs/ ------------------------------------------------------------ ------- ; $Id: mfindfile.pro,v 1.1 1998/10/09 19:53:32 mgs Exp $ ;----------------------------------------------------------- -- ;+ ; NAME: ; MFINDFILE ; ; PURPOSE: ; find all the files that match a given specification. ; On our system, the IDL findfile function does not ; work correctly!! ; ; CATEGORY: ; System routines ; ; CALLING SEQUENCE: ; listing = MFINDFILE(filemask) ; ; INPUTS: ; FILEMASK -> a path and filename specification to look ; for. ; ; KEYWORD PARAMETERS: ; none ; ; OUTPUTS: ; A string list containing all the files that match the ; specification. ; ; SUBROUTINES: ; ; REQUIREMENTS: ; ; NOTES: ; Spawns a unix ls -1 command ! ; ; EXAMPLE: ; list = mfindfile('~mgs/terra/chem1d/code/*.f') ; ; ; returns all fortran files in Martin's chem1d directory. ; ; MODIFICATION HISTORY: ; mgs, 14 Sep 1998: VERSION 1.00 ; ;- ; Copyright (C) 1998, Martin Schultz, Harvard University ; This software is provided as is without any warranty ; whatsoever. It may be freely used, copied or distributed ; for non-commercial purposes. This copyright notice must be ; kept with any copy of this software. If this software shall ; be used commercially or sold as part of a larger package, ; please contact the author to arrange payment. ; Bugs and comments should be directed to mgs@io.harvard.edu ; with subject "IDL routine mfindfile" ;----------------------------------------------------------- -- function mfindfile,mask if (!version.os_family eq 'unix') then begin ; make my own findfile path = extract_path(mask,filename=fname) path = expand_path(path) newpath = path+fname ; print,'fname:',fname,' path:',path,' newpath:',newpath command = 'ls -1' cstr = command+' '+newpath spawn,cstr,listing return,listing endif else begin ; other OS - use IDL's original return,findfile(mask) endelse end