| Reading sequence of HDF files fails after some number of successes [message #82103] |
Tue, 20 November 2012 13:49  |
BLesht
Messages: 89
Registered: March 2007
|
Member |
|
|
I'm using IDL8.1 running on MacOS10.6.8 to process a number of HDF files (NASA MODIS L2). My code (too involved to reproduce in entirety here) includes a loop in which I open each file, check it to see if it is useful for my analysis, and then, if it passes, does some processing. The code seems to run fine through a number of files and then stops with an "HDF_SD_START: Invalid HDF file or filename (filename is here)" error. I have checked the file using ncdump_hdf and it is fine. While debugging I inserted a test:
PRINT, 'About to test HDF file', inFile & $
testHDF = HDF_OPEN(inFile, /READ) & $ ; Open the HDF file
PRINT, 'testHDF = ', testHDF & $
IF (testHDF NE -1L) THEN PRINT, inFile, ' passes HDF_OPEN test ' & $
HDF_CLOSE, testHDF & $
That results in:
About to test HDF file <filename is here>
testHDF = 536872618
<filename is here> passes HDF_OPEN test
but then goes on to issue the:
% HDF_SD_START: Invalid HDF file or filename (<filename is here>).
% Execution halted at: $MAIN$
Continuing the debugging, I removed the apparently offending file from the input list and the code stopped at exactly the same point but now the next file in the original list was the one that apparently caused the failure. This leads me to think the problem has something to do with the number of HDF files I've been trying to open (and possibly not closing properly?). The portion of the code that fails is:
fid = HDF_SD_START(inFile, /READ) & $
sdsNoLat = HDF_SD_NAMETOINDEX(fid, 'latitude') & $
varid = HDF_SD_SELECT(fid,sdsNoLat) & $
HDF_SD_GETDATA, varid, latitude & $
HDF_SD_ENDACCESS, varid & $
sdsNoLon = HDF_SD_NAMETOINDEX(fid, 'longitude') & $
varid = HDF_SD_SELECT(fid,sdsNoLon) & $
HDF_SD_GETDATA, varid, longitude & $
HDF_SD_ENDACCESS, varid & $
HDF_SD_END, fid & $ ; End the HDF_SD
This seems like it should be obvious, but I'm not seeing the problem. Any help would be sincerely appreciated.
Barry
|
|
|
|
| Re: Reading sequence of HDF files fails after some number of successes [message #82176 is a reply to message #82103] |
Wed, 21 November 2012 08:32  |
BLesht
Messages: 89
Registered: March 2007
|
Member |
|
|
Thanks, Klemen - doesn't help, I'm afraid. The code I'm working on is a shell script that creates a SeaDAS batch file that includes both IDL and SeaDAS procedures. Now I'm wondering if the problem lies somewhere else in the code. I've tried different combinations of the the HDF file access calls without success. The code processes a number of files with no trouble but then stops with the "Invalid HDF file or filename" error, and that error occurs at the same point no matter what file is next in the input list. In the past I've had IDL code stop because I forgot to free a LUN and there was a limit to the number of fileids that could be assigned. That is what started me thinking about how I was closing the HDF files.
Barry
On Wednesday, November 21, 2012 8:22:02 AM UTC-6, Klemen wrote:
> I am not sure it it helps, but I think you can delete the first line with:
>
> HDF_SD_ENDACCESS, varid & $
>
>
>
> As long as you work on the same file it works like this as code below(I use it open MODIS albedo Level 3 product). This is how I processed over 1000 files.
>
>
>
> Cheers, Klemen
>
>
>
>
>
>
>
> FOR y=0,n-1 DO BEGIN
>
> ; Open EOS-HDF
>
> i_fid = HDF_ISHDF(s_list[y])
>
> IF i_fid EQ 0 THEN BEGIN
>
> PRINT, ' !!! ' + s_list[y]
>
> PRINT, 'The input file does not exist or is not HDF format!'
>
> CONTINUE
>
> ENDIF
>
> sd_id = HDF_SD_START(s_list[y], /READ)
>
> FOR i=0,6 DO BEGIN
>
> out[i,*,*] = albedo_process_read_HDF(sd_id, 'Albedo_WSA_Band' + STRTRIM(i+1, 1), extent[*,t], sds_id=sds_id)
>
> ENDFOR
>
> HDF_SD_ENDACCESS, sds_id
>
> HDF_SD_END, sd_id
>
>
>
>
>
> Where the called function "albedo_process_read_HDF" is:
>
>
>
> FUNCTION albedo_process_read_HDF, sd_id, sd_name, extent, sds_id=sds_id
>
> sd_index = HDF_SD_NAMETOINDEX(sd_id, sd_name)
>
> sds_id = HDF_SD_SELECT(sd_id, sd_index)
>
> HDF_SD_GETDATA, sds_id, tmp, START=extent[0:1], COUNT=extent[2:3]
>
> RETURN, tmp
>
> END
|
|
|
|
| Re: Reading sequence of HDF files fails after some number of successes [message #82180 is a reply to message #82103] |
Wed, 21 November 2012 06:22 |
Klemen
Messages: 80
Registered: July 2009
|
Member |
|
|
I am not sure it it helps, but I think you can delete the first line with:
HDF_SD_ENDACCESS, varid & $
As long as you work on the same file it works like this as code below(I use it open MODIS albedo Level 3 product). This is how I processed over 1000 files.
Cheers, Klemen
FOR y=0,n-1 DO BEGIN
; Open EOS-HDF
i_fid = HDF_ISHDF(s_list[y])
IF i_fid EQ 0 THEN BEGIN
PRINT, ' !!! ' + s_list[y]
PRINT, 'The input file does not exist or is not HDF format!'
CONTINUE
ENDIF
sd_id = HDF_SD_START(s_list[y], /READ)
FOR i=0,6 DO BEGIN
out[i,*,*] = albedo_process_read_HDF(sd_id, 'Albedo_WSA_Band' + STRTRIM(i+1, 1), extent[*,t], sds_id=sds_id)
ENDFOR
HDF_SD_ENDACCESS, sds_id
HDF_SD_END, sd_id
Where the called function "albedo_process_read_HDF" is:
FUNCTION albedo_process_read_HDF, sd_id, sd_name, extent, sds_id=sds_id
sd_index = HDF_SD_NAMETOINDEX(sd_id, sd_name)
sds_id = HDF_SD_SELECT(sd_id, sd_index)
HDF_SD_GETDATA, sds_id, tmp, START=extent[0:1], COUNT=extent[2:3]
RETURN, tmp
END
|
|
|
|
| Re: Reading sequence of HDF files fails after some number of successes [message #82184 is a reply to message #82103] |
Wed, 21 November 2012 02:42 |
lecacheux.alain
Messages: 325
Registered: January 2008
|
Senior Member |
|
|
Le mercredi 21 novembre 2012 04:42:20 UTC+1, Barry Lesht a écrit :
> Yup - there were two sections of code. The first, which I inserted as a test just to be sure the file wasn't really bad, had both an HDF_OPEN and an HDF_CLOSE. The second used just the HDF_SD_START to open the file and closed it with the HDF_SD_ENDACCESS, HDF_SD_END.
>
>
>
> I'm going back through your write-up in "Programming Techniques" and have scanned the group for ideas, but no joy yet. This might best be left for tomorrow.
>
>
>
> Thanks.
>
>
>
> On Tuesday, November 20, 2012 8:55:34 PM UTC-6, David Fanning wrote:
>
>
>
>>
>
>> Well, I saw an HDF_OPEN in the example you sent us. It it
>
>>
>
>> what prompted me to suggest HDF_CLOSE.
>
>>
>
>>
I am glad to congratulate PDP-11 users (were you using RSX11M or RT11 ?)! Unfortunately, I cannot help with HDF...
alx.
|
|
|
|
|
|
| Re: Reading sequence of HDF files fails after some number of successes [message #82190 is a reply to message #82103] |
Tue, 20 November 2012 18:55 |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
Barry Lesht writes:
> Well David, give me a break - I only go back to the PDP-11, though I do remember 'programming' by toggling switches on the front panel.
Oh, yes, PDP-11. :-)
> The IDL code is part of a larger script that gets fed to SeaDAS in batch mode. When passed to SeaDAS, IDL statements in loops need the continuation characters. Indeed, it is a bother.
OK, then.
> Right - there was no HDF_CLOSE, but neither was there an HDF_OPEN.
Well, I saw an HDF_OPEN in the example you sent us. It it
what prompted me to suggest HDF_CLOSE.
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Sepore ma de ni thue. ("Perhaps thou speakest truth.")
|
|
|
|
| Re: Reading sequence of HDF files fails after some number of successes [message #82191 is a reply to message #82103] |
Tue, 20 November 2012 18:47 |
BLesht
Messages: 89
Registered: March 2007
|
Member |
|
|
On Tuesday, November 20, 2012 6:13:11 PM UTC-6, David Fanning wrote:
> David Fanning writes:
>
>
>
>> Do you really program with all those line continuation characters!?
>
>> Sheesh! That must be a bother! :-)
>
>
>
> Just to clarify, I'm not trying to be my usual snarky self.
>
> I just really admire a guy who can stay true to the ol'
>
> DEC PDP 8 programming standards in this day and age! ;-)
>
>
>
> Cheers,
>
>
>
> David
>
>
>
>
Well David, give me a break - I only go back to the PDP-11, though I do remember 'programming' by toggling switches on the front panel.
The IDL code is part of a larger script that gets fed to SeaDAS in batch mode. When passed to SeaDAS, IDL statements in loops need the continuation characters. Indeed, it is a bother.
Right - there was no HDF_CLOSE, but neither was there an HDF_OPEN. I've been confused about the relationship between HDF_OPEN - HDF_CLOSE pairs and HDF_SD_START - HDF_SD_ENDACCESS - HDF_SD_END triplet. If I add an HDF_CLOSE, referencing the index number that goes with the HDF_SD_START, I get the same error as before. If I bracket the code with HDF_OPEN - HDF_CLOSE, it fails with "HDF_CLOSE: Invalid file id, FID(xxxxxxx).
I guess I can play with different combinations, but I'd like to understand what is going on and what I read baffles me. It was a lot simpler to toggle switches - there were only eight after all.
B.
>
>
>
> --
>
> David Fanning, Ph.D.
>
> Fanning Software Consulting, Inc.
>
> Coyote's Guide to IDL Programming: http://www.dfanning.com/
>
> Sepore ma de ni thue. ("Perhaps thou speakest truth.")
|
|
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
