| Re: Download files from the web [message #86942 is a reply to message #86940] |
Mon, 16 December 2013 05:41   |
Helder Marchetto
Messages: 520
Registered: November 2011
|
Senior Member |
|
|
On Monday, December 16, 2013 2:32:36 PM UTC+1, Mats Löfdahl wrote:
> Den måndagen den 16:e december 2013 kl. 13:47:14 UTC+1 skrev Helder:
>
>> On Monday, December 16, 2013 1:37:11 PM UTC+1, Mats Löfdahl wrote:
>
>>
>
>>> I need to make an idl program download a couple of text files from the web. I found the webget() function from astrolib, now also distributed with idl, see http://www.exelisvis.com/docs/webget.html
>
>>
>
>>>
>
>>
>
>>>
>
>>
>
>>>
>
>>
>
>>> My problem with that is that I don't know if the download succeeded. If I give a non-existing URL, I get the 404 error page downloaded and everything looks fine. If I set the COPYFILE keyword to some local file name, the file gets stored there and instead of the file contents, the return value is a scalar long. The web page does not say how this number should be interpreted but it seems to be unity whether or not the URL was really valid.
>
>>
>
>>>
>
>>
>
>>>
>
>>
>
>>>
>
>>
>
>>> I guess I could search the downloaded file for some variation of "404 - Page not found" but I don't know how much that string varies from web server to web server. And it seems a hassle anyway.
>
>>
>
>>>
>
>>
>
>>>
>
>>
>
>>>
>
>>
>
>>> I had a look at http://www.exelisvis.com/docs/socket.html, too. Promisingly it has an ERROR keyword but I don't understand how to tell it which file I want. Seems possible only to specify the host but not which file on the host. Near the bottom of the socket.html page there are two links to pages that promise to tell me how to read web pages and access ftp servers through socket, but when I click on them I get "Article does not exist or Permission Denied" errors.
>
>>
>
>>>
>
>>
>
>>>
>
>>
>
>>>
>
>>
>
>>> So, how does one make idl download files from the web - and tell you if it worked?
>
>>
>
>>
>
>>
>
>> Hi Mats,
>
>>
>
>> if it helps, i used the IDLnetUrl object and then use the getProperty method to get the Response_code value. Not sure if it helps, but I have been downloading files successfully with https. I also use the callback_function to make a progress bar.
>
>>
>
>> Not sure if it helps, but it might be a place to start...
>
>
>
> Looks promising.
>
>
>
> So I would create an instance of the class, initialize it, download with the get method, get the RESPONSE_CODE (anything but zero is a fail?) through the getproperty method, and then destroy the class instance?
Hi Mats,
here is what I do. Not sure if it solves the problem of a missing file.
FUNCTION UrlBigFileGetCallbackStatus, status, progress, oProgressbar
IF progress[0] THEN oProgressbar->Update, 100.0*progress[2]/progress[1]
print, 'Check for update: '+status
return, 1
END
PRO TEST_DownLoad
CurrDir = 'E:\MySoftware\'
oProgressbar = Obj_New('cgprogressbar', TITLE='Downloading FileName.sav...')
oUrl = OBJ_NEW('IDLnetUrl', URL_SCHEME = 'http', URL_HOSTNAME = 'www.#####/FileName.sav', URL_USERNAME = 'user', URL_PASSWORD = 'pwd', CALLBACK_FUNCTION ='UrlBigFileGetCallbackStatus', CALLBACK_DATA=oProgressbar)
VersionFileName = CurrDir+'FileName.sav'
oProgressbar->Start
retrievedFilePath = oUrl->Get(FILENAME=VersionFileName)
oUrl->GetProperty, RESPONSE_CODE=RespCode
oUrl->CloseConnections
OBJ_DESTROY, oUrl
oProgressbar->Destroy
END
The output I get is the following:
Check for update: Sending Http Get Request:
Check for update: http://www.#####/FileName.sav
Check for update: Http: get: received (483), total expected (483)
Check for update: Http: get: received (483), total expected (483)
Check for update: Http: get: received (1208), total expected (15501460)
Check for update: Http: get: received (834656), total expected (15501460)
Check for update: Http: get: received (2173400), total expected (15501460)
Check for update: Http: get: received (3384368), total expected (15501460)
Check for update: Http: get: received (4622924), total expected (15501460)
Check for update: Http: get: received (5867288), total expected (15501460)
Check for update: Http: get: received (7105844), total expected (15501460)
Check for update: Http: get: received (8354564), total expected (15501460)
Check for update: Http: get: received (9594572), total expected (15501460)
Check for update: Http: get: received (10831676), total expected (15501460)
Check for update: Http: get: received (12023768), total expected (15501460)
Check for update: Http: get: received (13295720), total expected (15501460)
Check for update: Http: get: received (14551700), total expected (15501460)
Check for update: Http: get: received (15501460), total expected (15501460)
Check for update: Http Get response written to:
Check for update: E:\MySoftware\FileName.sav
Check for update: Http Get completed.
When I download a big file the progress bar does not update after a few seconds. Any idea why? (Actually any widget stops updating...)
Cheers,
Helder
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
