comp.lang.idl-pvwave archive - RDF feed
type: channel
description: Messages from Usenet group comp.lang.idl-pvwave, compiled by Paulo Penteado
items: Download files from the web
items: Re: Download files from the web

Download files from the web
type: item
description: I need to make an idl program download a couple of text files from the web. I found the webget() function from astrolib, now also distributed with idl, see http://www.exelisvis.com/docs/webget.html My problem with that is that I don't know if the download succeeded. If I give a non-existing URL, I get the 404 error page downloaded and everything looks fine. If I set the COPYFILE keyword to some local file name, the file gets stored there and instead of the file contents, the return value is a scalar long. The web page does not say how this number should be interpreted but it seems to be unity whether or not the URL was really valid. I guess I could search the downloaded file for some variation of "404 - Page not found" but I don't know how much that string varies from web server to web server. And it seems a hassle anyway. I had a look at http://www.exelisvis.com/docs/socket.html, too. Promisingly it has an ERROR keyword but I don't understand how to tell it which file I want. Seems possible only to specify the host but not which file on the host. Near the bottom of the socket.html page there are two links to pages that promise to tell me how to read web pages and access ftp servers through socket, but when I click on them I get "Article does not exist or Permission Denied" errors. So, how does one make idl download files from the web - and tell you if it worked?
creator: Mats Löfdahl
date: 2013-12-16T12:37:11-00:00

Re: Download files from the web
type: item
description: Mats Löfdahl writes: > I need to make an idl program download a couple of text files from the web. I found the webget() function from astrolib, now also distributed with idl, see http://www.exelisvis.com/docs/webget.html > > My problem with that is that I don't know if the download succeeded. If I give a non-existing URL, I get the 404 error page downloaded and everything looks fine. If I set the COPYFILE keyword to some local file name, the file gets stored there and instead of the file contents, the return value is a scalar long. The web page does not say how this number should be interpreted but it seems to be unity whether or not the URL was really valid. > > I guess I could search the downloaded file for some variation of "404 - Page not found" but I don't know how much that string varies from web server to web server. And it seems a hassle anyway. > > I had a look at http://www.exelisvis.com/docs/socket.html, too. Promisingly it has an ERROR keyword but I don't understand how to tell it which file I want. Seems possible only to specify the host but not which file on the host. Near the bottom of the socket.html page there are two links to pages that promise to tell me how to read web pages and access ftp servers through socket, but when I click on them I get "Article does not exist or Permission Denied" errors. > > So, how does one make idl download files from the web - and tell you if it worked? It seems like FILE_TEST might be helpful. Cheers, David -- David Fanning, Ph.D. Fanning Software Consulting, Inc. Coyote's Guide to IDL Programming: http://www.idlcoyote.com/ Sepore ma de ni thue. ("Perhaps thou speakest truth.")
creator: David Fanning
date: 2013-12-16T12:46:00-00:00

Re: Download files from the web
type: item
description: On Monday, December 16, 2013 1:37:11 PM UTC+1, Mats Löfdahl wrote: > I need to make an idl program download a couple of text files from the web. I found the webget() function from astrolib, now also distributed with idl, see http://www.exelisvis.com/docs/webget.html > > > > My problem with that is that I don't know if the download succeeded. If I give a non-existing URL, I get the 404 error page downloaded and everything looks fine. If I set the COPYFILE keyword to some local file name, the file gets stored there and instead of the file contents, the return value is a scalar long. The web page does not say how this number should be interpreted but it seems to be unity whether or not the URL was really valid. > > > > I guess I could search the downloaded file for some variation of "404 - Page not found" but I don't know how much that string varies from web server to web server. And it seems a hassle anyway. > > > > I had a look at http://www.exelisvis.com/docs/socket.html, too. Promisingly it has an ERROR keyword but I don't understand how to tell it which file I want. Seems possible only to specify the host but not which file on the host. Near the bottom of the socket.html page there are two links to pages that promise to tell me how to read web pages and access ftp servers through socket, but when I click on them I get "Article does not exist or Permission Denied" errors. > > > > So, how does one make idl download files from the web - and tell you if it worked? Hi Mats, if it helps, i used the IDLnetUrl object and then use the getProperty method to get the Response_code value. Not sure if it helps, but I have been downloading files successfully with https. I also use the callback_function to make a progress bar. Not sure if it helps, but it might be a place to start... Cheers, H
creator: Helder Marchetto
date: 2013-12-16T12:47:14-00:00

Re: Download files from the web
type: item
description: Den måndagen den 16:e december 2013 kl. 13:46:00 UTC+1 skrev David Fanning: > Mats Löfdahl writes: > >> I need to make an idl program download a couple of text files from the web. I found the webget() function from astrolib, now also distributed with idl, see http://www.exelisvis.com/docs/webget.html > >> My problem with that is that I don't know if the download succeeded. If I give a non-existing URL, I get the 404 error page downloaded and everything looks fine. If I set the COPYFILE keyword to some local file name, the file gets stored there and instead of the file contents, the return value is a scalar long. The web page does not say how this number should be interpreted but it seems to be unity whether or not the URL was really valid. > >> I guess I could search the downloaded file for some variation of "404 - Page not found" but I don't know how much that string varies from web server to web server. And it seems a hassle anyway. > >> I had a look at http://www.exelisvis.com/docs/socket.html, too. Promisingly it has an ERROR keyword but I don't understand how to tell it which file I want. Seems possible only to specify the host but not which file on the host. Near the bottom of the socket.html page there are two links to pages that promise to tell me how to read web pages and access ftp servers through socket, but when I click on them I get "Article does not exist or Permission Denied" errors. > >> So, how does one make idl download files from the web - and tell you if it worked? > > It seems like FILE_TEST might be helpful. I'm not sure if you mean I should use it locally after downloading to see if the file was downloaded or remotely to see if the file is there before I try to download it. If the locally, it would find *a* file also if that file is only partially downloaded or if it is a 404 error html file. If the remotely, I guess it is nice to know that the file exists on the server but that does not guarantee that I will be able to download it. And, altho…
creator: Mats Löfdahl
date: 2013-12-16T13:24:59-00:00

Re: Download files from the web
type: item
description: Mats Löfdahl writes: > I'm not sure if you mean I should use it locally after downloading to see if the file was downloaded or remotely to see if the file is there before I try to download it. > > If the locally, it would find *a* file also if that file is only partially downloaded or if it is a 404 error html file. > > If the remotely, I guess it is nice to know that the file exists on the server but that does not guarantee that I will be able to download it. And, although I see there is a SOCKET keyword that I didn't know about, I still don't know how to specify the remote file with the socket command. I meant after the file was downloaded to tell if it was there, was the right size, etc. But, I think Helder's suggestion to use the functionality of the IDLnetURL object is likely to be more useful. Cheers, David -- David Fanning, Ph.D. Fanning Software Consulting, Inc. Coyote's Guide to IDL Programming: http://www.idlcoyote.com/ Sepore ma de ni thue. ("Perhaps thou speakest truth.")
creator: David Fanning
date: 2013-12-16T13:29:38-00:00

Re: Download files from the web
type: item
description: Den måndagen den 16:e december 2013 kl. 13:47:14 UTC+1 skrev Helder: > On Monday, December 16, 2013 1:37:11 PM UTC+1, Mats Löfdahl wrote: > >> I need to make an idl program download a couple of text files from the web. I found the webget() function from astrolib, now also distributed with idl, see http://www.exelisvis.com/docs/webget.html > >> > >> > >> > >> My problem with that is that I don't know if the download succeeded. If I give a non-existing URL, I get the 404 error page downloaded and everything looks fine. If I set the COPYFILE keyword to some local file name, the file gets stored there and instead of the file contents, the return value is a scalar long. The web page does not say how this number should be interpreted but it seems to be unity whether or not the URL was really valid. > >> > >> > >> > >> I guess I could search the downloaded file for some variation of "404 - Page not found" but I don't know how much that string varies from web server to web server. And it seems a hassle anyway. > >> > >> > >> > >> I had a look at http://www.exelisvis.com/docs/socket.html, too. Promisingly it has an ERROR keyword but I don't understand how to tell it which file I want. Seems possible only to specify the host but not which file on the host. Near the bottom of the socket.html page there are two links to pages that promise to tell me how to read web pages and access ftp servers through socket, but when I click on them I get "Article does not exist or Permission Denied" errors. > >> > >> > >> > >> So, how does one make idl download files from the web - and tell you if it worked? > > > > Hi Mats, > > if it helps, i used the IDLnetUrl object and then use the getProperty method to get the Response_code value. Not sure if it helps, but I have been downloading files successfully with https. I also use the callback_function to make a progress bar. > > Not sure if it helps, but it might be a place to start... Looks promising. So I would create an instance of the class, …
creator: Mats Löfdahl
date: 2013-12-16T13:32:36-00:00

Re: Download files from the web
type: item
description: On Monday, December 16, 2013 2:32:36 PM UTC+1, Mats Löfdahl wrote: > Den måndagen den 16:e december 2013 kl. 13:47:14 UTC+1 skrev Helder: > >> On Monday, December 16, 2013 1:37:11 PM UTC+1, Mats Löfdahl wrote: > >> > >>> I need to make an idl program download a couple of text files from the web. I found the webget() function from astrolib, now also distributed with idl, see http://www.exelisvis.com/docs/webget.html > >> > >>> > >> > >>> > >> > >>> > >> > >>> My problem with that is that I don't know if the download succeeded. If I give a non-existing URL, I get the 404 error page downloaded and everything looks fine. If I set the COPYFILE keyword to some local file name, the file gets stored there and instead of the file contents, the return value is a scalar long. The web page does not say how this number should be interpreted but it seems to be unity whether or not the URL was really valid. > >> > >>> > >> > >>> > >> > >>> > >> > >>> I guess I could search the downloaded file for some variation of "404 - Page not found" but I don't know how much that string varies from web server to web server. And it seems a hassle anyway. > >> > >>> > >> > >>> > >> > >>> > >> > >>> I had a look at http://www.exelisvis.com/docs/socket.html, too. Promisingly it has an ERROR keyword but I don't understand how to tell it which file I want. Seems possible only to specify the host but not which file on the host. Near the bottom of the socket.html page there are two links to pages that promise to tell me how to read web pages and access ftp servers through socket, but when I click on them I get "Article does not exist or Permission Denied" errors. > >> > >>> > >> > >>> > >> > >>> > >> > >>> So, how does one make idl download files from the web - and tell you if it worked? > >> > >> > >> > >> Hi Mats, > >> > >> if it helps, i used the IDLnetUrl object and then use the getProperty method to get the Response_code value. Not sure if it helps, but I have been downloading files successfully …
creator: Helder Marchetto
date: 2013-12-16T13:41:08-00:00

Re: Download files from the web
type: item
description: Helder writes: > When I download a big file the progress bar does not update after a few seconds. Any idea why? (Actually any widget stops updating...) You might try adding a EMPTY command after updating to flush the graphics buffer. Cheers, David -- David Fanning, Ph.D. Fanning Software Consulting, Inc. Coyote's Guide to IDL Programming: http://www.idlcoyote.com/ Sepore ma de ni thue. ("Perhaps thou speakest truth.")
creator: David Fanning
date: 2013-12-16T14:10:33-00:00

Re: Download files from the web
type: item
description: Den måndagen den 16:e december 2013 kl. 14:41:08 UTC+1 skrev Helder: > >>> if it helps, i used the IDLnetUrl object and then use the getProperty method to get the Response_code value. Not sure if it helps, but I have been downloading files successfully with https. I also use the callback_function to make a progress bar. > >>> Not sure if it helps, but it might be a place to start... Thanks. But it seems it has the same problem as the webget function, in that it can't tell the difference between a proper download and a 404 error web page. I simplified your code a bit (because I don't need the progress bar) and came up with this: function downloadurl, url, file url_scheme = (strsplit(url, ':',/extract))[0] url_hostname = strjoin((strsplit(url,'/',/extract))[1:*],'/') oUrl = OBJ_NEW('IDLnetUrl', URL_SCHEME = url_scheme, URL_HOSTNAME = url_hostname) retrievedFilePath = oUrl->Get(FILENAME=file) oUrl->GetProperty, RESPONSE_CODE=RespCode ; 200 = OK oUrl->CloseConnections OBJ_DESTROY, oUrl return, RespCode eq 200 ; True if OK end I tried it both with a url pointing to an existing web page and to a non-existing page. In both cases I get RespCode eq 200. With the non-existing page I again had downloaded a 404 error page. I got the value 200 for OK from the list at http://www.exelisvis.com/docs/IDLnetURL.html#objects_network _1009015_1417867
creator: Mats Löfdahl
date: 2013-12-16T14:14:10-00:00

Re: Download files from the web
type: item
description: Den måndagen den 16:e december 2013 kl. 15:14:10 UTC+1 skrev Mats Löfdahl: > Den måndagen den 16:e december 2013 kl. 14:41:08 UTC+1 skrev Helder: > >>>> if it helps, i used the IDLnetUrl object and then use the getProperty method to get the Response_code value. Not sure if it helps, but I have been downloading files successfully with https. I also use the callback_function to make a progress bar. > >>>> Not sure if it helps, but it might be a place to start... > > Thanks. But it seems it has the same problem as the webget function, in that it can't tell the difference between a proper download and a 404 error web page. > > I simplified your code a bit (because I don't need the progress bar) and came up with this: > > function downloadurl, url, file > > url_scheme = (strsplit(url, ':',/extract))[0] > > url_hostname = strjoin((strsplit(url,'/',/extract))[1:*],'/') > > oUrl = OBJ_NEW('IDLnetUrl', URL_SCHEME = url_scheme, URL_HOSTNAME = url_hostname) > > retrievedFilePath = oUrl->Get(FILENAME=file) > > oUrl->GetProperty, RESPONSE_CODE=RespCode ; 200 = OK > > oUrl->CloseConnections > > OBJ_DESTROY, oUrl > > return, RespCode eq 200 ; True if OK > > end > > I tried it both with a url pointing to an existing web page and to a non-existing page. In both cases I get RespCode eq 200. With the non-existing page I again had downloaded a 404 error page. > > I got the value 200 for OK from the list at http://www.exelisvis.com/docs/IDLnetURL.html#objects_network _1009015_1417867 Thought it might work better to use spawn and wget and read its exit status. But that seems to have the same problem: 404 error page downloaded in case the remote file doesn't exist, but exit status 0 (=OK) regardless. So this does not seem to be an IDL problem. It is just hard to get the information I want from the download process. The web server obviously knows the requested file does not exist but isn't there a way to make it tell the downloading process this in a more condensed way than constructing a …
creator: Mats Löfdahl
date: 2013-12-16T14:46:05-00:00

Re: Download files from the web
type: item
description: On 12/16/13, 7:46 AM, Mats Löfdahl wrote: > Den måndagen den 16:e december 2013 kl. 15:14:10 UTC+1 skrev Mats > Löfdahl: >> Den måndagen den 16:e december 2013 kl. 14:41:08 UTC+1 skrev >> Helder: >> >>>> > if it helps, i used the IDLnetUrl object and then use the >>>> > getProperty method to get the Response_code value. Not sure >>>> > if it helps, but I have been downloading files successfully >>>> > with https. I also use the callback_function to make a >>>> > progress bar. >> >>>> > Not sure if it helps, but it might be a place to start... >> >> Thanks. But it seems it has the same problem as the webget >> function, in that it can't tell the difference between a proper >> download and a 404 error web page. >> >> I simplified your code a bit (because I don't need the progress >> bar) and came up with this: >> >> function downloadurl, url, file >> >> url_scheme = (strsplit(url, ':',/extract))[0] >> >> url_hostname = strjoin((strsplit(url,'/',/extract))[1:*],'/') >> >> oUrl = OBJ_NEW('IDLnetUrl', URL_SCHEME = url_scheme, URL_HOSTNAME = >> url_hostname) >> >> retrievedFilePath = oUrl->Get(FILENAME=file) >> >> oUrl->GetProperty, RESPONSE_CODE=RespCode ; 200 = OK >> >> oUrl->CloseConnections >> >> OBJ_DESTROY, oUrl >> >> return, RespCode eq 200 ; True if OK >> >> end >> >> I tried it both with a url pointing to an existing web page and to >> a non-existing page. In both cases I get RespCode eq 200. With the >> non-existing page I again had downloaded a 404 error page. >> >> I got the value 200 for OK from the list at >> http://www.exelisvis.com/docs/IDLnetURL.html#objects_network _1009015_1417867 > >> > Thought it might work better to use spawn and wget and read its exit > status. But that seems to have the same problem: 404 error page > downloaded in case the remote file doesn't exist, but exit status 0 > (=OK) regardless. > > So this does not seem to be an IDL problem. It is just hard to get > the information I want from the download process. > > The web server obvio…
creator: Michael Galloy
date: 2013-12-16T21:27:02-00:00

Re: Download files from the web
type: item
description: Den måndagen den 16:e december 2013 kl. 22:27:02 UTC+1 skrev Mike Galloy: > On 12/16/13, 7:46 AM, Mats Lï¿½fdahl wrote: >> Den mï¿½ndagen den 16:e december 2013 kl. 15:14:10 UTC+1 skrev Mats Lï¿½fdahl: >>> >>> Thanks. But it seems it has the same problem as the webget >>> function, in that it can't tell the difference between a proper >>> download and a 404 error web page. >>> >>> I simplified your code a bit (because I don't need the progress >>> bar) and came up with this: >>> >>> function downloadurl, url, file >>> >>> url_scheme = (strsplit(url, ':',/extract))[0] >>> >>> url_hostname = strjoin((strsplit(url,'/',/extract))[1:*],'/') >>> >>> oUrl = OBJ_NEW('IDLnetUrl', URL_SCHEME = url_scheme, URL_HOSTNAME = >>> url_hostname) >>> >>> retrievedFilePath = oUrl->Get(FILENAME=file) >>> >>> oUrl->GetProperty, RESPONSE_CODE=RespCode ; 200 = OK >>> >>> oUrl->CloseConnections >>> >>> OBJ_DESTROY, oUrl >>> >>> return, RespCode eq 200 ; True if OK >>> >>> end >>> >>> I tried it both with a url pointing to an existing web page and to >>> a non-existing page. In both cases I get RespCode eq 200. With the >>> non-existing page I again had downloaded a 404 error page. >>> >>> I got the value 200 for OK from the list at >>> http://www.exelisvis.com/docs/IDLnetURL.html#objects_network _1009015_1417867 > >> Thought it might work better to use spawn and wget and read its exit >> status. But that seems to have the same problem: 404 error page >> downloaded in case the remote file doesn't exist, but exit status 0 >> (=OK) regardless. > >> So this does not seem to be an IDL problem. It is just hard to get >> the information I want from the download process. > >> The web server obviously knows the requested file does not exist but >> isn't there a way to make it tell the downloading process this in a >> more condensed way than constructing a web page with a 404 error? > > I use IDLnetURL in my routine and it is able to tell if there is a 404 > error: > > IDL> c = mg_get_url_content('ht…
creator: Mats Löfdahl
date: 2013-12-17T09:39:42-00:00

Re: Download files from the web
type: item
description: On Tuesday, December 17, 2013 10:39:42 AM UTC+1, Mats Löfdahl wrote: > Den måndagen den 16:e december 2013 kl. 22:27:02 UTC+1 skrev Mike Galloy: > >> On 12/16/13, 7:46 AM, Mats Lï¿½fdahl wrote: > >>> Den mï¿½ndagen den 16:e december 2013 kl. 15:14:10 UTC+1 skrev Mats Lï¿½fdahl: > >>>> > >>>> Thanks. But it seems it has the same problem as the webget > >>>> function, in that it can't tell the difference between a proper > >>>> download and a 404 error web page. > >>>> > >>>> I simplified your code a bit (because I don't need the progress > >>>> bar) and came up with this: > >>>> > >>>> function downloadurl, url, file > >>>> > >>>> url_scheme = (strsplit(url, ':',/extract))[0] > >>>> > >>>> url_hostname = strjoin((strsplit(url,'/',/extract))[1:*],'/') > >>>> > >>>> oUrl = OBJ_NEW('IDLnetUrl', URL_SCHEME = url_scheme, URL_HOSTNAME = > >>>> url_hostname) > >>>> > >>>> retrievedFilePath = oUrl->Get(FILENAME=file) > >>>> > >>>> oUrl->GetProperty, RESPONSE_CODE=RespCode ; 200 = OK > >>>> > >>>> oUrl->CloseConnections > >>>> > >>>> OBJ_DESTROY, oUrl > >>>> > >>>> return, RespCode eq 200 ; True if OK > >>>> > >>>> end > >>>> > >>>> I tried it both with a url pointing to an existing web page and to > >>>> a non-existing page. In both cases I get RespCode eq 200. With the > >>>> non-existing page I again had downloaded a 404 error page. > >>>> > >>>> I got the value 200 for OK from the list at > >>>> http://www.exelisvis.com/docs/IDLnetURL.html#objects_network _1009015_1417867 > >> > >>> Thought it might work better to use spawn and wget and read its exit > >>> status. But that seems to have the same problem: 404 error page > >>> downloaded in case the remote file doesn't exist, but exit status 0 > >>> (=OK) regardless. > >> > >>> So this does not seem to be an IDL problem. It is just hard to get > >>> the information I want from the download process. > >> > >>> The web server obviously knows the requested file does not exist but > >>> isn't there a way to make it tell the dow…
creator: Helder Marchetto
date: 2013-12-17T09:51:07-00:00

Re: Download files from the web
type: item
description: On Tuesday, December 17, 2013 10:51:07 AM UTC+1, Helder wrote: > On Tuesday, December 17, 2013 10:39:42 AM UTC+1, Mats Löfdahl wrote: > >> Den måndagen den 16:e december 2013 kl. 22:27:02 UTC+1 skrev Mike Galloy: > >> > >>> On 12/16/13, 7:46 AM, Mats Lï¿½fdahl wrote: > >> > >>>> Den mï¿½ndagen den 16:e december 2013 kl. 15:14:10 UTC+1 skrev Mats Lï¿½fdahl: > >> > >>>> > > >> > >>>> > Thanks. But it seems it has the same problem as the webget > >> > >>>> > function, in that it can't tell the difference between a proper > >> > >>>> > download and a 404 error web page. > >> > >>>> > > >> > >>>> > I simplified your code a bit (because I don't need the progress > >> > >>>> > bar) and came up with this: > >> > >>>> > > >> > >>>> > function downloadurl, url, file > >> > >>>> > > >> > >>>> > url_scheme = (strsplit(url, ':',/extract))[0] > >> > >>>> > > >> > >>>> > url_hostname = strjoin((strsplit(url,'/',/extract))[1:*],'/') > >> > >>>> > > >> > >>>> > oUrl = OBJ_NEW('IDLnetUrl', URL_SCHEME = url_scheme, URL_HOSTNAME = > >> > >>>> > url_hostname) > >> > >>>> > > >> > >>>> > retrievedFilePath = oUrl->Get(FILENAME=file) > >> > >>>> > > >> > >>>> > oUrl->GetProperty, RESPONSE_CODE=RespCode ; 200 = OK > >> > >>>> > > >> > >>>> > oUrl->CloseConnections > >> > >>>> > > >> > >>>> > OBJ_DESTROY, oUrl > >> > >>>> > > >> > >>>> > return, RespCode eq 200 ; True if OK > >> > >>>> > > >> > >>>> > end > >> > >>>> > > >> > >>>> > I tried it both with a url pointing to an existing web page and to > >> > >>>> > a non-existing page. In both cases I get RespCode eq 200. With the > >> > >>>> > non-existing page I again had downloaded a 404 error page. > >> > >>>> > > >> > >>>> > I got the value 200 for OK from the list at > >> > >>>> > http://www.exelisvis.com/docs/IDLnetURL.html#objects_network _1009015_1417867 > >> > >>> > >> > >>>> Thought it might work better to use spawn and wget and read its exit > >> > >>>> status. But that seems to have the same problem: 404 error page > >> > >>>> down…
creator: Helder Marchetto
date: 2013-12-17T09:53:06-00:00

Re: Download files from the web
type: item
description: Den tisdagen den 17:e december 2013 kl. 10:51:07 UTC+1 skrev Helder: > On Tuesday, December 17, 2013 10:39:42 AM UTC+1, Mats Löfdahl wrote: >> Hm. But how to test what happens when the file exists on the server but does not get downloaded properly? Would be great if one could get the file size or even better a checksum from the server and then compare that with the same property of the downloaded file. But I don't see any of those properties listed here http://www.exelisvis.com/docs/IDLnetURL_Properties.html. > > Hi Mats, > > the files size is given in my example. Look at the output that I print out. For example: > > Check for update: Http: get: received (9594572), total expected (15501460) OK, so that is echoed during the transfer? Can you access the size information through some keyword?
creator: Mats Löfdahl
date: 2013-12-17T09:59:06-00:00

Re: Download files from the web
type: item
description: On Tuesday, December 17, 2013 10:59:06 AM UTC+1, Mats Löfdahl wrote: > Den tisdagen den 17:e december 2013 kl. 10:51:07 UTC+1 skrev Helder: > >> On Tuesday, December 17, 2013 10:39:42 AM UTC+1, Mats Löfdahl wrote: > >>> Hm. But how to test what happens when the file exists on the server but does not get downloaded properly? Would be great if one could get the file size or even better a checksum from the server and then compare that with the same property of the downloaded file. But I don't see any of those properties listed here http://www.exelisvis.com/docs/IDLnetURL_Properties.html. > >> > >> Hi Mats, > >> > >> the files size is given in my example. Look at the output that I print out. For example: > >> > >> Check for update: Http: get: received (9594572), total expected (15501460) > > > > OK, so that is echoed during the transfer? Can you access the size information through some keyword? Not that I know of (and trust me, I don't know much). However, it is included in the callback function. In my example from before, I used this info to make the progress bar. The size of the file is in progress[1] and the downloaded size is in progress[2]. If you only want to get the size, you could exit the callback function by returning a zero. However, this method far from being elegant. Therefore experience teaches: there must be a better way. What you want to do is to check the directory content (size). I *think* that using some "url-get" method you will not get anywhere, because what you get is the html page. Another idea would be to parse the html for "href" content. To do this, the web page must contain a listing of the available files. Let me know if you come up with better ideas. cheers, h
creator: Helder Marchetto
date: 2013-12-17T10:24:58-00:00

Re: Download files from the web
type: item
description: Den tisdagen den 17:e december 2013 kl. 10:39:42 UTC+1 skrev Mats Löfdahl: > Den måndagen den 16:e december 2013 kl. 22:27:02 UTC+1 skrev Mike Galloy: > >> On 12/16/13, 7:46 AM, Mats Lï¿½fdahl wrote: > >>> Den mï¿½ndagen den 16:e december 2013 kl. 15:14:10 UTC+1 skrev Mats Lï¿½fdahl: > >>>> > >>>> Thanks. But it seems it has the same problem as the webget > >>>> function, in that it can't tell the difference between a proper > >>>> download and a 404 error web page. > >>>> > >>>> I simplified your code a bit (because I don't need the progress > >>>> bar) and came up with this: > >>>> > >>>> function downloadurl, url, file > >>>> > >>>> url_scheme = (strsplit(url, ':',/extract))[0] > >>>> > >>>> url_hostname = strjoin((strsplit(url,'/',/extract))[1:*],'/') > >>>> > >>>> oUrl = OBJ_NEW('IDLnetUrl', URL_SCHEME = url_scheme, URL_HOSTNAME = > >>>> url_hostname) > >>>> > >>>> retrievedFilePath = oUrl->Get(FILENAME=file) > >>>> > >>>> oUrl->GetProperty, RESPONSE_CODE=RespCode ; 200 = OK > >>>> > >>>> oUrl->CloseConnections > >>>> > >>>> OBJ_DESTROY, oUrl > >>>> > >>>> return, RespCode eq 200 ; True if OK > >>>> > >>>> end > >>>> > >>>> I tried it both with a url pointing to an existing web page and to > >>>> a non-existing page. In both cases I get RespCode eq 200. With the > >>>> non-existing page I again had downloaded a 404 error page. > >>>> > >>>> I got the value 200 for OK from the list at > >>>> http://www.exelisvis.com/docs/IDLnetURL.html#objects_network _1009015_1417867 > >> > >>> Thought it might work better to use spawn and wget and read its exit > >>> status. But that seems to have the same problem: 404 error page > >>> downloaded in case the remote file doesn't exist, but exit status 0 > >>> (=OK) regardless. > >> > >>> So this does not seem to be an IDL problem. It is just hard to get > >>> the information I want from the download process. > >> > >>> The web server obviously knows the requested file does not exist but > >>> isn't there a way to make it tell t…
creator: Mats Löfdahl
date: 2014-01-10T10:22:16-00:00

Re: Download files from the web
type: item
description: On Friday, January 10, 2014 11:22:16 AM UTC+1, Mats Löfdahl wrote: > Den tisdagen den 17:e december 2013 kl. 10:39:42 UTC+1 skrev Mats Löfdahl: > >> Den måndagen den 16:e december 2013 kl. 22:27:02 UTC+1 skrev Mike Galloy: > >> > >>> On 12/16/13, 7:46 AM, Mats Lï¿½fdahl wrote: > >> > >>>> Den mï¿½ndagen den 16:e december 2013 kl. 15:14:10 UTC+1 skrev Mats Lï¿½fdahl: > >> > >>>> > > >> > >>>> > Thanks. But it seems it has the same problem as the webget > >> > >>>> > function, in that it can't tell the difference between a proper > >> > >>>> > download and a 404 error web page. > >> > >>>> > > >> > >>>> > I simplified your code a bit (because I don't need the progress > >> > >>>> > bar) and came up with this: > >> > >>>> > > >> > >>>> > function downloadurl, url, file > >> > >>>> > > >> > >>>> > url_scheme = (strsplit(url, ':',/extract))[0] > >> > >>>> > > >> > >>>> > url_hostname = strjoin((strsplit(url,'/',/extract))[1:*],'/') > >> > >>>> > > >> > >>>> > oUrl = OBJ_NEW('IDLnetUrl', URL_SCHEME = url_scheme, URL_HOSTNAME = > >> > >>>> > url_hostname) > >> > >>>> > > >> > >>>> > retrievedFilePath = oUrl->Get(FILENAME=file) > >> > >>>> > > >> > >>>> > oUrl->GetProperty, RESPONSE_CODE=RespCode ; 200 = OK > >> > >>>> > > >> > >>>> > oUrl->CloseConnections > >> > >>>> > > >> > >>>> > OBJ_DESTROY, oUrl > >> > >>>> > > >> > >>>> > return, RespCode eq 200 ; True if OK > >> > >>>> > > >> > >>>> > end > >> > >>>> > > >> > >>>> > I tried it both with a url pointing to an existing web page and to > >> > >>>> > a non-existing page. In both cases I get RespCode eq 200. With the > >> > >>>> > non-existing page I again had downloaded a 404 error page. > >> > >>>> > > >> > >>>> > I got the value 200 for OK from the list at > >> > >>>> > http://www.exelisvis.com/docs/IDLnetURL.html#objects_network _1009015_1417867 > >> > >>> > >> > >>>> Thought it might work better to use spawn and wget and read its exit > >> > >>>> status. But that seems to have the same problem: 404 error page > >> >…
creator: Helder Marchetto
date: 2014-01-10T10:41:20-00:00

Re: Download files from the web
type: item
description: On Friday, January 10, 2014 11:41:20 AM UTC+1, Helder wrote: > On Friday, January 10, 2014 11:22:16 AM UTC+1, Mats Löfdahl wrote: > >> Den tisdagen den 17:e december 2013 kl. 10:39:42 UTC+1 skrev Mats Löfdahl: > >> > >>> Den måndagen den 16:e december 2013 kl. 22:27:02 UTC+1 skrev Mike Galloy: > >> > >>> > >> > >>>> On 12/16/13, 7:46 AM, Mats Lï¿½fdahl wrote: > >> > >>> > >> > >>>> > Den mï¿½ndagen den 16:e december 2013 kl. 15:14:10 UTC+1 skrev Mats Lï¿½fdahl: > >> > >>> > >> > >>>> >> > >> > >>> > >> > >>>> >> Thanks. But it seems it has the same problem as the webget > >> > >>> > >> > >>>> >> function, in that it can't tell the difference between a proper > >> > >>> > >> > >>>> >> download and a 404 error web page. > >> > >>> > >> > >>>> >> > >> > >>> > >> > >>>> >> I simplified your code a bit (because I don't need the progress > >> > >>> > >> > >>>> >> bar) and came up with this: > >> > >>> > >> > >>>> >> > >> > >>> > >> > >>>> >> function downloadurl, url, file > >> > >>> > >> > >>>> >> > >> > >>> > >> > >>>> >> url_scheme = (strsplit(url, ':',/extract))[0] > >> > >>> > >> > >>>> >> > >> > >>> > >> > >>>> >> url_hostname = strjoin((strsplit(url,'/',/extract))[1:*],'/') > >> > >>> > >> > >>>> >> > >> > >>> > >> > >>>> >> oUrl = OBJ_NEW('IDLnetUrl', URL_SCHEME = url_scheme, URL_HOSTNAME = > >> > >>> > >> > >>>> >> url_hostname) > >> > >>> > >> > >>>> >> > >> > >>> > >> > >>>> >> retrievedFilePath = oUrl->Get(FILENAME=file) > >> > >>> > >> > >>>> >> > >> > >>> > >> > >>>> >> oUrl->GetProperty, RESPONSE_CODE=RespCode ; 200 = OK > >> > >>> > >> > >>>> >> > >> > >>> > >> > >>>> >> oUrl->CloseConnections > >> > >>> > >> > >>>> >> > >> > >>> > >> > >>>> >> OBJ_DESTROY, oUrl > >> > >>> > >> > >>>> >> > >> > >>> > >> > >>>> >> return, RespCode eq 200 ; True if OK > >> > >>> > >> > >>>> >> > >> > >>> > >> > >>>> >> end > >> > >>> > >> > >>>> >> > >> > >>> > >> > >>>> >> I tried it both with a url pointing to an existing web page and to > >> > >>> > >> > >>>> >> a non-exi…
creator: Helder Marchetto
date: 2014-01-10T11:08:00-00:00

Re: Download files from the web
type: item
description: Den fredagen den 10:e januari 2014 kl. 11:41:20 UTC+1 skrev Helder: > On Friday, January 10, 2014 11:22:16 AM UTC+1, Mats Löfdahl wrote: > >> [...] I can execute the last line above after this and get a response code of 404 but at that point the program has stopped. I suppose I could add some error handling code but isn't the IDLnetUrl object supposed to take care of this? > > Hi Mats, > > this will not really help, but at least you feel my pain :-) > > I'm working around this problem with a call to catch and then check the error code (404). You can then either report that the file is missing if the error is 404, otherwise simply give out the error code. I did this now, as well. Seems to work. Thanks. > I found that this is the only way if you're using http. With ftp you can of course check the list of available files with the method GetFtpDirList. > > But I guess you already know all of this. No, I'm new to downloads from within IDL. I haven't tried ftp yet. > If there's a better way... happy to hear about it. Me too. :o)
creator: Mats Löfdahl
date: 2014-01-10T14:48:34-00:00