Download using curl

Can anyone tell me the command to download multiple files from a single webpage using curl.
I've tried curl -O http://www.domain.com/
but don't know what to append to the end for the following instances
1) all files of the same extension on a webpage
2) all files no matter the extension
i'm trying to grab mostly wmv and various video files but 20 of them at a time.

Curl will need to know the complete url for each file it downloads. I'm pretty sure there is no wild card expansion on the server side. If there are links to these files on the website, you might consider using wget to download the entire site:
wget -rkpq -l4 -nH -np <URL> 2>/dev/null
Or to see the links on a page you can use lynx:
lynx -dump http://www.domain.com | awk '/http/{print $2}'

Similar Messages

File download using CURL

Hi there,
I'm trying to download the file(s) from external server. The protocol which i'm using is https, to download the file i'm using CURL utility but this utility is not downloading the file. Command which i'm using is
curl -k -u {user_name}:{password} url/{filename}
the url will look something like https://filetransfer.abc.com
I tried to list out the file at least but its not working curl -kl -u {user_name}:{password} {url}
Can some one please suggest me how to use curl for downloading the files.
Thanks in advance,

I've tested it with downloading the latest openssl source from https://www.openssl.org. Can you do the same? that way you'll know your curl is working or not.
here's the output of my curl downloading the latest source:
vxlt090101:~ fritshoogland$ curl -o test.tgz -v -# https://www.openssl.org/source/openssl-0.9.8k.tar.gz
* About to connect() to www.openssl.org port 443 (#0)
*   Trying 195.30.6.166... connected
* Connected to www.openssl.org (195.30.6.166) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: /usr/share/curl/curl-ca-bundle.crt
CApath: none
* SSLv2, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server key exchange (12):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
*       subject: /C=GB/OU=Domain Control Validated/O=*.openssl.org/CN=*.openssl.org
*       start date: 2008-09-12 17:14:11 GMT
*       expire date: 2011-09-13 17:14:06 GMT
*       common name: *.openssl.org (matched)
*       issuer: /C=BE/OU=Domain Validation CA/O=GlobalSign nv-sa/CN=GlobalSign Domain Validation CA
* SSL certificate verify ok.
GET /source/openssl-0.9.8k.tar.gz HTTP/1.1
User-Agent: curl/7.16.3 (powerpc-apple-darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3
Host: www.openssl.org
Accept: */*
< HTTP/1.1 200 OK
< Date: Sun, 19 Jul 2009 10:50:43 GMT
< Server: Apache/1.3.33 (OpenPKG/2.5)
< Last-Modified: Wed, 25 Mar 2009 12:21:40 GMT
< ETag: "f0c99a-3ac7e3-49ca21d4"
< Accept-Ranges: bytes
< Content-Length: 3852259
< Content-Type: application/x-tar
< Content-Encoding: x-gzip
<
                                                                           0.0%{ [data not shown]
######################################################################## 100.0%* Connection #0 to host www.openssl.org left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
} [data not shown]

Downloading XCode using curl

I want to write a script for installing all my required tools and libraries automatically on SL and for this to work my script needs to download and install XCode on my machine. However downloading XCode requires an authenticated user and at the moment I cannot download it using curl. Is there an official mirror of XCode available that doesn't require authentication for downloading or is there a way to pass authentication info to apple.com for downloading?
At the moment the following command is not working:
curl -u [email protected]:pass --anyauth https://developer.apple.com/mac/scripts/downloader.php?path=/ios/iossdk_4.0.2__final/xcode_3.2.3_and_ios_sdk4.0.2.dmg -O -S -v
Thanks in advance,
Behi

xnav wrote:
You must use multiple curl commands with the cookie-jar option to interact with the developer site and actually log in before doing the download.
Check out the download script that comes with [this|http://blog.fieryferret.com/2008/10/appstore-clerk.html] for an example.
Message was edited by: xnav
xnav wrote:
You must use multiple curl commands with the cookie-jar option to interact with the developer site and actually log in before doing the download.
Check out the download script that comes with [this|http://blog.fieryferret.com/2008/10/appstore-clerk.html] for an example.
Message was edited by: xnav
Hi,
Thanks for the reply. I have created the following script but it's still not working:
curl -d "theAccountName=username&theAccountPW=pass" -c xcode-cookie -A "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1" https://daw.apple.com//cgi-bin/WebObjects/DSAuthWeb.woa/104/wo/W1df7WQblBsLekQzv S1tJ0/0.5.3.1.1.2.1.1.3.1.1"
curl -b xcode-cookie -c xcode-cookie -O -v -A "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1" http://adcdownload.apple.com/ios/iossdk_4.0.2__final/xcode_3.2.3_and_ios_sdk4.0.2.dmg
I can see that the first script correctly stores the cookies returned from the server in xcode-cookie, but executing the second command gives a 403 Error:
curl -b xcode-cookie -c xcode-cookie -O -v -A "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1" http://adcdownload.apple.com/ios/iossdk_4.0.2__final/xcode_3.2.3_and_ios_sdk4.0.2.dmg
* About to connect() to adcdownload.apple.com port 80 (#0)
* Trying 60.254.148.16... % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0connected
* Connected to adcdownload.apple.com (60.254.148.16) port 80 (#0)
GET /ios/iossdk_4.0.2__final/xcode_3.2.3_and_ios_sdk4.0.2.dmg HTTP/1.1
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1
Host: adcdownload.apple.com
Accept: /
Cookie: myacinfo=foo; DefaultAppleID=bar; ds01=baz
< HTTP/1.1 403 Forbidden
< Server: Apache-Coyote/1.1
< X-Powered-By: Servlet 2.4; JBoss-4.0.5.GA (build: CVSTag=Branch40 date=200610162339)/Tomcat-5.5
< no-cache: Cache-Control
< no-store: Cache-Control
< Content-Type: text/html;charset=ISO-8859-1
< Date: Mon, 30 Aug 2010 05:30:42 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
< Connection: Transfer-Encoding
< Cache-Control: no-store
<
{ [data not shown]
100 1932 0 1932 0 0 7531 0 --:--:-- --:--:-- --:--:-- 8548* Connection #0 to host adcdownload.apple.com left intact
* Closing connection #0
Any ideas what could be the problem?
Message was edited by: Behrang

Using curl in java

Hi
i am using libcurl in java..i have downloaded all the source files(5) of libcurl & dll files(3).
placed them in Test Package & compile them..
all the dll s are loading fine
& created my own curl.java
package curlPackage;
import Test.*;
public class curl
public static void main(String[] args)
try
//System.loadLibrary("javacurl");
String url = "http://www.shaadi.com";
CURL c = new CURL();
CurlGlue cg = new CurlGlue();
cg.setopt(c.OPT_URL,url);
int result = cg.perform();
cg.finalize();
System.out.println("result:"+result);
catch(Exception e)
System.out.println("Exception:"+e);
but when i am trying to run this.
it is giving me the following error
Exception in thread "main" java.lang.UnsatisfiedLinkError: Test.CurlGlue.jni_init()I
at Test.CurlGlue.jni_init(Native Method)
at Test.CurlGlue.<init>(CurlGlue.java:59)
at curlPackage.curl.main(curl.java:14)
Java Result: 1
PLEASE HELP ME.....
& if anybody used javacurl please help me

HI,
I am new to java,So I hav used curl functions in php.But in java I hav to use same curl function to connect URL thru pgm.So I hav i Got JCurl .But i dont have Curlclass files & Curl Glue etc.For installing curl in java any dll files must be loaded...? So can u help me out. Iam in Hectic .So just can u send links regardings this class files.please I eil be waiting for ur reply........
JCurl_code provides API.But I need java files of Curl.
Thanx In advnce.

Downloads used to open after download. NOw they just end up in the download folder and I have to go to the folder to find it and open it. I did not change seettings, maybe an update changed it? I cannot find a way to change this in settings.

Downloads used to open after download. NOw they just end up in the download folder and I have to go to the folder to find it and open it. I did not change seettings, maybe an update changed it? I cannot find a way to change this in settings.
== This happened ==
Every time Firefox opened
== A week or two ago. Not sure if result of most recent update.

Downloads used to open after download. NOw they just end up in the download folder and I have to go to the folder to find it and open it. I did not change seettings, maybe an update changed it? I cannot find a way to change this in settings.
== This happened ==
Every time Firefox opened
== A week or two ago. Not sure if result of most recent update.

I have two apple id accounts ,when i used to update application it used to prompt a default id and if the application was downloaded using the other id i would cancel and it would prompt the other id but not any more .

i have two apple id accounts ,when i used to update application it used to prompt a default id (the old one) and if the application was downloaded using the other id i would cancel and it would prompt the other id but not any more .
it just prompt the old id which im not signed in with im signed in using the new id and when i cancel nothing happens and when i try to update applications separately i have the same problem .
im signed in using the new id
the old one is my wife's so i can't delete it
the problem happens when i update all or each application by its own
using ios 5.0.1 on a iphone 3gs
the applications are downloaded using both the new and old id's
any help will be greatly appreciatedِ

These are user-to-user forums, you are not talking to Apple here and they don't monitor these forums - I've asked the hosts to remove yout account ids from your post.
In terms of combining accounts it's not currently possible to do so, nor to copy/transfer content from account to another.

HT4914 Will iTunes match match songs that have been downloaded using a different audible account?

Will iTunes match match songs that have been downloaded using a different audible account?

Hey there Chriskir,
It sounds like you would like to delete all the songs from your iPad but still be able to access them in iTunes Match. You can delete all the songs from the Music section of your Usage in Settings by swiping your music left:
Settings > General > Usage
iPhone, iPad, and iPod: Understanding capacity
http://support.apple.com/kb/ht1867
Thank you for using Apple Support Communities.
All the best,
Sterling

I forgot my Apple ID password and security question. Also the email I'd is not accessible as the organisation has stopped their email service. How can I now update my applications which I downloaded using this Apple ID. Please help

I forgot the password of my Apple ID that I used for downloading apps when I newly bought my mac book back in 2011 October.
Now I forgot the password and also the security question.
Also the email service provider has stopped their email service and hence m not able to use email authentication as well.
Now I am lost, how will i update the applications that I downloaded using this Apple ID.
Please help

Saunakhvshg,
contact Apple directly.

I have Lightroom 4 on disc which is installed on my Mac desktop. I want to use it on a new MacBook with no disc drive. Can I install it by download using the serial number? How?

I have Lightroom 4 on disc which is installed on my Mac desktop. I want to use it on a new MacBook with no disc drive. Can I install it by download using the serial number? If so< how?

1. Go to the following website and download the last version of Lightroom 4:Adobe - Lightroom : For Macintosh : Adobe Photoshop Lightroom 4.4.1
2. Run the installation package to install Lightroom.
3. Use your existing serial number to activate the program.

What is the best app for following of the download? I mean how much I downloaded using WIFI and how much 3D download.

I would like to follow the amount of MB I downloaded and to know how much data was downloaded using WIFI and how much using 3G (internet connection by phone network provider as AT&T and so...).
Thank you very much for any suggestions.

Personally I have tried LogMeIn Ignition, Splashtop, and AlwaysOnPC, as well as TeamViewer. Although LogMeIn Ignition is expensive I would also have to say that it is the best one I've tried. Apart from viewing the computer's screen, you can access the files and folders in a standalone file browser interface without seeing the computer's screen, this speeds things up a lot.
As for security, I'm not too sure, as the website for LogMeIn does not seem to list the details. But as with any other internet based service, like email and social networking sites, an additional account of any sort will increase your vulnerability, even if it's just by that small amount. You can try accessing the computer through a VPN when on a public wifi network for added security. However about the virus, I don't think that is possible... No need to worry about your iPad getting a virus.

Accessing BSP File Download using HTTPS URL

Hi,
I'm struggling with a problem of downloading a file from a https url. I wrote a BSP App for downloading a file from a unix server.. It works fine when I use a http URL with port 8080 and does not work when I use https.!!
Example:
https://comms.gmsanet.co.za/supplier [ download does not work ]
http://comms.gmsanet.co.za:8080/supplier [ download works ]
When I try to download using https.. it does not pull the file name and path
see code below and suggest me if anything to be chnaged.
In the Form Initialization method:
event handler fr data retrieval
DATA: i_file        type string,
      s_fields      TYPE tihttpnvp,
      s_fields_line TYPE ihttpnvp,
      multipart_form type ref to if_http_entity,
      file_upload    type xstring,
      lv_backend     type string,
      success        type string,
      entity         type ref to if_http_entity,
      file           type xstring,
      content_type   type string,
      content_filename type string,
      content_length type string,
      content_disposition type string,
      num_multiparts type i,
      i              type i value 1,
      doEcho         type string value 'X',
      value          type string,
      filename       type ZFILETAB-fileinfo,
      ext1           type string,
      ext2           type string,
      dsn            type string,
      bptype         like sy-uname,
      itab           TYPE ZFILETAB,
      itab_line      TYPE ZFILETABLINE,
      file_ext       type ZFILETABLINE,
      fileinfo       type c,
      zcount         type i.
    filename = '/NewMessge.doc'.
    content_filename = filename.
Check the extension and assign the content type
    split filename at '.' into ext1 ext2.
    case ext2.
      when 'zip'.
        content_type = 'application/x-zip-compressed'.
      when 'doc'.
        content_type = 'application/msword'.
      when 'txt'.
        content_type = 'text/plain'.
      when 'ppt' or 'pps'.
        content_type = 'application/vnd.ms-powerpoint'.
      when 'xls' or 'exe'.
        content_type = 'application/octet-stream'.
      when 'gif'.
        content_type = 'image/gif'.
      when 'jpg' or 'jpeg'.
        content_type = 'image/pjpeg'.
      when 'htm' or 'html'.
        content_type = 'text/html'.
    endcase.
    dsn = filename.
    OPEN DATASET dsn FOR INPUT IN BINARY MODE.
    IF sy-subrc NE 0.
      zmessage = 'Error opening file'.
      navigation->set_parameter( name = 'zmessage' value = zmessage ).
      navigation->goto_page( 'downloaderror.htm' ).
      exit.
    ENDIF.
    DO.
      READ DATASET dsn INTO <b>file</b>.
      EXIT.
    ENDDO.
    CLOSE DATASET dsn.
set response data to be the file content
runtime->server->response->set_data( <b>file</b> ).
runtime->server->response->set_header_field(
                                name = 'Content-Type'
                                value = content_type ).
concatenate 'attachment; filename=' filename into content_disposition.
runtime->server->response->set_header_field(
                                name = 'Content-Disposition'
                                value = content_disposition ).
set the file size in the response
content_length = xstrlen( file ).
runtime->server->response->set_header_field(
                            name = 'Content-Length'
                            value = content_length ).
runtime->server->response->delete_header_field(
                            name = 'Cache-Control' ).
runtime->server->response->delete_header_field(
                            name = 'Expires' ).
navigation->response_complete( ).
Thanks
Ajay

Hi Brian,
I have the same problem as Ajay Yeluguri. In http mode I can generate a download of an Excel document but when we use the portal in https it doesn't work.
When I try to download using https it does not pull the file name and path and when I choose download I have a error message : "Internet Explorer cannot download from ..."
I've test the point 3.2 "... including file up/download" of the BSP application IT00 and it works fine in http and https mode. My problem is not the upload but the download. And in this application the uploaded document is opened in the Internet Explorer window but I want to generate a Save as... window to download the file.
Have you an idea what i can do to solve my problem.
Thanks
Yann

Hi, I just purchased Adobe Illustrator and I cant find the serial number. When I try to download using Cloud it says it does not support my computer system. Just yesterday I finished my trial. What shall i do?

I just purchased Adobe Illustrator and I cant find the serial number. When I try to download using Cloud it says it does not support my computer system. Just yesterday I finished my trial. What shall i do?

To locate the serial number:
http://helpx.adobe.com/x-productkb/global/find-serial-number.html
http://helpx.adobe.com/x-productkb/policy-pricing/serial-number-retrieval-process-faq.html

ALV truncation while download using List--Save/send--File--Spread sheet

Experts,
In ALV output, I am trying to download using menu item
"List --> Save/send --> File --> Spread sheet" then in the popup appears
here if i GIVE the FILE NAME THEN,material number is getting truncated.
If I don't give any file name it is storing as ".xls" and not getting truncated.
How to correct this or avoid this truncation.
OR
If we can't do correction about truncation ...my function consultant is asking to
to disable "List --> Save/send --> File".
Thanks in advance.
Sam

Solved by my own....
We need to pass
i_fieldcat-no_zero = 'X'.
so that the zeros at the end won't get truncated.

File upload, download using ADF UIX and not JSP

I have examples for the file upload, download using JSP
But I want to use ADF UIX. Look and feel in this is impressing and I want to use this. Any one have example for this.
Users will select a file from their desktop
This will be stored into the database (Any document. Word, Excel)
When they query the records then the UIX column need to give a hyperlink. Clicking on the link should prompt to download the file to the local system

Sure, I use the Apache Commons File Upload package, so that is what I will discuss here ( [Commons File Upload|http://commons.apache.org/fileupload/] ).
The Commons File Upload uses instances of ProgressListener to receive upload progress status. So first create a simple class that implements ProgressListener:
public class ProgressListenerImpl implements ProgressListener {
    private int percent = 0;
    public int getPercentComplete() {
        return percent;
    public void update(long pBytesRead, long pContentLength, int pItems) {
        if (pContentLength > -1) { //content length is known;
            percent = (new Long((pBytesRead / pContentLength) * 100)).intValue();
}So in your Servlet that handles file upload you will need to create an instance of this ProgressListenerImpl, register it as a listener and store it in the session:
ServletFileUpload upload = new ServletFileUpload();
ProgressListenerImpl listener = new ProgressListenerImpl();
upload.setProgressListener(listener);
request.getSession().setAttribute("ProgressListener", listener);
...Now create another servlet that will retrieve the ProgressListenerImpl, get the percent complete and write it back (how you actually write it back is up to you, could be text, an XML file, a JSON object, up to you):
ProgressListenerImpl listener = (ProgressListenerImpl) request.getSession().getAttribute("ProgressListener");
response.getWriter().println("" + listener.getPercentComplete());
...Then your XMLHttpRequest object will call this second servlet and read the string returned as the percent complete.
HTH

I'm about to download using a gift card. What do I do?

I'm about to download using a gift card. However, whenever I sign in, I always get the message "Please review your account information." What do I do next?

Have a look here: http://support.apple.com/kb/ht1574

Download using curl

Similar Messages

Maybe you are looking for