The curl command uses to download multiple files at the same time.

That’s why we often get requests from customers to install curl command as part of our Technical Support Services.

Today, we’ll see the uses of curl command and how to use curl command in different ways.

Importance of Curl

Client URL, or cURL, is a library and command-line utility for data transfer through systems. Also, it uses for downloading files from the web.

It supports many protocols including HTTP, HTTPS, TELNET, SCP, FTP, etc, and tends to be installed by default on many Unix-like operating systems.

In a server environment, this command becomes a great option when need to download a file to the local machine.

How to download files with cURL

Here, we use the curl command to download a text file from a web server and view its contents, save it locally, and tell curl to follow redirects if files have moved.

1. Fetching remote files- The curl command will fetch a file and display its contents to the standard output.

Let us give it a try by downloading the robots.txt file from bobcares.com:

curl https://www.bobcares.com/robots.txt

We can see the file’s contents displayed on the screen as below.

User-agent: *
Disallow:

sitemap: https://www.bobcares.com/sitemap.xml
sitemap: https://www.bobcares.com/community/main_sitemap.xml.gz
sitemap: https://www.bobcares.com/community/questions_sitemap.xml.gz
sitemap: https://www.bobcares.com/community/users_sitemap.xml.gz

Also, if we curl a URL, then it fetches the resource and displays its contents.

2. Saving Remote Files- Curl command is also used to save the contents after downloading it

To save the remote file to the local system, we add the –remote-name argument or use the -O option with the filename that needs to be downloaded.

curl -O https://www.bobcares.com/robots.txt

[secondary_label Output
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 286 0 286 0 0 5296 0 --:--:-- --:--:-- --:--:-- 5296

Instead of displaying the contents of the file, it displays a text-based progress meter and saves the file to the same name as the remote file’s name. This can check with the cat command:

cat robots.txt

The file contains the same contents you saw previously:

Now, let us look at specifying a filename for the downloaded file.

3. Saving Remote Files with a Specific File Name- We can use the -o or –output argument, followed by the name of the local file that would like to save the contents. This helps to avoid overwriting the local file of the same name.

Execute the following command to download the remote robots.txt file to the locally named do-bots.txt file:

curl -o do-bots.txt https://www.bobcares.com/robots.txt

4. Following Redirects

By default, curl does not follow redirects, so when files move, it does not work as we expect.

For example, if we run the curl command without any protocol, it does not show any output, because Bobcares redirects requests from http:// to https://.

curl -I www.bobcares.com

The output tells that the URL redirected.

[secondary_label Output
HTTP/1.1 301 Moved Permanently
Date: Wed, 26 Aug 2020 19:01:33 GMT
Connection: keep-alive
Cache-Control: max-age=3600
Expires: Wed, 26 Aug 2020 20:01:33 GMT
Location: https://www.bobcares.com/robots.txt
cf-request-id: 04cdbea7a40000c5cc8d34d200000001
Server: cloudflare
CF-RAY: 5c8fcd52aea0c5cc-EWR

So, the use of the –location or -L argument solves the issue. It tells curl to redo the request to the new location whenever it encounters a redirect.

curl -L www.bobcares.com/robots.txt

In addition, we can use the argument -L with some of the aforementioned arguments to download the file to the local system:

curl -L -o do-bots.txt www.bobcares.com/robots.txt

Downloading multiple files with curl

Basically, instead of downloading multiple files one by one, we can download all of them simultaneously by running a curl command.

For that, we use the following syntax.

curl -O [URL1] -O [URL2]

Also, we can download multiple files from the FTP server using the Curl command. For that, we run the below command.

curl -u ftp_user:ftp_pass -O ftp://ftp_url/file_name.zip

Conclusion

In short, curl let’s download files simultaneously from a remote system. Today, we saw how our Support Engineers download multiple files using curl utility.