curl – some handy commands

Command line utility curl (on Linux, Mac and other operating systems) is pretty handy tool for web developers and can be used to multiple activities. Here are useful ways of using curl for web developers:

View request and response headers using curl

Using -v option request and response headers are printed on stderr. Here is an example:

curl -v https://infoheap.com/

Here is the sample output:

GET / HTTP/1.1
User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8y zlib/1.2.5
Host: infoheap.com
Accept: */*

HTTP/1.1 200 OK
Date: Fri, 02 May 2014 16:00:18 GMT
Server: Apache/2.2.22 (Ubuntu)
X-Powered-By: PHP/5.3.10-1ubuntu3.6
Vary: Accept-Encoding
X-Pingback: https://infoheap.com/xmlrpc.php
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8

Change Host header using curl

Sometimes we need to access a url using server IP or DNS name but change the Host header. This can be useful to hit a server which may be behind load balancer. Here is command to do this:

curl -H 'Host: infoheap.com' http://54.241.26.46/

request for compressed data to be sent using curl

Sometimes we may want to send Accept-Encoding: deflate, gzip so that data on the wire gets compressed. This may be useful when we want the request to be closer to real browser and measure latency.

curl --compressed https://infoheap.com/

Here is the extra request header which gets appended by curl for this request:

Accept-Encoding: deflate, gzip

Provide Basic auth username and password using curl

curl -u myusername:mypassword https://infoheap.com/

If you turn on verbose option -v then you can see that curl sends Authorization: header to the server as shown here:

Authorization: Basic bXl1c2VybmFtZTpteXBhc3N3b3Jk

Fetch page and follow redirection is any (http 301, 302) using curl

By default curl does not follow the redirects. Here is how you can use curl to follow redirection to curl mentioned in Location: header

curl -L https://www.infoheap.com/

Use curl to send page output to a file

Here is how you can save output to a file. It may be useful to downloads, etc.

curl https://wordpress.org/latest.tar.gz -o latest.tar.gz