cURL: An Essential Tool for Diagnostics

What is cURL? cURL, short for Client for URLs, is a command-line tool used for transferring data over various protocols. It supports a broad array of devices and can be employed to test several scenarios, such as:

• Proxy Support
• SSL Communication
• HTTP/HTTPS Requests
• HTTP Headers
• And more.

This tool is compatible with a variety of protocols, including: DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET, and TFTP.

This guide will explore common ways to utilize cURL.

Installing cURL on Windows

If you're working with a non-Linux/Unix system, such as Windows, you'll first need to install cURL, as it is not available by default in the Windows command prompt.

1. Download the cURL package from: https://curl.haxx.se/dlwiz/
2. Under “Select Type of Package,” click on “curl executable.”
3. Choose your operating system (either Win32 or Win64) and click “Select.”
4. In the "Select for What Flavor" section, pick "Generic" and then click “Select.”
5. Download the cURL version that fits your system from the options listed.

Examples of cURL Applications

Sending an HTTP GET Request

A basic use of the cURL command is sending an HTTP GET Request to retrieve a website or specific URL’s content:

curl [http or https]://www.example.com

This command will return the HTTP response from the specified URL.

Retrieving Only the HTTP Header for a URL

By using the -I flag with the cURL command, you can fetch only the HTTP headers of a specific URL (using the HEAD method):

curl -I [http or https]://www.example.com

Note: Avoid using the -I flag while debugging CDN behavior. Use the -svo option for accurate debugging, as it mimics user GET requests.

Saving cURL Command Output

The output of a cURL command can be saved as a file using the -o and -O options:

curl -o myfile.css [http or https]://cdn.vergecloud.com/css/example.css

curl -O [http or https]://cdn.vergecloud.com/css/example.css

Adding Extra HTTP Headers

To include additional headers in a GET request, use the -H (or --header) option. This is particularly useful when testing APIs to include headers like Set-Cookie, HOST, or custom headers:

curl -H "HEADER: VALUE" [http or https]://www.example.com

Remember to use a semicolon (;) after "VALUE" if not providing it directly in the command.

Specifying a Custom Address for a URL

With the --resolve flag, cURL allows you to define the server address for handling a request, bypassing DNS or the hosts file:

curl --resolve hostname:port:DESTINATIONIPADDRESS https://www.example.com

Getting More Detailed Information

If you need to see additional details, such as request headers or connection information, use the -v (or --verbose) flag:

curl -v https://www.example.com/

Saving HTTP Headers

To save the headers returned by a queried URL, use the -D (or --dump-header) flag. For example, it can be used to first capture headers, then retrieve cookies with the -b (or --cookie) flag:

curl -D - https://www.example.com/

The minus sign (-) after -D indicates that the output should be saved in stdout, typically displayed in the terminal.

GET Method

In HTTP, the GET method retrieves resources from a specified URL. By default, cURL uses the GET method, but the --request GET flag can explicitly set the request type to GET:

curl --request GET https://www.example.com/

POST Method

The POST method sends data to the server. Use the --request POST flag to send a POST request in cURL:

curl --request POST https://www.example.com/

PUT Method

The PUT method creates or updates a resource. This is useful for tasks like creating a new webpage or updating an existing one. The --request PUT flag tells cURL to use the PUT method:

curl --request PUT https://www.example.com/

DELETE Method

To delete a resource from the web server, use the DELETE method in cURL with the --request DELETE flag:

curl --request DELETE https://www.example.com

Sending Data with cURL

To send data, such as login credentials, through a POST request, use the -d (or --data) option:

curl -X POST https://www.example.com/login/ -d 'username=yourusername&password=yourpassword'

Resuming a File Download

If a download is interrupted, use the -C flag to resume the download:

curl -C - -O https://example.com/img/example.png

Testing Download Time Without Saving the File

If you want to test the download speed without saving the file, use /dev/null to discard the output:

curl -o /dev/null https://www.example.com

Setting Maximum Transfer Rate

To control the transfer speed, use the --limit-rate flag. This limits the rate in bytes per second, but it can be adjusted to kilobytes (K), megabytes (M), or gigabytes (G):

curl --limit-rate 200K -O https://assets.example.com/img/test.png

Checking for HTTP/2 Support

If you're using a modern version of cURL, the --http2 flag checks for support of HTTP/2. If the response shows HTTP/1.1 200 instead of HTTP/2 200, the server doesn’t support HTTP/2:

curl -I --http2 https://www.example.com/

Downloading Specific Parts of a File

Use the -r flag to download a range of bytes from a file:

curl -r 0-20000 -o test.png https://assets.exaample.com/img/example.png

Running cURL with a Custom User-Agent String

If a web server blocks the default cURL user-agent string, you can customize it with the --user-agent option:

curl --user-agent "USERAGENTSTRING" https://www.example.com/

The User-Agent string in HTTP headers contains details like the browser version and operating system.

Help

Use the -h option to view a list of available options and their explanations:

curl -h

Troubleshooting HTTP Errors

When diagnosing HTTP issues, it might be helpful to query the web server directly. If you don’t know the server’s IP, look it up in your domain’s DNS settings. Then, use the --header and --resolve flags to direct your request:

curl -svo /dev/null --header "Host: example.com" http://1.2.3.4/

curl -svo /dev/null --resolve example.com:80:ServerIPAddress http://example.com/

Performance Diagnosis

Time Command

You can measure the time it takes to complete an HTTP/HTTPS request with the time command:

time curl -svo /dev/null https://example.com/

The -w or --write-out Flag

Use this flag to test various variables in the cURL command’s performance. A list of available variables can be found at curl.haxx.se. An example:

curl -svo /dev/null https://example.com/ -w "\nContent Type: %{content_type} \
\nHTTP Code: %{http_code} \
\nHTTP Connect:%{http_connect} \
\nNumber Connects: %{num_connects} \
\nNumber Redirects: %{num_redirects} \
\nRedirect URL: %{redirect_url} \
\nSize Download: %{size_download} \
\nSize Upload: %{size_upload} \
\nSSL Verify: %{ssl_verify_result} \
\nTime Handshake: %{time_appconnect} \
\nTime Connect: %{time_connect} \
\nName Lookup Time: %{time_namelookup} \
\nTime Pretransfer: %{time_pretransfer} \
\nTime Redirect: %{time_redirect} \
\nTime Start Transfer: %{time_starttransfer} \
\nTime Total: %{time_total} \
\nEffective URL: %{url_effective}\n" 2>&1

Using cURL for SSL/TLS Diagnostics

cURL is useful for diagnosing SSL/TLS certificates. To gather details about a URL’s TLS handshake and certificate information, use the following command:

curl -svo /dev/null https://www.example.com/ 2>&1 | egrep -v “^{.*$|^}.*$|^\* http.*$”

The --resolve flag can also be applied here to inspect the SSL/TLS certificate of the website’s primary server.

TLS Version Test

To check if a website supports a specific TLS version, specify the version in the cURL command:

curl -svo /dev/null --tls1.2 https://www.example.com/ 2>&1 | egrep “^*  SSL”

This test forces cURL to use TLS 1.2 for testing if the server supports this version of TLS.

CDN and Domain Troubleshooting Examples Using cURL

Check the HTTP Response Headers

Verify if CDN-specific headers (e.g., CF-Cache-Status, X-Cache, CDN-Status) are present, indicating that the request is being served by the CDN:

curl -I https://example.com/

Check the Response from the Origin Server

To bypass the CDN and directly query the origin server, use the --resolve flag to map the domain to the origin server’s IP address:

curl -svo /dev/null --resolve example.com:443:192.168.1.1 https://example.com/

Debug Cache Issues

Inspect cache headers (Cache-Control, Expires, Pragma) to verify if your CDN is caching properly:

curl -svo /dev/null https://example.com/ | grep -i "Cache-Control\|Expires"

Test SSL/TLS Configuration

Check if your domain supports a specific version of TLS (e.g., TLS 1.2):

curl -svo /dev/null --tls1.2 https://example.com/

Verify DNS Resolution

Confirm the correct IP address is resolved for a domain:

curl -svo /dev/null https://example.com/ | grep "Connected to"

Measure Time-to-First-Byte (TTFB)

Evaluate the server's performance and measure the TTFB:

curl -svo /dev/null https://example.com/ -w "\nTTFB: %{time_starttransfer}\n"

Check Redirect Configuration

Analyze redirects and ensure they are configured correctly:

curl -svo /dev/null https://example.com/

If there’s a redirection loop, you’ll see multiple Location headers:

curl -svo /dev/null https://example.com/ | grep "Location"

Test Content Delivery from the CDN

Ensure specific assets are being delivered through the CDN by checking the cache status in headers like X-Cache:

curl -I https://assets.example.com/js/script.js

Simulate a Request from a Specific Geolocation

If your CDN provides regional edge servers, simulate requests from different regions using a proxy:

curl -x http://proxy-us.example.com:8080 https://example.com/

Check Domain Propagation

Troubleshoot DNS propagation issues using the --resolve flag to query the CDN endpoint or origin server:

curl -svo /dev/null --resolve example.com:80:203.0.113.1 http://example.com/

Test Large File Downloads

Diagnose download issues or verify CDN delivery performance by downloading a large file and measuring the speed:

curl -O --limit-rate 1M https://assets.example.com/largefile.zip

Inspect Compression

Check if the CDN is applying gzip or Brotli compression to the content:

curl -svo /dev/null -H "Accept-Encoding: gzip" https://example.com/ | grep "Content-Encoding"

Check the Content Length

Ensure the full response is being delivered by the CDN by comparing the Content-Length header value with the expected size:

curl -I https://example.com/ | grep "Content-Length"

Force a Cache Refresh

If you suspect stale content, add a random query parameter to bypass the CDN cache:

curl -svo /dev/null https://example.com/?cache-bypass=$(date +%s)