When I started learning about the HTTP protocol and had to interact with URLs and data, almost every tool that I tested either lacked proper documentation or had overly complicated instructions for something as simple as making an HTTP request. But one day, I realized that the tool that I've been using for a long time to download files from the internet could also be the greatest tool I've found so far to interact with web APIs.
💡 This article assumes that you have a basic understanding of the HTTP protocol, that you know what a web API is, and that you are familiar with the CLI of your operating system.
Table of contents:
⚙The setup
Well, just like almost every piece of software out there, curl needs to be installed on your operating system. The good news is that most likely, your system already comes with it built-in. Almost every Linux distribution comes with curl and Windows 10 also started shipping it from version 1803, too 😉.
Would you like to check if your system has it? Well, let's give it the very first test drive.
On Windows, you may try the following command either on CMD or PowerShell:
curl.exe --version
On Linux, you may try this on your terminal:
curl --version
The command should return some information about the curl version, libraries it counts with, release date, supported protocols and features. I'm a Linux user and in my case, I got the following output:
curl 7.68.0 (x86_64-pc-linux-gnu) libcurl/7.68.0 OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh/0.9.3/openssl/zlib nghttp2/1.40.0 librtmp/2.3
Release-Date: 2020-01-08
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets
If you get an error instead, you can also download and install curl. It's free! It's available for all major operating systems out there and last time I checked, it only takes a ~5MB download.
If you're exploring this article but don't currently have an API handy to play with, not to worry. The httpbin service is used in the examples of this post 😉.
💡 Note: From this point on, bash commands will be given as examples for clarity and readability. If you are using Windows or the example snippets don't work for you, try appending .exe after the curl command or try removing the line breaks.
🧩 Basic curl options
Using curl is pretty simple and though it can handle a plethora of different options, you might want to start by getting familiar with the following ones:
--request or -X: To specify the desired HTTP request method. If the option is not present, curl will treat the request as GET. Example: curl -X POST ...--header or -H: To include an extra header on the HTTP request. Any number of extra headers can be included. Example: curl -H "X-First-Name: John" -H "X-Last-Name: Doe" ...--data or -d: To include data on the HTTP request. The data can be either included on the same command line or it can reference a text-based file to read the data from. Example: curl -d "text data goes here" ...--form or -F: Lets curl emulate a filled-in form in which a user has pressed the submit button. This also enables uploading of binary files. Example: curl -F name=John -F shoesize=11 ...--user <user:password> or -u <user:password>: Specify the username and password to use for server authentication.
curl can present the whole HTTP request transaction with great detail in the terminal, however, I prefer it when it dumps the results into files for further review. To achieve this, you can make use of these options:
--dump-header or -D: Write the received protocol headers to the specified file. Example: curl -D result.header ...--output or -o: Write output to the specified file instead of stdout. Example: curl -o result.json ...
📩 The GET method
Well, now that you are ready with the basics, it's time to GET the party started! 🥳
First, I'd recommend you to change the working directory on your CLI to one that you're familiar with such as your user directory or your desktop so that you're able to easily find the resulting files after executing the commands.
Let's make our very first request with:
curl -X GET "https://httpbin.org/get" \
-H "accept: application/json" \
-D result.headers \
-o result.json
On the first line, we're telling curl to perform an HTTP request with the GET method to the specified URL. On the second line, we add a header to let the server know what we'll accept as a response. On the third line, we tell curl to dump the resulting headers of our request to a file called result.headers. Lastly, we tell curl to write the output of the request to a file called result.json.
If everything went nice and smoothly, you should be able to see the resulting files in your working directory after executing the command. In my case, I got:
result.headers
HTTP/2 200
date: Thu, 02 Sep 2021 05:47:14 GMT
content-type: application/json
content-length: 169
In the result.headers file, we can see that the request went through successfully with a 200 HTTP response code, the response timestamp and all headers returned by the server.
result.json
{
"args": {},
"headers": {
"Accept": "application/json",
"Host": "httpbin.org",
"User-Agent": "curl/7.68.0"
},
"url": "https://httpbin.org/get"
}
The result.json file contains the body of the response. In this case, the data that gets returned by the server following the JSON notation.
Pretty neat, don't you think?
📮 The POST method
🔤 Sending plain text
This time, let's try sending a plain text message to the server by using the POST method.
curl -X POST "https://httpbin.org/post" \
-H "accept: application/json" \
-H "Content-Type: text/plain" \
-H "Custom-Header: Testing" \
-d "I love hashnode" \
-D result.headers \
-o result.json
Note that we've included the header "Custom-Header: Testing". It should get reflected in the body of the response.
result.headers
HTTP/2 200
date: Fri, 03 Sep 2021 03:16:20 GMT
content-type: application/json
content-length: 182
result.json
{
"data": "I love hashnode",
"headers": {
"Accept": "application/json",
"Content-Length": "15",
"Content-Type": "text/plain",
"Custom-Header": "Testing"
}
}
Did you notice that the server received the message along with the testing header and echoed them? That means curl was able to send the data we wanted it to.
🖇 Query string parameters
Not only sending plain text is supported by curl but also query string parameters are supported:
curl -X POST "https://httpbin.org/post?name=Carlos&last=Jasso" \
-H "accept: application/json" \
-D result.headers \
-o result.json
result.headers
HTTP/2 200
date: Fri, 03 Sep 2021 03:30:47 GMT
content-type: application/json
content-length: 120
result.json
{
"args": {
"lastname": "Jasso",
"name": "Carlos"
},
"headers": {
"Accept": "application/json"
}
}
Whoa! The server echoed the parameters curl sent to it over the query string.
💌 Sending a JSON object
Let's give curl another task and make it send a json file to the server and see what happens:
curl -X POST "https://httpbin.org/post" \
-H "Content-Type: application/json; charset=utf-8" \
-d @data.json \
-o result.json
This time, we're including a header to let the server know we're sending a json file. The data file path gets specified to curl with the -d option followed by an at sign @ and the path to the file.
Here's the content of the data.json file for you to create it on your working directory:
{
"name": "Jane",
"last": "Doe"
}
result.json
{
"data": "{ \"name\": \"Jane\", \"last\": \"Doe\"}",
"headers": {
"Accept": "*/*",
"Content-Length": "38",
"Content-Type": "application/json; charset=utf-8"
},
"json": {
"last": "Doe",
"name": "Jane"
}
}
Yet again, curl was able to send the JSON file content to the server as we can see the data echoed on the response. Kudos for curl!
Sometimes, you might want to emulate submitting a form and curl can make that possible:
curl -X POST "https://httpbin.org/post" \
-H "Content-Type: multipart/form-data" \
-F "FavoriteFood=Pizza" \
-F "FavoriteBeverage=Beer" \
-o result.json
For curl to specify to the server that what's being sent is form data, you can use a request header with the appropriate form MIME type as seen above and use the -F parameter for each of the form fields and values.
result.json
{
"form": {
"FavoriteBeverage": "Beer",
"FavoriteFood": "Pizza"
},
"headers": {
"Accept": "*/*",
"Content-Length": "261",
"Content-Type": "multipart/form-data;"
}
}
In the result, we can confirm that the server received a form and interpreted each of its fields correctly.
🖼 Sending a file
So far, we've given curl lightweight work. How about sending a file on this round? Files are sent in a similar way forms are emulated. Would you like to copy any image to your working directory and try to send it?
curl -X POST "https://httpbin.org/post" \
-H "Content-Type: multipart/form-data" \
-F "FileComment=This is a JPG file" \
-F "[email protected]" \
-o result.json
result.json
{
"files": {
"image": "data:image/jpeg;base64,/9j/4AAQSkZJ..."
},
"form": {
"FileComment": "This is a JPG file"
},
"headers": {
"Accept": "*/*",
"Content-Length": "78592",
"Content-Type": "multipart/form-data;"
}
}
It might take some time for the file to upload but in the end, the server will return the same file we sent to it (base64 encoded). curl has correctly handled every request we've built with it so far.
🎯 The other methods
I've tested as many HTTP request methods as I've been able to think of with curl and all of them have worked splendidly. After all, what the API does with each request may vary depending on the implementation of the method.
🔐 Authentication
curl can handle server authentication when accessing a URL that requires the user to input credentials. In this case, httpbin receives the expected username and password through the URL with the format /basic-auth/{user}/{passwd} to match the values with the inputted credentials. This is what happens if said credentials are not entered:
curl -X GET "https://httpbin.org/basic-auth/carlos/secret" \
-H "accept: application/json" \
-D result.headers
result.headers
HTTP/2 401
date: Fri, 03 Sep 2021 04:08:44 GMT
content-length: 0
Dang! got a 401 (unauthorized) 🤔
Oh right, we forgot to include our credentials. Let's include them in the options:
curl -X GET "https://httpbin.org/basic-auth/carlos/secret" \
-u carlos:secret
-H "accept: application/json" \
-D result.headers
result.headers
HTTP/2 200
date: Fri, 03 Sep 2021 04:14:19 GMT
content-type: application/json
content-length: 48
result.json
{
"authenticated": true,
"user": "carlos"
}
Yes! got logged in 😎.
Another common authentication method is bearer tokens but they tend to be added within a header and we have already covered that case so, I bet an example for those can be skipped.
🤹♂️ Getting creative
How have you liked curl so far? Do you think it's useful? Wait until I let you know about some extra suggestions I think you might also like.
I've seen that common API testing tools are free as well, but they put price tags on their collaboration features and that's fine. That's how they raise funds to keep their motors running but people like me who prefer free products and services might want to look somewhere else. What I like to do is to have API documentation on markdown files on a git repo like this one and share them with other people. To me, that enables collaboration with curl without having to spend a single cent.
Also, other tools offer sophisticated GUIs but I find most of them to be too cluttered. I like simplicity and if you do, too, I'd recommend using VSCode if you're not using it already. I think VSCode + curl just make a perfect match ❤
That's the workflow I normally use. You can see the rendered markdown document to the left, my result.headers & result.json to the right and a terminal at the bottom to enter all my pre-designed commands. What do you think about it?
Do you think curl can only work fine with the REST architecture? well, I've also had the chance to use it to test SOAP with the same great results. I'm just waiting to have a new scenario to put curl to the test 🤓.
I hope this post has been both fun and useful to you. Now, you know how to take advantage of curl for your API testing purposes plus you'll also look like those hackers shown on TV when you're using your terminal and text-based tools at work. If that isn't cool, I don't know what is.
If you'd like to become a curl guru and keep learning about it, I'd encourage you to visit its documentation page. There's really a vast number of different scenarios it can be used for.
Fun fact: I don't know how to use Postman.
👋
Cover Credits:
Computer vector created by upklyak - www.freepik.com |
https://commons.wikimedia.org/wiki/File:Curl-logo.svg, curl contributors, MIT http://opensource.org/licenses/mit-license.php, via Wikimedia Commons
