cURL Scripting

INTRODUCTION

Why write this?

Because who needs browsers when you can just pipe curl into bash and parse with grep, sed, and awk 😹

Sometimes I find myself in a circumstance where I want to do a little bit of automation/scripting with web requests, but I don’t feel like writing a whole Python script for it. At least for me, a really common circumstance for me wanting to do this is when I want to automate some kind of login where I encounter a CSRF check.

Using my Burp proxy and my web browser’s Dev Tools pane… Too pendandic! TOO MANY CLICKS!!

Using Python + Selenium… WAY OVERKILL FOR SUCH SIMPLE TASK!

Sometimes I would like to have a bit of a code cookbook for writing the scripts more rapidly - that’s the biggest reason why I’m writing this.

Disclaimer
There are tons of ways to do this. I’m just presenting a way that I like to use when I want to do something quick.

SNIPPETS

Below are some self-contained ways to make a request, that I find useful.

Deal with Self-signed Certs

curl -sk https://subdomain.domain.tld/resource

Use -s to suppress all the loading bars. Use -k to perform an insecure HTTPS request

POST to an API, Parse the JSON

Here, we’re expecting a 40-character hex value from a JSON response from an API

DATA="username=myname&password=password123"
COOKIE="PHPSESSID=1234567890abcdef1234567890"
RESP=$(curl -sk -b $COOKIE -d $DATA https://subdomain.domain.tld/resource/authenticate)
echo $RESP | grep -oE '[0-9a-fA-F]{40}'

Use -o with grep to only return the matched text.

Proxy through Burp

I usually have my Burp proxy on port 8081.

curl --proxy "127.0.0.1:8081" -k https://subdomain.domain.tld/resource

Combine grep and awk to extract the cookie

RESPONSE=$(curl -isk https://subdomain.domain.tld/give-me-a-cookie.php)
COOKIE=$(echo "$RESPONSE" | grep -i 'Set-Cookie' | head -n 1 | awk '{print $2}')

Use the -i flag with curl to be able to parse the response headers

Trim Off Quotes, Remove Semicolon

Easiest to do this in two operations:

RESPONSE=$(curl -isk https://subdomain.domain.tld/page-holding-important-value.php)
BODYVAL=$(echo "$RESPONSE" | grep -i 'nsp_str' | awk '{print $4}' | sed 's/\"//g')
BODYVAL=${BODYVAL%?} # Use parameter expansion to remove final character

Here, the value is the fourth word in the matched string. We immediately strip off the quotation marks using sed substitution. Then use parameter expansion to remove the final character of the value.

CONCLUSION

Using the above snippets, all kinds of crazy scripts can be written to mimic browser interaction. The possibilities are endless!