Using Charles from the command line

If you are testing network traffic you are probably familiar with Fiddler. Fiddler is a nice, easy to use tool for monitoring network traffic.

It works very easy. You start up Fiddler and it configures your Internet Settings. Now when you start up a web browser, it automatically routes traffic through Fiddler. As you hit web pages on the browser, the HTTP requests and response show up in Fiddler. It is very easy to read and understand what is going on.

If you are using a macOS computer you will be sad to learn that Fiddler does not exist for macOS. It is a Windows only product. If you check for free options to do the same thing you find Wireshark (formerly Ethereal). But Wireshark's configuration and output assumes you have knowledge of TCP, HTTP, Sockets, packets, etc. You can get the information that you need but it is not as easy as Fiddler.

Additionally, to play back a request with some modifications is a lot harder with Wireshark than with Fiddler.

So what do you do? Charles Proxy. Unfortunately, it is not free but at $50 it is a good investment. If you are working at a company with many people needing it, there are discounts available as well.

Now if you get Charles you will find it automatically starts up and changes the Network Settings on your macOS. So all the browsers and anything which uses Network Settings, will automatically go through Charles. 

What about command line? For example, I have a Docker script which creates a container, deploys a web service and waits for someone to hit it. What if I'm creating automation using Python, Java, bash script, etc.? These do not use the macOS Network Settings. So you will see nothing in Charles.

The solution is to add the necessary information to the shell before you launch your test scripts.

The way Charles works is rather simple. If my machine is using 192.168.0.4 and I want to hit www.google.com (209.52.144.114) it might following the following route:

• 192.168.0.4
• 64.114.101.7
• 209.121.102.146
• 209.52.144.114

The way Charles works is creating a MITM (Man-In-The-Middle). So if I want Charles to be able to observe the traffic the route might be:

• 192.168.0.4
• CharlesProxy
• 64.114.101.7
• 209.121.102.146
• 209.52.144.114

The way it does this is by creating proxy settings in Network Settings. To create proxy settings on the command line you need to set certain variables. For HTTP traffic and HTTPS traffic Charles tells macOS to set it to use IP address 127.0.0.1 and port 8888.

For the command line you want to use:

export http_proxy="http://127.0.0.1:8888"
export https_proxy="http://127.0.0.1:8888"

Additionally, Charles tells the macOS to bypass certain addresses. What I do is go to System Preferences, select Network, select the Advanced... button, to to the Proxies tab.

On this page, assuming you are running Charles, you will see a bunch of addresses in the Bypass proxy settings box. Select all of them, copy them into the clipboard, go back to the command line and enter:

export no_proxy="<paste>"

With these three settings, anything you run from the command line will go through Charles.

However, if you close the shell you lose all the settings. If you want to keep the settings you can add them to your ~/.bash_profile text file. Every time you open a shell it will add the proxy information to the shell. HOWEVER, you don't want this if you are not using Charles. To disable this you need to enter:

unset http_proxy
unset https_proxy
unset no_proxy

So what I do is add the following to my ~/.bash_profile text file:

# Charles shortcuts
function charles_on {
        export http_proxy="http://127.0.0.1:8888"
        export https_proxy="http://127.0.0.1:8888"
        export no_proxy="127.0.0.1:6258, 127.0.0.1:6263, 127.0.0.1:10191, 127.0.0.1:14821, 127.0.0.1:24861, 127.0.0.1:25007, 127.0.0.1:38151, 127.0.0.1:46360, 127.0.0.1:49801, 127.0.0.1:55730, 127.0.0.1:59483"
}
function charles_off {
        unset http_proxy
        unset https_proxy
        unset no_proxy
}

By adding this to my ~/.bash_profile text file I can use:

charles_on

to enable Charles. And I can use:

charles_off

to disable Charles. Whenever Charles is not running I MUST disable Charles on the command line.

DIRSTACK

I was recently poking around on my Terminal (Mac OS X) and I noticed one of the environment variables was:

DIRSTACK=()

So I checked the man page for the bash shell to see what I could find about it:

man bash

Reading the man page I find DIRSTACK is an array relating to popd, pushd and dirs. Rather than using cd to change to a directory I can use pushd. For example:

pushd ~/Downloads

This will change directory to ~/Downloads plus it will add the directory to the DIRSTACK array. I can add some more to the DIRSTACK using:

pushd ~/Documents
pushd /Volumes

Now if I issue a dirs I will see:

/Volumes ~/Documents ~/Downloads

If you search for popd, pushd and dirs on the bash man page you will find all the settings for these builtin commands:

dirs [-clpv] [+n] [-n]

+n display the nth entry from the left, e.g. +2 will display the entry in position 2, this is zero-indexed
-n displays the nth entry from the right, just like the +n this is zero-indexed, e.g. -0 is the first entry
-c clears the DIRSTACK
-l displays a longer list, e.g. ~ gets expanded to the full directory name /Users/darrell
-p display one entry per line
-v display one entry per line with a number at the start of each line

You might thing the -v option is just line numbers but they are more than that. The numbers are directly related to the -n and +n option. Additionally, I can refer to specific entries in the list using ~n. For example, if the dirs -v displays:

 0  ~/Public
 1  ~/Downloads
 2  ~/Documents
 3  ~

then ls -l ~2 will be the same as ls -l ~/Documents. I can also use the tilde notation for popping elements off the stack as well. The next command, popd, has the following format:

popd [-n] [+n] [-n]

-n is literally -n, when you normally popd it will change to the directory you pop, -n will suppress this

+n removes n entries from the left, e.g. +2 will remove from third element from left (zero-indexed)

-n removes n entries from the right, e.g. -1 will pick the second element from right

The pushd commands looks similar:

pushd [-n] [dir]
pushd [-n] [+n] [-n]

-n is literally -n, and like popd it adds to the stack but does not cd to the new directory.

[dir] will push [dir] on the DIRSTACK then cd [dir]

+n will rotate the stack so the nth directory from the left is at the top

-n will roate the stack so the nth directory from the right is at the top

.