Using Charles from the command line

If you are testing network traffic you are probably familiar with Fiddler. Fiddler is a nice, easy to use tool for monitoring network traffic.

It works very easy. You start up Fiddler and it configures your Internet Settings. Now when you start up a web browser, it automatically routes traffic through Fiddler. As you hit web pages on the browser, the HTTP requests and response show up in Fiddler. It is very easy to read and understand what is going on.

If you are using a macOS computer you will be sad to learn that Fiddler does not exist for macOS. It is a Windows only product. If you check for free options to do the same thing you find Wireshark (formerly Ethereal). But Wireshark's configuration and output assumes you have knowledge of TCP, HTTP, Sockets, packets, etc. You can get the information that you need but it is not as easy as Fiddler.

Additionally, to play back a request with some modifications is a lot harder with Wireshark than with Fiddler.

So what do you do? Charles Proxy. Unfortunately, it is not free but at $50 it is a good investment. If you are working at a company with many people needing it, there are discounts available as well.

Now if you get Charles you will find it automatically starts up and changes the Network Settings on your macOS. So all the browsers and anything which uses Network Settings, will automatically go through Charles. 

What about command line? For example, I have a Docker script which creates a container, deploys a web service and waits for someone to hit it. What if I'm creating automation using Python, Java, bash script, etc.? These do not use the macOS Network Settings. So you will see nothing in Charles.

The solution is to add the necessary information to the shell before you launch your test scripts.

The way Charles works is rather simple. If my machine is using 192.168.0.4 and I want to hit www.google.com (209.52.144.114) it might following the following route:

• 192.168.0.4
• 64.114.101.7
• 209.121.102.146
• 209.52.144.114

The way Charles works is creating a MITM (Man-In-The-Middle). So if I want Charles to be able to observe the traffic the route might be:

• 192.168.0.4
• CharlesProxy
• 64.114.101.7
• 209.121.102.146
• 209.52.144.114

The way it does this is by creating proxy settings in Network Settings. To create proxy settings on the command line you need to set certain variables. For HTTP traffic and HTTPS traffic Charles tells macOS to set it to use IP address 127.0.0.1 and port 8888.

For the command line you want to use:

export http_proxy="http://127.0.0.1:8888"
export https_proxy="http://127.0.0.1:8888"

Additionally, Charles tells the macOS to bypass certain addresses. What I do is go to System Preferences, select Network, select the Advanced... button, to to the Proxies tab.

On this page, assuming you are running Charles, you will see a bunch of addresses in the Bypass proxy settings box. Select all of them, copy them into the clipboard, go back to the command line and enter:

export no_proxy="<paste>"

With these three settings, anything you run from the command line will go through Charles.

However, if you close the shell you lose all the settings. If you want to keep the settings you can add them to your ~/.bash_profile text file. Every time you open a shell it will add the proxy information to the shell. HOWEVER, you don't want this if you are not using Charles. To disable this you need to enter:

unset http_proxy
unset https_proxy
unset no_proxy

So what I do is add the following to my ~/.bash_profile text file:

# Charles shortcuts
function charles_on {
        export http_proxy="http://127.0.0.1:8888"
        export https_proxy="http://127.0.0.1:8888"
        export no_proxy="127.0.0.1:6258, 127.0.0.1:6263, 127.0.0.1:10191, 127.0.0.1:14821, 127.0.0.1:24861, 127.0.0.1:25007, 127.0.0.1:38151, 127.0.0.1:46360, 127.0.0.1:49801, 127.0.0.1:55730, 127.0.0.1:59483"
}
function charles_off {
        unset http_proxy
        unset https_proxy
        unset no_proxy
}

By adding this to my ~/.bash_profile text file I can use:

charles_on

to enable Charles. And I can use:

charles_off

to disable Charles. Whenever Charles is not running I MUST disable Charles on the command line.