Hello old friend

It has been a while since I wrote to my blog. What happened? Life.

But why am I back here? The macOS Catalina 10.15.xx.

I'm still working as a consultant testing software my colleagues have written. Recently, I got a new laptop for work and it came with the Catalina OS.

Over the last year I have noticed that Apple Macintosh has become the new Microsoft Windows. Whenever I get an update or install something new and things behave badly (Wifi won't connect) I have learned to reboot my computer. Even though the update or installation didn't say it needed to reboot my computer, I'm finding rebooting my computer actually fixes things from time to time.

Until recently. Normally, to save time, I never shutdown or log out of my laptop unless I feel I REALLY need to. So at the end of the day, I close my laptop and it goes to sleep. I have it plugged in all day. So when I close it, it is at 100% battery power. The next morning, I open my laptop and it is often at 100% power. Sometimes it drops to 99% power.

Since I upgraded to Catalina this is no longer true. I closed my laptop and didn't use it over the weekend. Monday morning I open my laptop and it is at 7% battery remaining. What the heck!?!!

So after a little digging I find that the default settings for a lot of things leave background processes running, even when it should be in sleep mode. Here is what I did to stop my laptop from dying when the lid is closed:

1. Go to System Preferences -> Energy Saver
2. Go to the Battery settings and uncheck Enable Power Nap while on battery power
3. Go to Bluetooth, click the Advanced... button
4. Uncheck Allow Bluetooth devices to wake this computer
5. If you are using an external mouse and keyboard this could screw them up but I don't use bluetooth devices
6. Go to Notifications, select Do Not Disturb
7. Check on When display is sleeping and When the screen is locked

With these changes, you should find your laptop is not draining power when the lid is close and on battery. If you are still seeing the battery drain, you might have to reset the System Management Control (SMC). To do this:

1. Shut down your laptop
2. Once shutdown, press Shift-Control-Option on the left side of the keyboard
3. While still holding down the 3 keys, press the power button and hold it for around 5 seconds
1. The Apple logo will appear then disappear while you do this
2. If the Apple logo appears a second time, you held the keys for too long
3. Go back to step 1 and try again
4. Release all the keys before the logo appears the second time
5. Wait a second then press the power button to turn on the computer
6. The SMC should now be reset

Clearing space on a macOS hard drive

As a tester I'm often hoarding articles, data, VMs and tools on my hard drive.

The hard drives a decade ago where IDE or SATA hard drives. It is easy to find 1 terabyte (1000 gigabyte) hard drives. So I tend to put 1 or 2 terabyte hard drives in my computer.

Today you tend to find computers using SSD hard drives. When they first came out all you could find was 128 gigabyte or 256 gigabyte hard drives. Getting 128 gigabytes was normal because the 256 gigabyte hard drive was often a few hundred dollars more.

As time went on, the cost of SSD drives came down. Now it is common to find 256 gigabyte hard drives and it is a few hundred dollars more for a 512 gigabyte hard drive. Still 256 gigabyte is a LOT smaller than a 1 terabyte hard drive.

Large data sets and VMs for testing different operating systems and browsers would often take up a LOT of space. Add to that all the tools I typically use (development, automation, communication, productivity, etc.) and I quickly find a 256 gigabyte hard drive full.

Recently, I kept finding I would have to backup something on my hard drive and delete it to make room for something on a new project. I kept trying to figure out how to free up more space on my macOS hard drive. I have only a 250 gigabyte flash drive in my laptop and it always seemed to be at 230 gigabytes used.

Recently I read about how backing up your hard drive using Time Machine would create these local backups on your hard drive. The article talked about where the folders with the local backup were and how you could just delete the folder and it would free up a lot of room.

With the latest version of macOS (High Sierra 10.13), it uses APFS. This new file system has a number of advantages around speed, encryption, reliability, etc.. However, new file systems often mean just deleting things from the Terminal or using older hard drive clean up applications, could result in the file system getting a little messed up.

So rather than just delete local Time Machine backups, I would recommend using the tools which come with the operating system. In this case the tool is called tmutil.

If you go to a Terminal and enter tmutil it will give me a list of things it can do. One of those things is listing the local backups. To see them you can use:

tmutil listlocalsnapshots /

This will list the local snapshots on the root hard drive. From the output, you can then delete one or more of the local backups. I deleted all the local backups and found my hard drive went from 230 gigabytes used down to 117 gigabytes used. Essentially, the local backups were taking 113 gigabytes (almost half my hard drive).

To delete a local backup, I'd use:

tmutil deletelocalsnapshots <snapshot_date> 

where <snapshot_date> would be the date of the snapshot. Basically, the output of listlocalsnapshots would be:

com.apple.TimeMachine.<snapshot_date>

where <snapshot_date> is something like 2018-01-14.

Using Charles from the command line

If you are testing network traffic you are probably familiar with Fiddler. Fiddler is a nice, easy to use tool for monitoring network traffic.

It works very easy. You start up Fiddler and it configures your Internet Settings. Now when you start up a web browser, it automatically routes traffic through Fiddler. As you hit web pages on the browser, the HTTP requests and response show up in Fiddler. It is very easy to read and understand what is going on.

If you are using a macOS computer you will be sad to learn that Fiddler does not exist for macOS. It is a Windows only product. If you check for free options to do the same thing you find Wireshark (formerly Ethereal). But Wireshark's configuration and output assumes you have knowledge of TCP, HTTP, Sockets, packets, etc. You can get the information that you need but it is not as easy as Fiddler.

Additionally, to play back a request with some modifications is a lot harder with Wireshark than with Fiddler.

So what do you do? Charles Proxy. Unfortunately, it is not free but at $50 it is a good investment. If you are working at a company with many people needing it, there are discounts available as well.

Now if you get Charles you will find it automatically starts up and changes the Network Settings on your macOS. So all the browsers and anything which uses Network Settings, will automatically go through Charles. 

What about command line? For example, I have a Docker script which creates a container, deploys a web service and waits for someone to hit it. What if I'm creating automation using Python, Java, bash script, etc.? These do not use the macOS Network Settings. So you will see nothing in Charles.

The solution is to add the necessary information to the shell before you launch your test scripts.

The way Charles works is rather simple. If my machine is using 192.168.0.4 and I want to hit www.google.com (209.52.144.114) it might following the following route:

• 192.168.0.4
• 64.114.101.7
• 209.121.102.146
• 209.52.144.114

The way Charles works is creating a MITM (Man-In-The-Middle). So if I want Charles to be able to observe the traffic the route might be:

• 192.168.0.4
• CharlesProxy
• 64.114.101.7
• 209.121.102.146
• 209.52.144.114

The way it does this is by creating proxy settings in Network Settings. To create proxy settings on the command line you need to set certain variables. For HTTP traffic and HTTPS traffic Charles tells macOS to set it to use IP address 127.0.0.1 and port 8888.

For the command line you want to use:

export http_proxy="http://127.0.0.1:8888"
export https_proxy="http://127.0.0.1:8888"

Additionally, Charles tells the macOS to bypass certain addresses. What I do is go to System Preferences, select Network, select the Advanced... button, to to the Proxies tab.

On this page, assuming you are running Charles, you will see a bunch of addresses in the Bypass proxy settings box. Select all of them, copy them into the clipboard, go back to the command line and enter:

export no_proxy="<paste>"

With these three settings, anything you run from the command line will go through Charles.

However, if you close the shell you lose all the settings. If you want to keep the settings you can add them to your ~/.bash_profile text file. Every time you open a shell it will add the proxy information to the shell. HOWEVER, you don't want this if you are not using Charles. To disable this you need to enter:

unset http_proxy
unset https_proxy
unset no_proxy

So what I do is add the following to my ~/.bash_profile text file:

# Charles shortcuts
function charles_on {
        export http_proxy="http://127.0.0.1:8888"
        export https_proxy="http://127.0.0.1:8888"
        export no_proxy="127.0.0.1:6258, 127.0.0.1:6263, 127.0.0.1:10191, 127.0.0.1:14821, 127.0.0.1:24861, 127.0.0.1:25007, 127.0.0.1:38151, 127.0.0.1:46360, 127.0.0.1:49801, 127.0.0.1:55730, 127.0.0.1:59483"
}
function charles_off {
        unset http_proxy
        unset https_proxy
        unset no_proxy
}

By adding this to my ~/.bash_profile text file I can use:

charles_on

to enable Charles. And I can use:

charles_off

to disable Charles. Whenever Charles is not running I MUST disable Charles on the command line.

DIRSTACK

I was recently poking around on my Terminal (Mac OS X) and I noticed one of the environment variables was:

DIRSTACK=()

So I checked the man page for the bash shell to see what I could find about it:

man bash

Reading the man page I find DIRSTACK is an array relating to popd, pushd and dirs. Rather than using cd to change to a directory I can use pushd. For example:

pushd ~/Downloads

This will change directory to ~/Downloads plus it will add the directory to the DIRSTACK array. I can add some more to the DIRSTACK using:

pushd ~/Documents
pushd /Volumes

Now if I issue a dirs I will see:

/Volumes ~/Documents ~/Downloads

If you search for popd, pushd and dirs on the bash man page you will find all the settings for these builtin commands:

dirs [-clpv] [+n] [-n]

+n display the nth entry from the left, e.g. +2 will display the entry in position 2, this is zero-indexed
-n displays the nth entry from the right, just like the +n this is zero-indexed, e.g. -0 is the first entry
-c clears the DIRSTACK
-l displays a longer list, e.g. ~ gets expanded to the full directory name /Users/darrell
-p display one entry per line
-v display one entry per line with a number at the start of each line

You might thing the -v option is just line numbers but they are more than that. The numbers are directly related to the -n and +n option. Additionally, I can refer to specific entries in the list using ~n. For example, if the dirs -v displays:

 0  ~/Public
 1  ~/Downloads
 2  ~/Documents
 3  ~

then ls -l ~2 will be the same as ls -l ~/Documents. I can also use the tilde notation for popping elements off the stack as well. The next command, popd, has the following format:

popd [-n] [+n] [-n]

-n is literally -n, when you normally popd it will change to the directory you pop, -n will suppress this

+n removes n entries from the left, e.g. +2 will remove from third element from left (zero-indexed)

-n removes n entries from the right, e.g. -1 will pick the second element from right

The pushd commands looks similar:

pushd [-n] [dir]
pushd [-n] [+n] [-n]

-n is literally -n, and like popd it adds to the stack but does not cd to the new directory.

[dir] will push [dir] on the DIRSTACK then cd [dir]

+n will rotate the stack so the nth directory from the left is at the top

-n will roate the stack so the nth directory from the right is at the top

.

Update to using WebDriver / Selenum 2.0 with Safari

Around this time last year I wrote an article about using Safari with WebDriver. You can find this article at http://darrellgrainger.blogspot.com/2011/02/using-selenium-20-with-webdriver-and.html.

It has been a year and there is still no SafariDriver which extends WebDriver. Unfortunately, Safari 5.0 and 5.1 have hit the market and the Selenium 1.0 "*safari" driver is no longer working. As the Selenium look at dropping support for older web browsers they are also looking into supporting things like Safari 5.x.

Ideally, I would like to see:

    WebDriver driver = new SafariDriver();


or

    URL hub = new URL("http://localhost:4444/wd/hub");
    DesiredCapabilities dc = DesiredCapabilities.safari();
    WebDriver driver = new RemoteWebDriver(hub, dc);


Until this happens, we'll have to rely on using Selenium 1.0 to create an instance of Safari. But if "*safari" is no longer working and there are no plans to update it, what do we do? The solution appears to be use "*safariproxy" instead.

Looking at the code for SeleneseCommandExecutor() there are two constructors. The first takes as input a Selenium instance. The second takes as input a CommandProcessor. The second constructor actually calls the first constructor. So either method will work. So here it is:

    String baseURL = "http://www.google.com";
    Selenium sel = new DefaultSelenium("localhost", 4444, "*safariproxy", baseURL);
    CommandExecutor executor = new SeleneseCommandExecutor(sel);
    DesiredCapabilities dc = new DesiredCapabilities();
    WebDriver browser = new RemoteWebDriver(executor, dc);


or

    String baseURL = "http://www.google.com";
    CommandProcessor cp = new HttpCommandProcessor("localhost", 4444, "*safariproxy", baseURL);
    CommandExecutor executor = new SeleneseCommandExecutor(sel);
    DesiredCapabilities dc = new DesiredCapabilities();
    WebDriver browser = new RemoteWebDriver(executor, dc);


Additionally, if the browser cannot be found, i.e. it is not in the default locations, then you can specific the path to the Safari executable using:


    "*safariproxy /Application/Safari.app/Contents/MacOS/Safari"


This code is untested but hopefully better then the previous example.

Happy automating.

.

Random crashes

New application under test is a web based application. We have Windows and Mac OS X customers. All our Mac OS X customers are using Safari 5.0 or 5.1. Most our Windows customers are using Internet Explorer. We use the meta tag to force Internet Explorer 8 compatibility if you are using Internet Explorer 9. A handful of our Windows customers are using Firefox (whatever is latest).

So I need to create an automated test suite for:

• Safari 5.0 on Mac OS X 10.6 (Snow Leopard) 32-bit OS
• Safari 5.1 on Mac OS X 10.7 (Lion) 64-bit OS
• Internet Explorer 8 on Windows XP 32-bit OS
• Internet Explorer 9 on Windows 7 64-bit OS
• Firefox 8 on Windows 7 64-bit OS

With this combination I feel I have adequately covered the combinations our customers will care about. If time allowed and everything was written with one automation technology, there is no reason I couldn't run it on other combinations. For example Internet Explorer 9 on Windows XP 32-bit OS or Firefox 7 on Windows XP 64-bit OS.

The assumption is that the above 5 combinations will find most the defects.

My problem is that Safari on Mac OS X is as equally important as Internet Explorer on Windows.

After poking around the web I found Selenium 2.0 (WebDriver) supports all the platforms, sort of. There is no Safari driver for WebDriver but I have a snippet here for using Selenium 1.0 to create a Safari instance, from that I can create a SeleneseCommandExecutor and with the Executor I can create a RemoteWebDriver. A bit hacky but it works.

So I started creating my test automation. I got up to 39 test cases when I decided to run them as a single suite. Ran them on Firefox 8 and everything worked perfectly. Ran them on Internet Explorer. Around 25 test cases in and Internet Explorer pops up a dialog telling me something went wrong with Internet Explorer. I can debug it (because I have Visual Studio installed) or just close it. Regardless, my automation is dead. If my automation dies after the 25th test case and I build it up to 45,000 that would be totally useless. I wouldn't be completely happy with it but if it at least recovered and continued on to the next test case it would be better. But it does not.

So the hunt goes on. I have found a commercial product which claims to work for all the above platforms and more. I'll split up time between evaluating the commercial product and debugging Selenium. Hopefully by  the end of the week I'll have a decision one way or the other.

Stay tuned...

Using Oracle SQL Developer with SQL Server

If you are looking for a SQL Client for both Oracle DB and Microsoft SQL Server you can use the Oracle SQL Developer with the jTDS plugin.

Get Oracle SQL Developer from the Oracle web site. Just use your favourite search engine to search for 'Oracle SQL Developer'. These instructions have been verified to work with SQL Developer 3.0.

Once you have installed Oracle SQL Developer, run it and go to the Help menu. There you should find the Check For Updates... option. There will be a dialog with Third Party SQL Developer Extensions as an option. Make sure this is selected.

On the next page there will be an option to select the jTDS extension. Select this for SQL Server and Sybase support.

Once you accept the license agreement and restart SQL Developer, you will now have a tab for SQL Server when creating a new connection.

Using the features of Selenium 1.0 with Selenium 2.0 code

I recently posted about doing a screen capture on exception. This got me thinking about how I can maximize the browser window before I do the screen capture. In Selenium 1.x I would use:

Selenium sel = new DefaultSelenium("localhost", 4444, "*firefox", "http://www.google.com");
    sel.windowMaximize();

But how do I do this using WebDriver? The solution is to use a WebDriver instance to create a WebDriverBackedSelenium instance. Here is the code:

WebDriver driver = new FirefoxDriver();
    Selenium sel = new WebDriverBackedSelenium(driver, "http://www.google.com");
    sel.windowMaximize();

and it is that simple. If I want to use Selenium 2.0 features, I access things via the driver variable. If I want to use Selenium 1.0 features, I access things via the sel variable.

Quite simple.

My new Mac OS X

It has been a short while since I posted. For a while there I was without a computer. My Windows box had decided I needed an updated video driver. Last time it did this I was reduced to 320x200 16 colour graphics. This time it just totally puked my machine. I tried booting in safe mode; everything appeared to start okay but after all the drivers loaded, it switched to the Windows XP splash screen (just before the login dialog) and hung there. Booted the thing with a Dr.DOS CD I created (just for such an emergency). The registry had been corrupted.

Decided now was the time to buy a new computer. Had been looking at the Mac for a while. It is really a UNIX box with an Apple GUI. The thing makes as much noise turned on as it does turned off. The Windows box sounds like a jet taking off in the distance. I connected my ethernet cable, powered it up and it immediately found the DSL modem, knew what kind of protocol it used, asked me two questions (username and password for my ISP) and bang I was on the net.

The really cool thing is I don't have to install something like Cygwin to do Bourne shell scripting. The REALLY cool thing is what I can do without much knowledge of UNIX command line. There is a utility on the thing called 'Network Utility'. You go to Spotlight (Command-Spacebar) and enter 'network'. The first thing it finds is the Network Utility.

A lot of the stuff on this is pretty easy to use from the command line, e.g. netstat or ping. But there is a feature labeled 'Port Scan'. This thing is like nmap but a lot easier to use. You give it the name or IP address of a machine and it will probe the machines ports. It will see if something is at each port and figure out what that something is. You might thing, it sees something at port 80 and assumes it is http but I've put an application server at a different port and it found it.

I'm scanning a site now and seeing that they have turned off telnet (I think it is a Linux machine so telnetd is off by default) but they have sshd running on the standard port 22. It is a web site so no surprise there is an httpd running on port 80.

It is not clear where this thing has a passive mode (or if it is always in passive mode). I'm going to guess it is not and the host I'm probing sees me peeking at all the ports. If you want to do some stealth probing, this is probably not the tool for you. You'd have to go for command line.

Makes me wonder, why would you want this? Your average joe user can find you are running iiop on port 3528. This will mean nothing to them. Most the people who know about this sort of stuff, know command line utilities like nmap. Still, kind of fun to play with.

The typical UNIX or Linux start does not apply to the Mac. After the initial rc.d everything is started by a launchd application. Additionally, things you run from the GUI are run very differently then on say SuSE Linux. A GUI application has the elaborate structure and there is a special program that runs the application. For example, if the GUI indicates Foo is an application in /homes/darrell then to run it from the command line I'd have to issue: "/System/Library/Frameworks/Carbon.framework/Versions/A/Support/LaunchCFMApp /homes/darrell/Foo.app/Contents/MacOS/Foo"

The Foo.app is what you see as Foo from the GUI. You don't see the Contents folder or anything else inside the Foo.app directory.

System V versus BSD UNIX

Now a days you will most likely be exposed to AT&T System V UNIX. Years ago there were dozens of different 'versions' of UNIX. At some point things started to converge to either AT&T System V (pronounced A T and T System Five) or BSD UNIX.

If you go for a job interview they might ask you things like, "How do you list all processes?" If you are using System V then the answer is "ps -ef" but it is "ps -aux" if you are using BSD UNIX.

Additionally, a lot of the System V installations have the BSD implementations of commands as well. The idea is that I might have a script that assumes BSD format commands. If I was to parse the output of ps to do a mass kill of all processes and its children then the format of the ps command matters.

If the script was written assuming the BSD version of ps and my computer was configured for System V, the script will fail. The solution, the default path might be /usr/bin but the BSD version of the commands are stored in /usr/ucb. So I could just edit the script so it finds the /usr/ucb/ps command rather than the default /usr/bin/ps.

Finally, if you are using MacOS X you are using a GUI front end to an implementation of UNIX. The UNIX is BSD UNIX. So things like "ps -ef" will not work.

So if you want to impress an interviewer who asks how to list all the processes on a UNIX machine, the answer is "If your computer is configured for System V UNIX then ps -ef will do the trick, but if /usr/ucb exists in the path before /usr/bin then ps -aux will do the trick. Additionally, if you are using MacOS X, which is BSD UNIX, then the default path will use ps -aux."

There are some obvious wrinkles to this. I'm assuming the default set up and configuration for the system. You'd impress them more if you asked which verison of UNIX and then noted, "if they are using the common configuration..."

If you want to read more of the details behind the history of UNIX, give this web site a try. Additionally, this site talks about creation of different versions of UNIX. Click on the picture at the top to see how nuts it got.