K.I.S.S.

 Keep It Simple Silly

I started working on a project that had been under development for 6 months. There was unit testing in the code base but there was also a framework they had created which was independent of the production code base.

The idea of the independent framework was that the subject matter experts (SMEs) would generate data to test the application. They would think about all the different scenarios they would encounter in their job, before the software existed.

The independent test framework was great at confirming things were working. However when it failed it was very difficult to debug. The inability to maintaining a framework is the number one reason automation fails. So when things started failing, someone had to step through the framework, figure out the input data and reproduce the scenario in the production code. This way the developer could step through the failing scenario to understand what was going wrong. Without using the integrated development environment (IDE), it was impossible for the developer to figure out what was going wrong.

The scenario the developer created in the IDE was to create the data as setup variables then run the data through the application in debug mode. There was a huge amount of data that was very complex and difficult to set up. Occasionally the setup was wrong because the developer was not a SME. So there was a lot of back and forth with the SME to track down what was really going on.

My thought was, we have the data from the SME in JSON form and reference data was in spreadsheets. Rather than pulling all the data out of the JSON and into the unit testing framework, couldn't we just read the JSON and spreadsheets into a unit test?

The short answer was yes. It took a little while to create the new framework but once we did this for one scenario, doing it for subsequent scenarios was just dropping the SME data into the test section of the application. If it failed, the developer had the test data with the application code in the IDE and debugging it was a lot easier.

Bottom line, keep it simple, silly.

What is QA to me

Before I did testing and QA I was a software developer and a lecturer at university.

I actually wanted to teach high school (mathematics, computer science, statistics, etc.). I applied to Teacher's College but due to a bureaucratic mistake, I was told I had to apply again a year later.

I could have fought it but I figured I'd just find a job for a year and try again a year later.

Funny enough, I found a job testing software development tools for a company which created computer chips. Testing an IDE, compiler, assembler, micro-kernel, etc. required hardware and software development knowledge. I had both and thought working in 'QA' would be ideal as it wasn't a 'real' job.

I took that job in the Fall of 1998 with the full intention of going back to school in the Fall of 1999. It is now May 2021 and I'm still doing QA and testing software. I just fell in love with it.

Automated testing wasn't really a think. Many people, myself included, where still doing Waterfall software development.

Over the years I moved from testing desktop applications to testing web applications. Initially, when software developers starting doing Agile software development, QA or testing was still an after thought. One signatory of the Agile Manifesto actually published an article about how QA was dead and Agile software developers didn't need software testers anymore. He was wrong and others pointed it out.

But it still got me wondering about the future of QA and software testing. Software development was evolving. So without skipping a beat, I evolved too.

With manual testing, I'd get a story from developers. I'd manually test it. Then I'd get a second story. I'd manually test it. But I'd also re-test story number one. By the time the developers had written story 500, I was unable to do regression testing on stories 1 through 499. First thought was to order the stories in terms of importance. Test the most important stories first. Maybe I could test 300 of the stories before the Project Manager ask if the product was ready to ship. I was nervous to say yes. He shipped it anyway.

I'd keep testing after the product ship. If I found anything wrong, we'd create a "service patch".

This didn't feel ideal. So I started automating UI testing. Tools like Selenium and HP Quality Center started appearing. Being a former software developer, I like the programmability of Selenium. I could use the tools and best practices of software development to develop Selenium test suites.

But I remembered an article from IBM about how maintaining code was 4/5th of the work. I quickly realized that maintaining my test automation would become a huge part of the development. If I couldn't maintain it, it would fail.

Try as I might, maintaining UI automation which tested everything wasn't maintainable. It tested a lot but it was still lacking.

Along comes Mike Cohn, Mountain Goat Software, and the test pyramid. It made sense to me. I started learning about flipping the pyramid or as Alister B. Scott put it, the test ice cream cone. Rather than doing a lot of testing early, I was testing everything at the end. I almost saw it as the testing lollipop.

So I started encouraging developers to test as much as they could at the unit level. Kent Beck was promoting JUnit, an xUnit test framework. This seemed ideal. I could take one of my tests at the top of the pyramid and see that it had 10 reasons it might be failing. If I found a defect, could the developers write a unit test that would detect the defect? If yes, then I no longer had to maintain my UI automation for that potential failure. I started to realize that they could have unit tests that caught maybe 6 of the 10 possible failures.

Then we started learning about mocking, test doubles, etc.. Gerard Meszaros wrote a great book titled XUnit Test Patterns. You can read more about it at http://xunitpatterns.com/. 

As I looked more and more into this I started finding about integration testing, contract testing, etc.. All these other lower level tests that were easier for developers to write and maintain.

I started realizing that UI testing an application wasn't really valuable at all. The test pyramid showed me that UI testing, at the top of the pyramid, was the smallest portion. Trying to convince people I needed to write more UI tests was self-defecting. If I REALLY wanted to assure quality, I needed to enable the team to write better software.

Looking at patterns for software failure and thinking about how could this have been caught and fixed early was now a priority for me. Writing automation was starting to shift from my job to the job of the software developers. Was that Agile Manifesto signatory right? Was I no longer needed?

He wasn't. The software developers still have a different mindset. I still notice to this day that a software developer tends to make sure the application does what it is supposed to do. But a QA tends to make sure the application doesn't do what it isn't supposed to do. It is a subtle difference but an important one.

Are their software developers who know how to think like a software developer AND think like a QA? Absolutely! Does my working with a developer help them think like a QA and potentially put me out of a job? Well, kind of. When I pair with a developer a little bit of me rubs off on them. But there are so many other developers I can help. I've been doing this for almost 20 years now and I still haven't found people I can't help.

I still write automation but I also try to understand where problems might arise in the application. Enterprise architecture can get really complex. How can we reduce that complexity? How can I find and eliminate defects? It seems to be a never-ending task. I keep trying to put myself out of a job but so far I've just made myself and my team better.

Bottom line, stop doing Quality Control (you can't test quality into software) and start doing Quality Assurance.

Using git show to find breaking changes

 First, sorry for not being around much. I've been lucky enough to be busy with work.

We had all the unit tests passing, smoke and integration tests passing but when we put the full enterprise environment to the test things which were failing.

The technology we are using can't be run locally. So stepping through the code and debugging the issue wasn't possible. We had to go old school and add print statements to the code to see what was happening.

First, you should be using something like log4j to control the log levels. This way you can add debug statements to the code but be able to turn them off when you deploy to a production environment.

But how do you determine how much is enough log statements? How do you determine what is too little?

I looked at our build. The end to end tests were passing on Monday. They only run nightly. In the course of the day there were 11 commits to the git repository. First, was the end to end test failing because of a commit in the code? Or was there a change in the enterprise environment?

My first step was to deploy the code from the last time it was working.

Fortunately, most build pipelines will let you build and deploy based on the git commit hash. So I looked up the last time the build was working for the end to end tests. The pipeline would tell me which git commit was used to build it.

So I run the pipeline and ask it to deploy that specific git commit. Then I run the current end to end tests on the deployed application. The end to end test passed. Now I have the git commit for when it was working.

I also have the git commit for the current master commit. This commit is failing today.

Let's say that the commit which was working was 0a803c1 and the latest commit was ea3dbb9. There are 11 commits. So how do I figure out what commit broke it?

I can see all the files which were changed using the git show command. The exact syntax is:

git show --pretty="" --name-only 0a803c1..ea3dbb9

 This will provide a list of all the files which have changed. If a file was changed in different commits, multiple times, it will show the file more than once in the list. If all you care about is a short list of files which changed I'd actually use:

git show --pretty="" --name-only 0a903c1..ea3dbb9 | sort | uniq

This will get rid of duplication.

Once I know which files have been affected, I can start adding debug statements to these files and not bother focusing on any of the other files in the repository.