Difference between QA and Dev

Difference Between Quality Analyst and Developer

In 1998, I was testing software development tools. I was testing an IDE, compiler, assembler, micro-kernel, scripting language, etc. It started off with me testing the work of a few dozen developers. It was my first QA job.

In 2001, one of the developers was talking about this new way of developing software. Previously we had Waterfall. This morphed into V-Model and later Spiral SDLC. QA was silo'd from Development and there were a lot of processes and procedures. Developing an application took months and sometimes years.

The new way of developing software was called Agile and included things like Unit Testing, eXtreme Programming, etc..

QA were the gatekeepers. We were the last line of defense against bad code. In some companies it was almost adversarial. Some developers looked at QA as the department that criticized their work. They were blocking them from getting their work done. 

I remember a time when a develop refused to talk to me because they honestly thought I was calling them a liar and just wanted to make things difficult for them. I had to call their team lead. I explained the defect to him. I showed him. He realized the junior developer fixed A defect, just not the one I reported. They worked closely with him and had a great deal of respect for him. I was just someone from a different department on a different floor who they really didn't know socially. In other words, there was a little bit of hostility between QA and Devs.

Shortly after Agile started to take off, someone posted a blog explaining why QA were no longer necessary. With unit testing and automated testing, QA was no longer necessary. A few QA people in my home town were honestly concerned about the future of QA in software development.

But shortly after the blog post, a number of high level software developers, some of which are signatories on the Agile Manifesto, started point out that QA were still necessary. They definitely didn't want management to get the idea they could fire all the QA staff and make the developers responsible for all testing.

I started wonder what does QA bring to the table? Why is having QA or software testers necessary? This questions was in the back of my mind as I learned how my role changed in software development.

I started working more closely with the developers. At first I would test the software a sprint later than the developer. For example, feature #1 is implemented in sprint 1 then I would test it in sprint 2. But if I found defects, someone would have to fix it in sprint 3 (can't disrupt the current sprint). After a month, developers had to remember what feature #1 was and fix the defect. Not efficient. 

So we started having developers and QA work together in the same sprint. The time between implementation and testing got smaller and smaller. I would even talk to developers as them worked on a feature.

Then I started to noticed a pattern (that is what good QA do). I'd ask, "What happens of the customer does XXX? And the developers would usually ask, "Why would anyone do that?" Essentially, many developers were brilliant technically, but not great at understand people who weren't technically smart. To this day I still test some software with the question, "What would my eighty year old mother do?"

A number of times I would paint a negative scenario and developers would counter with, "That would never happen." Only to find out later than the negative scenario happened more often than the positive scenario.

So I got thinking, what would be a good way to explain the difference between a QA and a Developer. I came up with:

Developers tend to make sure the code does what it is supposed to do; QAs tend to make sure code DOESN'T do what it's NOT supposed to do. -- Darrell Grainger

I still use this quote today. 

.

K.I.S.S.

 Keep It Simple Silly

I started working on a project that had been under development for 6 months. There was unit testing in the code base but there was also a framework they had created which was independent of the production code base.

The idea of the independent framework was that the subject matter experts (SMEs) would generate data to test the application. They would think about all the different scenarios they would encounter in their job, before the software existed.

The independent test framework was great at confirming things were working. However when it failed it was very difficult to debug. The inability to maintaining a framework is the number one reason automation fails. So when things started failing, someone had to step through the framework, figure out the input data and reproduce the scenario in the production code. This way the developer could step through the failing scenario to understand what was going wrong. Without using the integrated development environment (IDE), it was impossible for the developer to figure out what was going wrong.

The scenario the developer created in the IDE was to create the data as setup variables then run the data through the application in debug mode. There was a huge amount of data that was very complex and difficult to set up. Occasionally the setup was wrong because the developer was not a SME. So there was a lot of back and forth with the SME to track down what was really going on.

My thought was, we have the data from the SME in JSON form and reference data was in spreadsheets. Rather than pulling all the data out of the JSON and into the unit testing framework, couldn't we just read the JSON and spreadsheets into a unit test?

The short answer was yes. It took a little while to create the new framework but once we did this for one scenario, doing it for subsequent scenarios was just dropping the SME data into the test section of the application. If it failed, the developer had the test data with the application code in the IDE and debugging it was a lot easier.

Bottom line, keep it simple, silly.

What is QA to me

Before I did testing and QA I was a software developer and a lecturer at university.

I actually wanted to teach high school (mathematics, computer science, statistics, etc.). I applied to Teacher's College but due to a bureaucratic mistake, I was told I had to apply again a year later.

I could have fought it but I figured I'd just find a job for a year and try again a year later.

Funny enough, I found a job testing software development tools for a company which created computer chips. Testing an IDE, compiler, assembler, micro-kernel, etc. required hardware and software development knowledge. I had both and thought working in 'QA' would be ideal as it wasn't a 'real' job.

I took that job in the Fall of 1998 with the full intention of going back to school in the Fall of 1999. It is now May 2021 and I'm still doing QA and testing software. I just fell in love with it.

Automated testing wasn't really a think. Many people, myself included, where still doing Waterfall software development.

Over the years I moved from testing desktop applications to testing web applications. Initially, when software developers starting doing Agile software development, QA or testing was still an after thought. One signatory of the Agile Manifesto actually published an article about how QA was dead and Agile software developers didn't need software testers anymore. He was wrong and others pointed it out.

But it still got me wondering about the future of QA and software testing. Software development was evolving. So without skipping a beat, I evolved too.

With manual testing, I'd get a story from developers. I'd manually test it. Then I'd get a second story. I'd manually test it. But I'd also re-test story number one. By the time the developers had written story 500, I was unable to do regression testing on stories 1 through 499. First thought was to order the stories in terms of importance. Test the most important stories first. Maybe I could test 300 of the stories before the Project Manager ask if the product was ready to ship. I was nervous to say yes. He shipped it anyway.

I'd keep testing after the product ship. If I found anything wrong, we'd create a "service patch".

This didn't feel ideal. So I started automating UI testing. Tools like Selenium and HP Quality Center started appearing. Being a former software developer, I like the programmability of Selenium. I could use the tools and best practices of software development to develop Selenium test suites.

But I remembered an article from IBM about how maintaining code was 4/5th of the work. I quickly realized that maintaining my test automation would become a huge part of the development. If I couldn't maintain it, it would fail.

Try as I might, maintaining UI automation which tested everything wasn't maintainable. It tested a lot but it was still lacking.

Along comes Mike Cohn, Mountain Goat Software, and the test pyramid. It made sense to me. I started learning about flipping the pyramid or as Alister B. Scott put it, the test ice cream cone. Rather than doing a lot of testing early, I was testing everything at the end. I almost saw it as the testing lollipop.

So I started encouraging developers to test as much as they could at the unit level. Kent Beck was promoting JUnit, an xUnit test framework. This seemed ideal. I could take one of my tests at the top of the pyramid and see that it had 10 reasons it might be failing. If I found a defect, could the developers write a unit test that would detect the defect? If yes, then I no longer had to maintain my UI automation for that potential failure. I started to realize that they could have unit tests that caught maybe 6 of the 10 possible failures.

Then we started learning about mocking, test doubles, etc.. Gerard Meszaros wrote a great book titled XUnit Test Patterns. You can read more about it at http://xunitpatterns.com/. 

As I looked more and more into this I started finding about integration testing, contract testing, etc.. All these other lower level tests that were easier for developers to write and maintain.

I started realizing that UI testing an application wasn't really valuable at all. The test pyramid showed me that UI testing, at the top of the pyramid, was the smallest portion. Trying to convince people I needed to write more UI tests was self-defecting. If I REALLY wanted to assure quality, I needed to enable the team to write better software.

Looking at patterns for software failure and thinking about how could this have been caught and fixed early was now a priority for me. Writing automation was starting to shift from my job to the job of the software developers. Was that Agile Manifesto signatory right? Was I no longer needed?

He wasn't. The software developers still have a different mindset. I still notice to this day that a software developer tends to make sure the application does what it is supposed to do. But a QA tends to make sure the application doesn't do what it isn't supposed to do. It is a subtle difference but an important one.

Are their software developers who know how to think like a software developer AND think like a QA? Absolutely! Does my working with a developer help them think like a QA and potentially put me out of a job? Well, kind of. When I pair with a developer a little bit of me rubs off on them. But there are so many other developers I can help. I've been doing this for almost 20 years now and I still haven't found people I can't help.

I still write automation but I also try to understand where problems might arise in the application. Enterprise architecture can get really complex. How can we reduce that complexity? How can I find and eliminate defects? It seems to be a never-ending task. I keep trying to put myself out of a job but so far I've just made myself and my team better.

Bottom line, stop doing Quality Control (you can't test quality into software) and start doing Quality Assurance.

Using git show to find breaking changes

 First, sorry for not being around much. I've been lucky enough to be busy with work.

We had all the unit tests passing, smoke and integration tests passing but when we put the full enterprise environment to the test things which were failing.

The technology we are using can't be run locally. So stepping through the code and debugging the issue wasn't possible. We had to go old school and add print statements to the code to see what was happening.

First, you should be using something like log4j to control the log levels. This way you can add debug statements to the code but be able to turn them off when you deploy to a production environment.

But how do you determine how much is enough log statements? How do you determine what is too little?

I looked at our build. The end to end tests were passing on Monday. They only run nightly. In the course of the day there were 11 commits to the git repository. First, was the end to end test failing because of a commit in the code? Or was there a change in the enterprise environment?

My first step was to deploy the code from the last time it was working.

Fortunately, most build pipelines will let you build and deploy based on the git commit hash. So I looked up the last time the build was working for the end to end tests. The pipeline would tell me which git commit was used to build it.

So I run the pipeline and ask it to deploy that specific git commit. Then I run the current end to end tests on the deployed application. The end to end test passed. Now I have the git commit for when it was working.

I also have the git commit for the current master commit. This commit is failing today.

Let's say that the commit which was working was 0a803c1 and the latest commit was ea3dbb9. There are 11 commits. So how do I figure out what commit broke it?

I can see all the files which were changed using the git show command. The exact syntax is:

git show --pretty="" --name-only 0a803c1..ea3dbb9

 This will provide a list of all the files which have changed. If a file was changed in different commits, multiple times, it will show the file more than once in the list. If all you care about is a short list of files which changed I'd actually use:

git show --pretty="" --name-only 0a903c1..ea3dbb9 | sort | uniq

This will get rid of duplication.

Once I know which files have been affected, I can start adding debug statements to these files and not bother focusing on any of the other files in the repository.

Hello old friend

It has been a while since I wrote to my blog. What happened? Life.

But why am I back here? The macOS Catalina 10.15.xx.

I'm still working as a consultant testing software my colleagues have written. Recently, I got a new laptop for work and it came with the Catalina OS.

Over the last year I have noticed that Apple Macintosh has become the new Microsoft Windows. Whenever I get an update or install something new and things behave badly (Wifi won't connect) I have learned to reboot my computer. Even though the update or installation didn't say it needed to reboot my computer, I'm finding rebooting my computer actually fixes things from time to time.

Until recently. Normally, to save time, I never shutdown or log out of my laptop unless I feel I REALLY need to. So at the end of the day, I close my laptop and it goes to sleep. I have it plugged in all day. So when I close it, it is at 100% battery power. The next morning, I open my laptop and it is often at 100% power. Sometimes it drops to 99% power.

Since I upgraded to Catalina this is no longer true. I closed my laptop and didn't use it over the weekend. Monday morning I open my laptop and it is at 7% battery remaining. What the heck!?!!

So after a little digging I find that the default settings for a lot of things leave background processes running, even when it should be in sleep mode. Here is what I did to stop my laptop from dying when the lid is closed:

1. Go to System Preferences -> Energy Saver
2. Go to the Battery settings and uncheck Enable Power Nap while on battery power
3. Go to Bluetooth, click the Advanced... button
4. Uncheck Allow Bluetooth devices to wake this computer
5. If you are using an external mouse and keyboard this could screw them up but I don't use bluetooth devices
6. Go to Notifications, select Do Not Disturb
7. Check on When display is sleeping and When the screen is locked

With these changes, you should find your laptop is not draining power when the lid is close and on battery. If you are still seeing the battery drain, you might have to reset the System Management Control (SMC). To do this:

1. Shut down your laptop
2. Once shutdown, press Shift-Control-Option on the left side of the keyboard
3. While still holding down the 3 keys, press the power button and hold it for around 5 seconds
1. The Apple logo will appear then disappear while you do this
2. If the Apple logo appears a second time, you held the keys for too long
3. Go back to step 1 and try again
4. Release all the keys before the logo appears the second time
5. Wait a second then press the power button to turn on the computer
6. The SMC should now be reset

Clearing space on a macOS hard drive

As a tester I'm often hoarding articles, data, VMs and tools on my hard drive.

The hard drives a decade ago where IDE or SATA hard drives. It is easy to find 1 terabyte (1000 gigabyte) hard drives. So I tend to put 1 or 2 terabyte hard drives in my computer.

Today you tend to find computers using SSD hard drives. When they first came out all you could find was 128 gigabyte or 256 gigabyte hard drives. Getting 128 gigabytes was normal because the 256 gigabyte hard drive was often a few hundred dollars more.

As time went on, the cost of SSD drives came down. Now it is common to find 256 gigabyte hard drives and it is a few hundred dollars more for a 512 gigabyte hard drive. Still 256 gigabyte is a LOT smaller than a 1 terabyte hard drive.

Large data sets and VMs for testing different operating systems and browsers would often take up a LOT of space. Add to that all the tools I typically use (development, automation, communication, productivity, etc.) and I quickly find a 256 gigabyte hard drive full.

Recently, I kept finding I would have to backup something on my hard drive and delete it to make room for something on a new project. I kept trying to figure out how to free up more space on my macOS hard drive. I have only a 250 gigabyte flash drive in my laptop and it always seemed to be at 230 gigabytes used.

Recently I read about how backing up your hard drive using Time Machine would create these local backups on your hard drive. The article talked about where the folders with the local backup were and how you could just delete the folder and it would free up a lot of room.

With the latest version of macOS (High Sierra 10.13), it uses APFS. This new file system has a number of advantages around speed, encryption, reliability, etc.. However, new file systems often mean just deleting things from the Terminal or using older hard drive clean up applications, could result in the file system getting a little messed up.

So rather than just delete local Time Machine backups, I would recommend using the tools which come with the operating system. In this case the tool is called tmutil.

If you go to a Terminal and enter tmutil it will give me a list of things it can do. One of those things is listing the local backups. To see them you can use:

tmutil listlocalsnapshots /

This will list the local snapshots on the root hard drive. From the output, you can then delete one or more of the local backups. I deleted all the local backups and found my hard drive went from 230 gigabytes used down to 117 gigabytes used. Essentially, the local backups were taking 113 gigabytes (almost half my hard drive).

To delete a local backup, I'd use:

tmutil deletelocalsnapshots <snapshot_date> 

where <snapshot_date> would be the date of the snapshot. Basically, the output of listlocalsnapshots would be:

com.apple.TimeMachine.<snapshot_date>

where <snapshot_date> is something like 2018-01-14.

How to use secrets

From time to time you heard about how someone's site has been compromised and customer's personal information has been leaked. For example, Target Got Hacked Hard.

What can you do? Good passwords is a start.

First is understanding how do we choose a password.

If the password can be 10 characters with uppercase, lowercase, digits and symbols it could be (26 + 26 + 10 + 32) to the power of 10 or 5.386151140948997e19 different possible passwords. Even if you could attempt thousands of password attempts a second, it would take you eons to be able to crack this. Most systems will also throttle how many attempts to you can make per minute.

A good hacker won't attempt all possible combinations. They would use a dictionary to restrict the attempts to known password combinations. They might be able to reduce the number of different possible passwords to something they could attempt within a year. So systems will often request you change your password every 90 days.

Additionally, if the system allows for more characters (password length of 20 or 30 characters) just adding a few characters will make it exponentially harder to crack.

So don't use known patterns (dictionary attack), make your password as long as possible, change it every few months. The change should also be significant.

Now maybe you have to type the password into a mobile device and a really long, complicated password is going to be a bother. So you end up shortening it. Also, if you password is something like "CBVcP3Zj/i}3mK,xUL7U", you are never going to remember that. You will be tempted to make it shorter, easy to remember and you won't want to change it too often. So something like "!Passw0rd!" seems like a good password. It has uppercase, lowercase, digit, symbols, it is easy to remember, it is 10 characters long. However, it is easy to guess. A hacker will DEFINITELY have this in their dictionary.

What about changing the password? If your password was "My!S3cr3t!Passw0rd!" and I changed it to "My!S3cr3t!Passw0rd!01", next I used "My!S3cr3t!Passw0rd!02", someone will guess this with a dictionary attack.

Additionally, it used to be that a 6 character password wasn't crackable by brute force. As computers got faster we needed 8 character passwords. Today it seems that companies are recommending 12 character passwords. So if you want to be safe, you should use at least 20 character passwords.

So REALLY the password of "CBVcP3Zj/i}3mK,xUL7U" is what we want to use. That said, I've known security staff walk around the office looking for post-it notes. People will write their password on a post-it note and stick the note to a monitor. A lot of time, systems are hacked because of an insider getting a password which was written down. When I first got into computers, a high school teacher wrote the password to the system on a sheet in his desk. I know this because students would see him looking in his desk then logging in the system. One of the students stole the password from his desk.

What about saving it on a file in the computer. That works even better for some things. I need to log into a website. I open my text file, copy the password, paste it into the website. What happens if someone gets a copy of my text file. Then they have all my passwords.

I can put them in a spreadsheet with a password or a zip file with a password. Is the password to the spreadsheet going to be "CBVcP3Zj/i}3mK,xUL7U"? And I don't write it down? Probably not.

This brings us to dynamic and static storage. If my password is in a variable or the clipboard, I can turn off my computer and it is gone. If it is stored in a text file, someone can make a copy of the file and take days trying to crack it. Hackers will sometimes get encrypted database files then spend weeks trying to crack it. This is where they get common passwords for their dictionary attacks.

What if your files are automatically backed up to iCloud or Google? Then someone might be able to intercept them and take weeks trying to crack them.

Also, how do you think I came up with "CBVcP3Zj/i}3mK,xUL7U"? The answer is a password vault program. There are a few. I use 1Password (because it works on all the devices I use). There are free options like LastPass, KeePass, Dashlane. Have a look at List of password managers for these and other options. Now you just have to pick one good, strong password to secure your vault.

Additionally, if you don't backup your vault to a cloud or let people have access to it, they cannot hack it.

Another bad practice I see people doing is saving passwords in variables. This is less for your personal passwords and more for programmers accessing enterprise sites. I might have a utility which accesses the XYZ system. If variables are set, it never asks me for a password. I might have:

export XYZ_USERNAME='abssass'
export XYZ_USERNAME='2eD$g^^nJk5wHki6Lsst4Gwr'

Now if I run the utility it will see these variable are set and use them to log in. Or I might  have something like: "http://$XYZ_USERNAME:$XYZ_PASSWORD@hostname" and so long as the variables are set, it will automatically log me into the website.

But now we are back to saving passwords in a text file. Things like export statements are saved in ~/.bash_profile on my computer.

For the team they might save the passwords in a team password manager. For example, vault by Hashicorp. To get my secrets I might execute something like:

vault read /secrets/team/storage/xyz

This might return something like:

Key	Value
xyz_username	abssass
xyz_password	2eD$g^^nJk5wHki6Lsst4Gwr

So now I can use my UNIX/Linux skills to parse the username and password out and save them in a variable. For example:

export XYZ_PASSWORD=$(vault read /secrets/team/storage/xyz | grep xyz_password | awk '{print $2}')export XYZ_USERNAME=$(vault read /secrets/team/storage/xyz | grep xyz_username | awk '{print $2}')

This will read the information from the vault, parse out the appropriate information and save it to a variable. When I turn off my computer the variables are gone and the password isn't saved in a text file... not even encrypted.