Technical Debt

What is technical debt? 

For the past ten years I have been hearing the expression "Technical Debt" in reference to programming. But what is technical debt?

At the same time I noticed the best project managers would have a summary chart for upper management. It broke things down into red light, yellow light or green light. The idea was that upper management have a lot of information to sift through so we needed to keep it simple.

I think the same idea applies to technical debt. You don't need a precise measurement of technical debt but you need to know if it is growing (red light), stable but still there (yellow light) or reducing (green light).

This still leaves the question, what is technical debt? If I'm running a business and my business needs a piece of machinery to finish a project. I cannot get money from the customer until I deliver the product but I need money to buy the machine to finish the project. So I get a small business loan, buy the machine, finish the project and sell it to the customer. At this point I have debt. Is the interest on the debt acceptable? If the interest is equal to the profit I made on the sale then I'll never get ahead (yellow light). If the interest is greater than the profit I made then I'll slowly go deeper and deeper into debt (red light). But if the interest is less than the profit I made, I can pay down or pay off my debt. BEFORE I get the loan, I need to know the profit will be greater than the interest on the loan. I need to know I will still turn a profit and can get out of debt.

Technical debt is similar. It MUST be a conscience decision to do something which is not good from a development point of view but which I'll be able to correct after the product is out the door. For example, I'm going to pick an inexpensive or easy to implement technology which I know will not scale. I know it will exceed the current load. I can predict, if successful, this choice will be a bottleneck in 18 months. In other words, the 'interest' on this 'loan' will eat away all my profit in 18 months. The long I wait to change to a more scalable technology the less profitable my project will be.

Worse would be to not only fail to pay off or at least pay down this technical debt but to incur more technical debt. Most people would like to have a nice home, a good car, annual trips to Europe or the Caribbean, send their kids to the best schools, etc. but would you use a credit card to pay for it all? When the bill collector comes knocking would you apply for another credit card to pay off the first credit card? Hopefully, you answered no. Even if you didn't, credit bureaus like Equifax or TransUnion would quickly make it impossible for you to get more credit.

Unfortunately, in software development there is no Equifax or TransUnion. Creating technical debt is a LOT easier than creating financial debt. This means you need to monitor your technical debt and make sure you are paying it down.

For short periods of time your technical debt might rise but the over all trend should be a reduction.

What is NOT technical debt? 

It is MORE important to understand what is not technical debt. A lot of people will chalk things up to technical debt when it is really lack of planning.

It isn't even poor planning but more a total lack of planning. Many times I have worked on projects where the project manager (or above) have made it very clear that the project must ship by a specific date. No excuses. What are you telling the programmers? You are telling them you don't care what the cost it, it has to ship by a specific date. The truth is that no one REALLY means 'money is no object'.

If you don't look at the cost of a technical compromise and just implement it, it is the equivalent of going to the local loan shark and signing a piece of paper without reading it. No successful business person would do this but I have seen Fortune 500 companies do the technical equivalent.

As a programmer, if my management tells me something has to be completed by a specific date, no matter what then they are telling me technical debt is no object. Take on whatever technical debt is necessary to get the project done on time. As a senior programmer I know they don't REALLY mean this. However, I have worked in companies where I tried to point out the technical debt is too high only to be told they don't want to hear excuses. Another programmer will swoop in and tell them he can get it done. I get demoted, the other programmer gets promoted.

Inevitably, the project ships on time and under budget. The programmer who swooped in and saved the day might even get a bonus. He might even do this again on the next release of the product. But then I noticed a pattern. Sooner or later, the programmer does a lateral shift to another department or worse, leaves the company to go work for the competition. Shortly after his departure from the project, the new developer starts trying to point out how much technical debt the previous programmer created. For example, a 50,000+ line function which uses lots of GOTOs and stack manipulation to make things work. Adding a new feature which should take 2 days to implement takes months and introduces dozens of new bugs in other features.

Reducing technical debt.

So what if you already have technical debt, planned or otherwise? You need to pay it down. I worked at one company where they went to their customers and let them know they had incurred a lot of unplanned technical   debt. They needed to pay it down. This meant the product would not be growing new features but it would be more stable and scalable. Most the customers looked at the market and believed we could still stay ahead of the competition plus the cost of re-training staff on a new product, creating a business relationship with a new company, etc. would be more costly than giving us a chance. Some major clients walked away and we took a big hit. The clients who did give us a second chance wanted to see that we had a plan. They wanted to see how we would make sure it didn't happen again. They wanted to see a road map of how we'd get back to adding new features. We had to regain their trust.

Hopefully you never get to this point. You want to start addressing unplanned technical debt before it gets this bad. But how do you do it?

Try and leave this world a little better than you found it. - Robert Baden-Powell.

This is a quote from the father of scouting. The idea was, when you went camping to leave the camp site in better condition than you found it. If you show up at a camp site and someone has left some garbage, pick up their garbage as well as any garbage you create. If they trampled some saplings, plant new saplings.

I used to be a tradesman. Many homes in North America have aluminium wiring. Aluminium wiring was used because copper was in high demand and electricians could save money using aluminium wiring. However, the aluminium cannot handle as high a load. Homes were 40 to 60 Amp service. Today we are finding homes with 100 to 200 Amp service. The aluminium wiring cannot handle the higher amperage and fires occur.

New home owners buy a used home only to find out that it has aluminium wiring. At first, insurance companies were telling home owners you had to replace all the wiring or no insurance. Mortgage companies would then say, no insurance == no mortgage. So the home owner needed to come up with $50,000 to $70,000 to re-wire their new home.

This would be the equivalent of saving, I have too much technical debt so for the next year I'm going to add no new features and just refactor the code to remove technical debt. After a year the product will look the same from a customer's point of view but you will have spent hundreds of thousands of dollars. No customer is going to pay extra for a product that looks the same from their perspective. So you are out the money. Just like a new home owner saddled with $50,000 of extra debt on day 1 of their new home, this sort of debt can ruin a company. There is no way most companies can afford to take a year to remove technical debt.

The insurance companies soon realized they were asking the impossible and losing a LOT of potential customers. So they told the new home owners that whenever they renovated a room they needed to make removing the aluminium wiring part of the project.

Software companies need to do the same thing. Rather than give the customer nothing and spend 100% of your time reducing technical debt, whenever you adding a feature you should remove the technical debt as part of adding the feature. In other words, try and leave the software a little better than you found it.

Rather than removing all technical debt in a year it might take you 3 to 5 years but you are still adding new features and therefore getting revenue from customers.

How much time do you spend removing technical debt and how much time adding new features? This is a judgement call. You have to add enough new feature to make the customer feel they are getting their money's worth but leave enough time to reduce the technical debt.

Remember, it might have taken you 5 to 10 years to create the technical debt. It is not unreasonable to take a few years to remove it.

Testing under a deadline -- the definition of insanity

Ever have to test something with limited time? Or course you have. When I first started testing the plan for the Software Development Life Cycle (SDLC) was:

- get the requirements

- design the application

- start thinking about how to test it

- start implementing the design

- start testing the application

- fix the defects

- test it again

- ship it

Generally speaking there is a lot more iterating over parts of this and occasionally the application gets redesigned in the middle of everything.

On many projects, the plan is laid out so that each step gets a set amount of time. They put a buffer in for unanticipated problems. As the project progresses, when it looks like we aren't going to meet the target date, they put more resources on it, reduce the scope or both.

If things really aren't going well (at some point putting more people on the project just makes things worse and if you cut too much it just isn't worth shipping), they do Albert Einstein's definition of insanity: doing the same thing over and over again and expecting different results. They just do it faster and more frequently. In the end, the Spiral development model becomes the Death Spiral and testing time gets shorter and shorter on each iteration.

So what do you do as a tester?

If you started off well intentioned, you asked for say 2 weeks (10 business days) to do testing. After you get the initial design and more detailed information, you reestimate and ask for 3 weeks (15 business days). You create a wonderful test suite with hundreds of necessary tests. You feel you should be able to complete all the testing in 12 days (80% of 15 days is 12 days; keep 20% in reserve for unforeseen problems).

You ran the entire test suite and filed a few hundred defect reports. Development threw a new version of the product over the wall but you now have 5 days to test it again. Do you run the entire test suite? There are a few possible choices.

The first is to order the tests by importance. This way, if you don't get through all the tests before the ship date, you have covered the most important ones. But 5 days is only 33% of the expected 15 day test period. Even if we assume nothing goes wrong, 5 days is less than 42% of 12 day test period. This means best case scenario we have 58% of the tests not run. Not good.

We can test those things which will be hardest or impossible to patch once the product has shipped. Then we continue testing after the product has shipped, create a patch and hope we didn't lose too many customers.

Another option is to examine source code and determine what areas have definitely not changed. Determine which test cases apply to only those areas which have not changed and drop them from the test suite. If the test failed previously, we know they had to make changes to that code or the defect still exists. So tests which failed must be kept. How much interdependency does the product code have? The poorer the development the harder it will be to test this way. From a QA standpoint, good development comes before good testing.

If your staff is fairly senior and good at adhoc testing, you might have a better chance at just re-testing the failed tests and using good judgement to test the related features. But now we are off the test plan. How do we record what testing was done and the results? What I like to do is keep a copy of notepad/textpad/vi/etc. open and write point form notes on what I'm doing, as I do it, and what results I see. If the application is very graphical and that matters, use something like Microsoft Word or Apple Pages so you can copy and paste images into the document. The important thing is being able to reproduce the tests if needed.

I once had the idea of recording the video output onto video tape. Then you could play the recording back and transcribe the video to a test plan, if necessary.

All of these ideas are accepting the idea that you have to deal with the limited time to test. How about talking to project management (or their management) about doing a post mortem. Figure out what went wrong and put things in place to make sure it does not happen again.

The most important thing is to not do it again; remember Einstein's definition of insanity. Features will have been dropped because the project was behind schedule. There will be a strong pressure to release a patch or the next version in order to get those dropped features out to the customer. Resist the urge to just dive in and do it all over again.

Good list of books you should read

A group of people/companies got together and created The SoftWare Engineering Body Of Knowledge or SWEBOK. It is basically a reference to software engineering covering everything from requirements to maintenance. There are chapters on all the stages of the software development life cycle.

A lot of that book is based on traditional, proven methodologies. A lot of what you see in Agile or eXtreme Programming environments is not well reflected in the SWEBOK. It is still a good starting point for anyone wanting to understand things like Testing or Software Quality Assurance.

The book is a list of known definitions and descriptions plus, and this is the important part, a reference to where the definition/description came from. It is the list of reference material at the back of each chapter which makes the SWEBOK most valuable. It is a great place to build a good library of books from.

When I started out in programming, I was self taught. I only knew things from trial and error or based on whatever books I could find in stores or the public library. Books like The Art Of Programming by Donald Knuth or even Algorithms by Robert Sedgwick were not available in computer stores. Once I found these books I started to realize how little I knew. To paraphrase Socrates, "I am smarter than most because I know I know nothing." Read the SWEBOK and you'll know just how much there is to know and hopefully you'll be able to realize how much more there is for you to learn.