New term... soak test

I was just reviewing some documentation for my new job and came across the term soak test. I had never heard the term before and tried to imagine what it might be.

Turns out the term soak test is essentially a load test conducting for a period of time.

A stress test is putting a load on a system which is beyond acceptable limits. The purpose is to see how the system handles the load. All systems will fail but a proper system will fail gracefully.

A load test is putting a load on a system which is within acceptable limits. The purpose is to confirm the system will handle what is considered an acceptable load. For example, when a university opens the online course selection system, thousands of students will log on and try to select their courses for the year. We know how many students are enrolled and there is the theoretical possibility they will all submit selections at the same time. That would be an acceptable load test.

I've looked at load testing over a period of time before but never knew it was called a soak test. From the descriptions I have read, you would put a load on the system and leave it there for a period of time. In some examples they talk about leaving the load for days or weeks.

For me, I would record metrics on the system and plot a graph. If any of the recorded metrics increased over time but with no change in the load, I would suspect a failure. For example, if I am recording used memory and the application continues to increase the amount of memory required, I can extrapolate (a) the application will run out of memory and (b) the approximate time when this will happen. There was never a need to run the test for days or weeks.

Technically, this was a soak test. I was running a test with a load for a period of time. However, I took the data from the load test and projected where it would go over time. By doing that I could see trends and determine when they would have an impact.

For example, on a system which is performing well I might see the memory usage go to 10,000 megabytes then plateau. On a system with a memory leak I would see it go to 10,000 megabytes then it would start to slowly climb. It might be that it leaks 100 megabytes ah hour. If the system had 200,000 megabytes of memory for the application, I can predict it will take 1,900 hours or approximately 80 days for an out of memory error to occur.

If I ran a soak test for 30 days I might assume the application is fine. I would still need to analyze the data to see that it was leaking memory. It would be easier to see the memory leak after 30 days but not always necessary.

Personally, I would run a soak test if I found no obvious problem using load testing and an analysis of the data. If I found a memory leak or data I/O issue without the soak test then a load test is sufficient to find the error.

Once the load test with analysis passes however, it would be prudent to run a soak test if time and resources allowed. You could even run the soak test after releasing the application. If you  do find an issue, create a patch.