Ship your service into production and then start testing!

Think about Services shifting the Testing into Production

The topic of Testing in Production (TiP) is a growing area of debate in SaaS and S+S testing groups. While I don’t personally really believe you should ship your service and then start testing it, I often introduce the topic this way. It is very challenging to get testers to consider shipping anything that hasn’t been thoroughly tested. The notion that a tester could build out an adequate test environment or set of environments and find every single major bug in their service is just as flawed as ship then test.

I use this “Ship first and then Test,” approach to jar audiences of testers and to move the conversation from what must be done in a lab to what can and would be best tested in production. I will try to make a much longer post on the subject in the future but for now simply think about these questions.

How often have you shipped a service and found that you missed a bug because the test environment wasn’t enough like production? Now ask yourself as scale and complexity of services increases is it reasonable to think you can really make test enough like production that you can catch every bug?

The process of thinking about what can be tested in production can be an exhilarating exercise. You quickly get to a discussion around what testability features need to ship with the service to make this possible. What impact can TiP have on revenue and customer experience, how do we isolate this impact, and what tests are best conducted in labs are other very important questions.

Think about it and let me know what opinions you have and questions you would like answered.