Two different approaches to testing a server

I had a discussion yesterday with a tester on the team about predicting and measuring server performance (with several OneNote applications syncing to notebooks on the server, of course). Anyway, my first thoughts were about building a mathematical model to describe the expected behavior. The other tester wanted to focus on testing the implementation and was not so focused on trying to predict the expected performance.

Both approaches here make sense. In favor of the mathematical model, a Markovian Chain, you can really come up with some interesting metrics. If you want to define the users as editing a page in the notebook once every half hour to hour by adding 1K of text, and have some other number of clients simply downloading the page, you can start to predict how many uploads will be made, how many downloads will be made and some statistics around bandwidth usage. When the testing is done, you can compare the tested results against the model, and if the model is complete, you can get a good gauge of the efficiency of the implementation.

As a side note, this latched into my mind probably because I have a math degree, and building these models can be a lot of fun!

An advantage of the pure testing approach is that you avoid the work of developing the model, get real world numbers and avoid the errors that can creep into the model (for instance, did you remember to model some percentage of packet loss? Latency? Users exiting OneNote before sync was complete? Etc…). It can also let you see which areas of your implementation are weak and can be used to direct further testing and development work.

The primary disadvantage of the "just jump in and test approach" is that you may never know when you are done. If you had a model that suggested you should be able to support 100 users per CPU core, you can know that 75 users per core is not good enough by a large factor. If you just jumped in you may find out you can support those 75 users, but don't know what is expected.

And that leads to a fundamental theorem of software testing: If you don't know what to expect, you are not testing. You are just clicking buttons.

Kind of an interesting conversation from both a test and math point of view.

Questions, comments, concerns and criticisms always welcome,

John

Comments

• Anonymous
  June 23, 2011
  Sometimes you need to click buttons for a while to get a baseline of data to use for the mathmatical models.