All About Load Test Planning (Part 4-Building the Load Profile)
Let’s start off with a review of this series. In the first post, we set our goals and objectives and defined the “purpose” of our testing. In the next post, we looked at data considerations that are independent of our goals, but still need to be considered. In the third post, we laid out the design and criteria for the items we wish to use to generate load. Now we have enough information defined that we can assemble all of these pieces into the appropriate load test. (NOTE: This series does not cover the building of the web or unit tests from the plan defined in the third post. I am assuming that those tests were already built between the third post and this post)
Here are some terms we will use in this post:
| Term | Scope | Definition |
| User | Real World | A person who will use the system that is being tested. |
| vUser | Harness | A representation of a 'user' inside a testing tool (virtual user) |
| Iteration | Harness | An Iteration is a single vUser performing the entire steps of a single use case. At the end of the iteration, the vUser is ready to perform another iteration. The next iteration can be the same User or a different User (SEE NOTE BELOW FOR CLARIFICATION). |
| Work Load | General Term | The Work Load represents the total amount of work that a system is subject to. It is obtained by combining the number of users [real world] or vUsers [test harness] and the amount of work each user or vUser does during a certain timeframe. |
| Think Time | Harness | Think time is the amount of time a vUser waits between each step of a use case (web test). |
| Pacing | Harness | Pacing is the amount of time a vUser waits at the end of a test iteration before starting the next iteration. |
NOTE about iterations. In the definition, I stated that “the next iteration can be the same USER or a different USER. The key item here is that the next iteration will always happen as the same vUser, but the real world user being simulated can be either a NEW user or a RETURN user. A NEW user is a person who has not already been on the site in question and therefore does not have any browser cache or session data/cookies available. A RETURN user has cache and session/cookie data.
Explanation of the difference between USERS and VUSERS
A USER is the “real world” person that is described in the use cases and scenarios used to create a test harness. The number of USERS, as well as their usage pattern is determined by analyzing real world numbers from production environments or from projected usage analysis and has nothing to do with the operation of a Visual Studio Load Test.
A VUSER is the set of resources that Visual Studio uses to simulate the work of one of the above users. The total number of VSERS is determined by understanding the maximum number of CONCURRENT TESTS that need to be executing and NOT by the total number of USERS that need to be simulated.
There are three ways I like to measure USER counts and each maps back to a VUSER differently:
- Total USERS – This is the total number of UNIQUE people that use the system over the timeframe being measured. I use this number to determine how many rows of data and/or user accounts, etc. I will need to feed into the test harness. This value does not necessarily have any direct correlation to the number of VUSERS.
- Total Sessions (Tests) – this represents the total number of times all of the users connect, do work and disconnect during the timeframe being measured. This number will always be as large as “Total USERS” and may be larger (if a given unique user goes to the web site 3 different times then the total users is 1, but the Total Sessions is 3). This value does not necessarily have any direct correlation to the number of VUSERS.
- Maximum Concurrent Users – this represents the maximum number of sessions that can be concurrently connected and doing work. This value defines the number of VUSERS we need to have in our test mix.
Choosing the Proper Load Model
The load model in this case is the test mix option you choose inside a Visual Studio load test for the scenarios you create. For this post, I will focus on only two test mixes and show when and how I use each one. I will look at my Goals to determine which load model to use. After I choose the load model, I will figure out the numbers to use. Each model requires a different way of calculating the values to use. Below are the two test mixes I use, as well as the pros and cons of each:
Based on User Pace
- Useful for modeling a known and consistent load against a system for the purpose of obtaining the application’s behavior at a given load.
- Easy to create specific user rate
- Load Test Scenarios may be more complex and require planning.
Based on total number of tests
- Useful for driving load at various different rates (step loads) for the purpose of identifying bottlenecks and/or determining scalability limits.
- Easy to create a maximum load.
- Difficult to create and reproduce a very specific user rate.
Choosing the Proper Numbers
Based on User Pace
From our previous post, we have a plan for “School’s Back In Session” with some numbers. We want to see if the system can handle the expected load, so the “User Pace” option is perfect. I will have two scenarios, one for teachers and one for students. The table of values from the test plan is below. Notice that I have added an extra column (# timer per user per hour). I added this because I do not want to run my test for a week, so I need to “translate” the value. I assume a 40 hour work week.
| Scenario | Use Case | # users | #/user/week | #/user/hour |
| Teachers | Login | 150 | 5 | 0.125 |
| Teachers | Enter Grades | 150 | 5 | 0.125 |
| Teachers | Update HW | 150 | 25 | 0.625 |
| Students | Login | 3000 | 5 | 0.125 |
| Students | Check Grades | 3000 | 4 | 0.1 |
| Students | Get HW | 3000 | 20 | 0.5 |
From this table, we can see that the numbers don’t work. I cannot put fractional numbers into the test harness, so I need to do some more “translation.” I will first calculate the total number of iterations that have to occur for each use case, then I can divide that up in order to get a new set of values for “# vUsers” to use (in some cases I will need to round the numbers):
| Scenario | Use Case | # users | #/user /hour | ~ Total | # vUsers | #/vUser/hour |
| Teachers | Login | 150 | 0.125 | 19 | 19 | 1 |
| Teachers | Enter Grades | 150 | 0.125 | 19 | 19 | 1 |
| Teachers | Update HW | 150 | 0.625 | 94 | 19 | 6 |
| Students | Login | 3000 | 0.125 | 375 | 75 | 5 |
| Students | Check Grades | 3000 | 0.1 | 300 | 75 | 4 |
| Students | Get HW | 3000 | 0.5 | 1500 | 75 | 20 |
We had to drastically change the number of vUsers we allocate for the test, but this isn’t a big concern as long as the load that the server has applied to it is similar to what we expect in the real world. In this case it is. (There are situations where the math is not this simple, and there are other calculations you should make to verify the accuracy of the load. I will cover these topics in the next post. )
Based on total number of tests
For this example, let’s focus on “Are you ready for some football”. To recap, we wanted to find the breaking point of the system. Let’s review the numbers again. We do not define the TOTAL number of users. We define the ratio of users that perform a given task compared to the other task. Think of this as “For every person that buys a ticket, 100 people will play in a draft league, and 2000 people will follow the game online.” The next column is similar; “Each user will only buy one ticket for a game, but will check the status of the game 180 times during the 3 hour game window.
(Did you notice anything unusual about the last sentence? I used the word “week” to describe one value in the fourth column and “3 hour window” to describe another value. This is a great example of a place where reviewing the values before moving on is critical. I would immediately go back to the business partner and make sure that I understood what the real rate of work is, especially since you can imagine that the NFL website is MUCH busier on Sunday than any other day. Therefore, I didn’t calculate the values for the fifth column. I got the values for the fifth column by getting more info from the owners of the site.)
| Scenario | Use Case | # users ratio | #/user/week | #/user/hour |
| Fantasy | Draft | 100 | 0.1 | 10 |
| Fantasy | Stats | 100 | 20 | 10 |
| Real | Tickets | 1 | 1 | 1 |
| Real | follow game | 2000 | 180 | 60 |
Now I can build the model. First I will use percentages for the loads, so I need to translate the fifth column from a ratio to a percentage. I can choose to run all four use cases in one Visual Studio Scenario, or I can keep them split in two Visual Studio Scenarios. For the example, I will keep them split. This is a more complex way, but it helps me illustrate all of the considerations needed for modeling load.
The percentage for each of the two “Fantasy” use cases is 50%. The percentage for the “Real” Scenario is split up by taking 100 and dividing it by the total (61), then multiplying each Use Case’s ratio by the result and rounding: Tickets = 1.7, rounded to 2. Follow = 98.3 rounded to 98. For this load, I want to use a step load pattern to increase the load, so I define the scenarios to run with the percentages I just calculated, then I define the overall load by setting the number of vUsers to step through various quantities. Because I chose to keep the two scenarios separate, I need to account for the different volumes for each scenario. I have an average of 20 items for “Fantasy” compared to 61 for “Real” Therefore I will use a 1 to 3 ratio for the vUser counts in the test (This is the extra consideration I mentioned in the previous paragraph):
Conclusion
In this post I showed you how to take several different requirements from the test plan and start to convert those into real tests. I showed you situations where you needed to change (or translate) some of the values to allow the tests to be created, but hopefully you can see how those changes did not change the perceived load that the System Under Test is seeing.
In the next post, I am going to expand on several concepts from this post and show you more examples of choosing the right mixes and additional issues you may encounter in the conversion, as well as suggested ways to handle these issues.
Comments
- Anonymous
September 11, 2014
The comment has been removed