Thoughts on Testing - Breaking Down the Test Plan
Test plans, or test specs, are designed to share with a product team exactly what a tester plans on covering to ensure that their feature is ready to ship to customers.
Problem 1: Who Is the Test Plan For?
Thinking about who the test plan is for can greatly influence the results of what comes out of the actual document. There are usually a bunch of vested parties such as developers, program managers, peer testers, test managers, test vendors, sustained engineering teams and security experts just to name a few. Each one is looking for something different out of the test plan document.
Here’s a breakdown of the lifetime of a test spec and who cares about at each milestone:
Milestone
Who Cares
What They Care About
Design Phase
Developers
Program Managers
Peer Testers
Test Managers
Testability of a feature
Design of a feature
Manual and automated test coverage
Impacts to other features
How long testing will take
Test Strategy
Implementation Phase
Developers
Program Managers
Test Managers
Test Vendors
Security Experts
Test Coverage
Checklist of signoff items
Security Scenarios
Test Strategy
Stabilization Phase
Test Managers
Program Managers
Peer Testers
Test Vendors
Sustained Engineering
Test Coverage
Checklist of signoff items
Test Strategy
Post Release
Sustained Engineering
Security Experts
Peer testers
Test Coverage
Test Strategy
Security Scenarios Checklist of signoff items
The two topics that are cared about in every milestone are test coverage and test strategy. Put simply, it is what is covered and why. Most of the time, test specs and test plans focus more on items in the design phase because this is when test plans are usually reviewed and iterated on. The people in the test plan reviews chose to focus on those items.
Problem 2: What Is the Test Plan Used For?
It isn’t always clear what a test plan is used for once it has been written. Often times, there are notes of work items and costs, lists of tests that need to be run or automated and perhaps specific performance and behavior goals to reference but it usually isn’t looked at again until sustained engineering handoff or something’s gone wrong.
Making test plans something that are consistently referenced, reviewed and used should be a goal for any test team. Otherwise, the exercise of writing the test plan is simply to think about how someone would test the feature and highlight areas of concern in a developer’s design.
Let’s go back to a subset test plan users:
Users
Needs
Feature Tester
Design Phase
Testability of a feature
Manual and automated test coverage
Test Strategy
Implementation Phase
Test coverage
Test work items
Test cases
Security scenarios
Stabilization Phase
Test Coverage
Test Cases
Checklist of signoff items
Test Managers Feature Team
Design Phase
How long testing will take
Test strategy
Test coverage
Implementation Phase
Test coverage
Test work items
Test Cases
Stabilization Phase
Test Coverage
Checklist of signoff items
Sustained Engineering Security Experts Future Feature Testers
Stabilization Phase
Test Strategy
Test Cases
Test Coverage
Checklist of signoff items
Problem 3: Keeping the Test Plan Up-to-Date
If developers are following more traditional design then implement models, the test plan will usually be signed off prior to a developer starting implementation work. What happens to the test plan now?
There are three scenarios to consider:
Scenario
Test Plan Needs Changes
Developer Design and Spec do not change, feature is implemented exactly as described
No
User requirements change and minor updates or design changes are needed. Feature goals modestly updated and changed
Yes
Major changes to requirements and design cause implementation to drastically change. Feature is implemented completely differently from original design.
Yes
But what usually happens in these situations? If deadlines or milestones are tight, the actual test plan might not be updated. Or, more often than not, the test cases and scenarios are updated on a case by case basis in automation but not necessarily captured in the original document.
Test Plan of the Future™: Features and Requirements
So, after all of that what does the Test Plan of the Future™ need? Here are some of the requirements that I thought of that will address the needs of the primary users:
- Test Items
- Test Cases
- Manual
- Automated
- Security Specific Tests
- Test Coverage
- Code Coverage
- Scenario Coverage
- Checklist for Signing off
- Items discussed among feature/product owners
- General Test Strategy Notes
- Test Cases
- Feature Spec/Dev Items
- Features
- Prioritized Scenarios
- Easy to Update
- Reflects the current state of the product
This list of requirements satisfies a majority of the needs laid out above. If test plans can reflect the current state of the product, highlight feature and scenario coverage all while being easy to update, it can become a really valuable tester resource.
Test Plans: Looking Ahead…
As things progress, I’m going to try investigating and implementing different types of Test Plans of the Future™. Stay tuned for future posts on this topic…