How I Learned to Use Scenarios to Evaluate Things

Note: This article is updated at How I Learned to Use Scenarios to Evaluate Things.

I first learned to use scenarios to evaluate things when I was working on security guidance for the Microsoft platform.   It was the most obvious way to really get hardcore:  Map out the most important scenarios that people perform, and use those as tests to evaluate.

I know “scenario” is an overloaded term, but in software development it typically means one of three definitions:

1. Same as use case
2. Path through a use case
3. Instance of a use case

#3 is often preferred because it provides a testable result.

In the broader industry, here are some common definitions …

• A description of several possible descriptions of a situation
• A plausible and often simplified description of how the future may develop, based on a coherent and internally consistent set of assumptions about driving forces and key relationships. Scenarios may be derived from projections, but are often based on additional information from other sources, sometimes combined with a “narrative storyline”.
• A postulated sequence of possible events
• A story of "myth" about the future events that projects a personal view of the future. This view constitues the context for planning process. Scenario building is a method to objectify "hidden design assumptions" and to play with various "what if's" about the future
• The entire spectrum of environmental considerations that have interaction with system(s) under analysis or those of interest for training purposes. The spectrum includes physical environment, threat conditions, rules of engagements, and systems performance and effectiveness.

Growing up at Microsoft, I saw “scenarios” used in the following ways:

• Product Teams
  • With customers:
    • “What’s your scenario?
    • ”What are you trying to accomplish?”
    • “What’s your context?”
  • In Specs:
    • Scenarios up front set the context for the feature description.
• Marketing
  • Encapsulate/communicate key customer pain/problems

Our thinking and maturity around scenarios has evolved a lot since then, but the most important thing to keep in mind is that sometimes people use scenarios to focus on the problem side, sometimes they use them to focus on the solution side, and sometimes they use them as a bridge between the problem and the solution.

When people use “future scenarios” they typically are showing “Future Capability Visions”, or storyboards of possibility.   They are a great way to help people envision what’s possible.

Scenarios themselves are incredibly powerful for analysis, putting requirements into context, and shaping design.

This brings us back to my story about how I learned to use scenarios to evaluate design and implementation in a very deep way.   When I was working on security guides for the Microsoft platform, I focused on scenarios to map out the priorities and test cases.  Here is a sampling of some of the scenarios we used to help us scope the security work, focus our efforts, and score our results:

Arch / Design Solutions

• How to identify and evaluate threats
• How to create secure designs
• How to perform an architecture and design review

Development Solutions

• How to write secure managed code
• How to handle exceptions securely
• How to perform security reviews of managed code
• How to secure a developer workstation
• How to write least privileged code
• How to constrain file I/O
• How to prevent SQL injection
• How to prevent cross-site scripting
• How to manage secrets
• How to call unmanaged code securely
• How to perform secure input validation
• How to secure Forms authentication

Administrative Solutions

• How to implement patch management
• How to secure a database server
• How to secure an application server
• How to secure Web services
• How to secure session state
• How to manage application configuration securely
• How to secure against denial of service attacks

I used simple, goal-oriented language (how to blah), and focused on the solution side.   We were already well aware of the problem side of the equation.   Our best scenarios were the result of asking:  “What do you want to accomplish?”

The big idea here was that I wanted to use scenarios to shape our work and score our success.

It was like having the questions for the final exam up front.   We used the best experts to really identify a complete map of what developers, architects, and administrators need to accomplish.

By mapping out all the scenarios that mattered the most, we had a comprehensive map of what to solve for.   Then it was just a matter of rallying the best brains inside and outside of the company to solve the challenges.

Think of it as the ultimate crowd-sourced quiz where the best in the world could share their best knowledge, experience, and insights to advance the art and science of software security.   (Yeah, scenarios really can be a powerful thing.)

When our security guidance made a big impact in the industry, changing how people actually did software security, I recognized the power of scenarios.    We basically leap frogged ahead in terms of our capabilities because we used real-world scenarios to focus our efforts.

By using scenarios, we performed extremely well in any platform comparisons because we actually had a full map of the most important scenarios and we solved them by design.   We didn’t luck or hope our way into nailing the things that mattered.  We mapped out the scenarios that mattered, and drove from there.

From there, I became “the scenarios guy” for the team, and started to dive deeper into ways we could use them.   My next move was to use them more deeply for performing inspections and for raising quality.   I learned to appreciate the distinction between “usage scenarios” for functional things vs. “architecturally significant scenarios” for cross-cutting things.

Along the way, I dove into various software evaluation methods, including:

• ALMA - Architecture-level Modifiability Analysis
• ATAM - Architecture Tradeoff Analysis Method
• CBAM - Cost Benefit Analysis Method
• FAAM - Family Architecture Analysis Method
• SAAM - Software Architecture Analysis Method

But, the most important insight, (in hindsight), was that utility trees could help visualize the trade-offs among quality attributes.   The other lesson though was that it was tough to integrate and hop around a lot of different tools and techniques for designing robust architectures.  Related, it was even tougher because very few people knew the different vocabularies, approaches, and tools.

As a result, I focused on creating the Agile Architecture Method.  I wanted a more rapid way to prototype or evaluate architecture and designs, in an iterative and incremental way.   Also, I wanted to use the one language everybody seems to speak:  “whiteboard.”   (Why?   Because it’s visual.)

But when did I learn to really use scenarios for my own evaluations of every day things?   The first time I bought a digital camera.   I thought about my main usage scenario – I wanted to snap pictures of the whiteboard and share them.

It was simple enough.   The problem was that when I bought the camera, I only focused on one part of the scenario – taking the pictures.   It didn’t occur to me that because I got a fancy pants camera, that the memory disk was an odd size and didn’t fit in my laptop.   So whenever I took a picture, if I wanted to share it, I had to connect USB to my camera.

It sucked.

It was just enough friction that I think I did it three times, then stopped.

So the other lesson I learned, aside from using scenarios for evaluation, was how important friction is, both in terms of adoption on the user side, and how important of a consideration it is on the producer side:  you have to eliminate as much friction as possible if you want people to adopt your stuff.

The moral of the story is that the better you know the real-world usage scenarios, the better you can prioritize, design, implement, and validate your ideas, and, most importantly, get your ideas adopted.

You Might Also Like

Agile Architecture Method Business Scenarios for the Cloud IT Scenarios for the Cloud Evaluation Criteria Example Scenario Scoreboards for Testing Your Product Success Test-Driven Product Design