API usability and the cognitive dimensions framework
Brad Abrams posted a
slide deck that I use to talk about the cognitive dimensions framework which we use
at Microsoft to evaluate and describe API usability. I thought I would take this space
to provide a little more explanation about how we use the framework.
The cognitive dimensions framework presents a set of dimensions that describe aspects
of an API that impact its usability. For example, one of the dimensions is 'Abstraction
Level' which describes the types of abstractions that the API exposes. Another
dimension is 'Working Framework' which describes the mental load that the API places
on its users. An API may be found to be unusable because the abstractions it exposes
are too low level. Or it might place too much of a mental load on its users, requiring
them to keep track of multiple objects in their head as they write code to perform
a task. Or it could suffer from a combination of both problems, exposing multiple
primitives that the developer has to keep track of in their head as they write code
to accomplish a given task.
By evaluating an API with respect to each of these dimensions, you can form a picture
of what the API looks like overall with respect to the framework. We like to create
pretty graphs to do this. For example, here is a graph of an evaluation of some fictional
API:
The black dotted line in the graph represents the API being evaluated. The spokes
in the 'wheel' represent each of the different dimensions. End points (the middle
and outer edges) on each spoke represent the end points on the scale for each
dimension. Thus the point on each spoke where the black dotted line crosses it
corresponds to the point on the scale for that dimension that the API has been valued
at.
The most common way that we use to perform such an evaluation is to perform a usability
study in our usability labs here at Microsoft. We invite developers from outside of
Microsoft to come in to the labs and use an API on a given set of tasks, over a number
of hours. We observe each developer as they work from behind a one way mirror and
make notes about what parts of the API work well and what parts don't work well. We
usually bring at least five developers in for at least two hours each for every study
we do. So after ten hours worth of watching developers write code against the API,
we spend time reviewing and analysing the data we collected and producing our evaluation
of the API with respect to the cognitive dimensions framework.
However, there really isn't much point in going to all this bother if all we end up
with is a graph like that shown above. It's not very helpful to provide the team developing
the API with a graph showing how their API fits on the cognitive dimensions - there
is nothing on that graph to tell them whether or not the API was judged usable or
not. There is no baseline with which to make a comparison. That is where the individual
developer personas come in to play.
We have defined what the three developer personas (Opportunistic, Pragmatic and Systematic)
would judge to be the ideal API with respect to the cognitive dimensions framework. In
other words, we have created profiles for each of the personas that describe
for each dimension, the ideal point on the scale for that persona. The personas define
quite different working styles so the profiles have ended up being quite unique. Thus
we now have a way to judge whether or not an API is usable by comparing its evaluation
with the profile we produced for the developer persona that best matches the target
audience for that API. We can do this comparison in the same graph:
The blue line on the graph represents the profile of a particular developer persona.
For example, it shows that this persona prefers APIs that expose aggregates (the outer
edge on the abstraction level scale) and APIs that have small working frameworks (the
inner edge, or middle, of the working framework scale). If you compare the points
where the blue lines cross each spoke with the point where the black lime crosses
each spoke, you get an idea for how well the API matches the persona's ideal API.
Hopefully you can see that this particular API doesn't really hit the mark for this
persona (there are big differences between the points where both lines cross on most,
if not all, of the spokes in the graph).
Fundamentally the cognitive dimensions framework provides developers with a common
vocabulary with which to talk about and discuss API usability. The benefits of such
a common vocabulary are twofold. First, the language shapes the distinctions that
are considered important to attend to. By explicitly describing each of the dimensions,
it is more likely that they will be attended to. Secondly, the language allows developers
to discuss issues using terminology that they can assume will be understood by others.
We've found that the cognitive dimensions have really helped us get to the root cause
of issues that we have observed in the usability labs. For example, in one study,
we observed lots of developers spending a large amount of time in the help docs looking
for code samples that would show them how to accomplish a given task. The first interpretation
of this data was simply 'Fix the help docs!'. However, when we used the cognitive
dimensions framework to describe the issues, it became clear that the reason the developers
weren't successful when they were searching through the help was because what they
were looking for in the help simply didn't exist. The API they were working with exposed
a set of abstractions that were at the wrong level for these particular developers.
They expected a particular type of abstraction to be exposed by the API but since
it wasn't, they couldn't find anything about it in the help docs. As a result of
this, the API team redesigned this API to expose abstractions more in line with what
developers were expecting. When we retested the API, it worked much better.
The dimensions in the framework have been adapted from decades worth of work done
at the University of Cambridge on programming language usability. So far we've used
the framework successfully around the company to evaluate APIs and to help to design
usable APIs. It's still evolving though, so it would be great to hear your
comments and feedback on this. I'd love to know about other frameworks or methods
for evaluating and describing API usability also.
Comments
- Anonymous
October 08, 2003
Interesting article, but unfortunately your images seem to be broken links. :( - Anonymous
October 09, 2003
What about evaluating usability simply on the quantity of user code needed for a set of programming scenarios, relative to the number of classes / functions / properties / function parameters provided? One side is simplicity of coding, the other simplicity of comprehension? Is this too simple?In any design the programming scenarios are the toughest things to produce. That seems to be more a result of years of experience in the domain than anything else. If you don't have the right scenarios picked, the user simply cannot accomplish his task, as you mentioned. So I think the experience of the API designer is the biggest factor in usability, but numerical metrics might uncover problems. - Anonymous
October 09, 2003
Sorry about the links, my mistake! I'm trying to get them fixed. - Anonymous
October 09, 2003
Good questions Frank. The cognitive dimensions framework measures simplicity via the dimension called work-step unit which is a measure of the quantity of user code required for a single unit of work, or step. For example, connecting to a database might be a single step towards accomplishing a final goal of getting results of a query from a database. Developers have their own expectations of how much work is required to complete a particular step and some developers prioritise this dimension over others.The number of classes/functions/properties provided is not directly measured by the cognitive dimensions. Instead, the cognitive dimensions focuses on the number of classes/functions/properties that the developer must comprehend before they can do any meaningful work. If the developer need not be aware of some subset of classes in the API to accomplish a particular task, we don't include them in this measurement. Other dimensions look at how easy it is to select the classes/methods/APIs that you need from all those exposed by the API.We haven't really looked yet at relative comparisons between individual dimensions which is a good suggestion. It's clear that there are relationships between the different dimensions (they are not orthogonal) and that's something that I would like to explore in more detail in the future. - Anonymous
October 09, 2003
Thanks for the excellent explanation. Nice to see someone is thinking about these things.I always judged an API design by whether tasks can be accomplished, the amount of code needed to accomplish tasks, and whether the API can be enhanced in the future, in that order. I see this is a little simplistic.I think there is an inverse relationship between amount of code to accomplish tasks and the number of classes/functions/properties/parameters. - Anonymous
May 20, 2004
stevencl,
This is a really excellent application for Blackwell & Green. I hope MS is using it in other ways.
What op definitions do you use for scoring? - Anonymous
June 15, 2004
Duncan, to answer your question about the other ways we use the cognitive dimensions framework, we have been trying to use it to derive design guidelines.
Regarding the definitions we use for scoring, I wrote up a series of articles on my blog describing how we score each dimension. Hope these are useful. - Anonymous
July 15, 2006
Charles has a good post where he talks about the right balance of skills between developers and... - Anonymous
July 15, 2006
PingBack from http://microsoft.wagalulu.com/2006/07/16/how-technical-a-good-technical-pm-should-be/