Can Specs Really Reflect the Reality of the Product?

A common complaint about specs is that they “never reflect reality with 100% accuracy”, meaning that once written entropy sets in and they become increasingly inaccurate over time. In a world where the software product is implemented in code, not Word, the code is king. If the product is not implemented as spec-ed, you can argue all day that the product is not in compliance, but at the end of the day you ship code, not specs, so the product wins. Fortunately, those deviations are typically minute, and are mostly there for good reasons. In this entry I’d like to discuss some of the reasons for this, why it is a normal thing, and mostly focus on how to manage this.

The Code and Spec Will be Different

If you think about it, on the “race to the product” the spec starts first, and the code is working on catching up. Sometimes the code takes a different path, and that results in either a code defect bug or a spec bug. These course corrections are an integral part of this process- they are an unavoidable reality and we need to accept this and be ready for it.

One reason why the code might differ from the spec is non-implement-able specs. For example, in a team that I used to work on, the spec’s UI design was found during coding to be too complex to render in real time on the computers and operating systems that the product targeted and they had to be simplified. In that particular case the non-implement-able-ness was could only be discovered at coding/prototyping time, but there are other cases where this can be avoided with adequate spec review.

Other reasons are usability and cross-product alignment on fine details. I saw and example of this when 2 specs that were written by 2 PMs and defined different APIs for different (yet related) parts of the platform. The areas had similar features named differently and once someone looked at the big picture and realized that it does not make sense, the implementation had to change.

There are many other reasons for the code to deviate from the spec, such as simply not following the spec, issues uncovered during detailed design, post-spec-signoff changes that are discussed in meetings or over email and do not make their way back to the spec, incomplete or inaccurate specs (discussed below), incomplete or inaccurate implementation, external change requests (when the VP tells you that the button should be blue and not purple, you just change it J), and more.

Specs are Practical Documents

A good spec describes the product with great details, but there is always the product change (as discussed above), the forgotten detail, the question that was left unanswered, or the issue that was uncovered and needs addressing.

The spec is a formal (electronic) document and the reality of the product is fluid – you can try to keep the spec up to the minute with the product, but unless the changes to the code and (e)doc (sorry, I couldn’t avoid the palindrome J) are made in sync, the product and the spec will be different until the product is complete. In fact, at the end of the product cycle the specs are might still be different from the product since updating the spec to the latest implementation details is not always the most efficient use of time.

Specs are means to an end, and therefore one should take a pragmatic approach – the specs need to be “good enough” for their purpose (more on that below). If we spend all the time required to perfect the spec, describing the product down to the finest level of detail, we will never ship a product in a reasonable timeframe. This is one of the reasons why modern project management techniques focus on small release cycles, iteration, and correcting course.

Managing the Delta

So how should we manage the delta between the spec and the product?

The first 2 questions a PM asks himself with regards to deliverables are (1) “Who is the customer?” and (2) “What are the scenarios?” Specs, their consumers, and how they are used have been discussed at length in many other places, (let me know if you want me to summarize), but the main takeaway is that it is important that the specs are required to be up-to-date at critical junctions in the project: land before or together with dev detailed planning, provide test with all the info they need for their test planning, and give UE what they need in order to plan the documentation and document the behavior before the product is complete.

In between these checkpoints details would change, questions will be asked, and issues would be uncovered. Just like in the case of code defects, these spec issues need to be captured in a bug tracking system. Just like code changes, they have priorities and severities, details about the wrong/missing information (the bug) and what details are expected. Just like in the case of code defects, the issue is analyzed by the PM and either addressed immediately or left for later resolution. Spec omissions, wrong information, and behavioral changes that may prevent the implementation or testing of the feature should be handled immediately. Issues like “here’s an error case that we didn’t think about before, what is the exact text to display to the user in this case?” can be left for later resolution. As long as the issues are not lost and the critical information is provided in a timely manner, the spec can be updated on the predefined check-points. A benefit of updating the spec at set points is managing resource drain - updates could be time consuming and process intensive for the PM and for all the spec reviewers and consenters. A side benefit of updating at set points versus constant update is that the deltas between spec versions are easier to track and manage, and so people who are not intimate with the feature can review the differences and see what changed.

Assuming that specs are indeed updated at certain points, the project needs an immediate-response mechanism to support resolving urgent spec issues and in a timelier manner. Sometimes the wrong/missing information is well understood by everyone (the button is blue since the VP said to do so and the spec still says that it is purple), sometimes it is conveyed verbally or out-of-band to the defect tracking system (via email, other specs, new UI template, etc.), sometime a meeting is required, but in all cases that the wrong/missing information, and the fix if it is known, must be captured in the bug tracking system until the next revision of the spec is published and the bug is resolved. This will ensure that the details of the solution are not lost, and reduce the chances for confusion over what solution was agreed upon later in the game.

In Conclusion

Yes, specs can represent the reality of the product, when accompanied by a change request and tracking mechanism. When you make a change to the product, open a spec bug to make the change in the document. If there is something missing or unclear in the spec, open a spec bug to track the required spec work. The most critical part of the process is communication. My door is always open for people with spec questions, and if they asked about something that was missing or unclear I always ask them to open a spec bug and assign it to me. In addition to making my life easier (J), this is a great way to make sure that the people on the team get invested in the quality of the spec - when I resolve the bug it will automatically be assigned back to them and they will need to verify that the issue was covered to their satisfaction.

Main Takeaways

• A spec is a practical, living document – convey this(!) and make sure that everyone understand the spec update process
• Publish your spec update points – spec consumers need to know when to expect updates
• Use a defect tracking mechanism to track spec issues in a public, transparent manner
• Address spec bugs according to their priority and severity, just like you would address code defects – be proactive about possible issues and unblock people who are dependent on spec details ASAP
• No process or issue tracking system can replace human interaction – communication is in the core of being a PM – talk to the spec consumers often and regularly, make sure they understand the spec, and answer any questions that they may have