Measuring the quality of content submissions

In this blog entry I examine the practice that we apply for evaluating quality with respect to glossary entry submissions for the Microsoft Dynamics AX glossary.

While the focus of this blog entry is on glossary entry submissions, the practice I document is applicable to all product content that we deliver, such as user interface strings, help topics, and so on.

What is quality?

Quality can be defined for many contexts. For example, qualitycan be defined in context of a distinguishing characteristic, a perception, or with respect to a condition.

If I asked you what quality means to you, you might offer a response based on your perceptions of quality, or based on your experience with an item’s condition or its characteristics. Your response might evolve as a result of how much or how little you value the item. And your perception of quality would likely include a consideration of the weight of the value of quality against the specific item that you are evaluating and how important that item is to you or to your organization.

For example, if you and I were to evaluate the same chair for purchase, our perceptions of quality might differ with respect to the condition of workmanship or its upholstery.

So would a difference in our conclusions make either your or my evaluation of quality more or less valid in this example? No, it would not.

What this example indicates, though, is that the concept named quality can be classified as either subjective or objective. With respect to our terminology practices, we developed a practice of evaluating quality that is objective.

Our need for a practice for evaluating terminology quality objectively was borne out of:

• The need to move people away from using terms that existed in other applications that they had developed or used, many of which contain terms that are not verifiable in their respective domains.
• The need to align the diversity of roles in the domain, industries, locales, and so on that the Microsoft Dynamics AX product supports.
• The need to align with the Microsoft Dynamics AX product design. We build a platform that our partners tailor, so the concepts and their names need to remain neutral and consistent with concepts and their terms that are verifiable in the domain.

How do we measure the characteristics of quality?

The concept quality cannot stand alone as the criterion for an objective evaluation. The concept quality requires a measurement to ensure that the evaluation can be calibrated objectively and that the conclusion of the evaluation is repeatable, no matter who the reviewers might be.

When measured, quality is measured against a normative system of measurement, and any variations to this measurement result in nonconformance.

Normative systems of measurement can be units of measure that produce an exact measure of some countable operational or financial characteristic, and it is represented by a number. For example, systems of measurement that are applied to software design might include measurements of technical requirements, test requirements, conformance requirements, performance requirements, audit requirements, and so on.

The specific system of measurement that we apply to the evaluation of product content is conformance to sets of writing guidelines. Product content, including glossary entries, user interface strings, help topics, and so on, should read as though it was written by one team and not by many team members.

When objective guidelines are documented and accepted, such as rules of grammar and syntax, product content, such as proposed glossary entries, can be evaluated against established guidelines and objective feedback can be offered that documents conformance or nonconformance to guidelines. An example of an objective measure with respect to definition authoring might be: Does the definition follow prescribed grammar and syntax guidelines?

The Microsoft Dynamics AX terminology practice requires that:

• Concept definitions and term selections be accurate and precise and verifiable in the real-world domain. For example, the language and terms that appear in the Microsoft Dynamics AX product should align with the established language and terms that people learn when they study a domain and in which they get certified in order to perform a certain role.
• The concept must be realized, implemented, and aligned product-wide. For example, the concepts named vendor and supplier are unique and are not synonymous, so the Microsoft Dynamics AX product should reflect the concept that has been realized and implemented in the product—that is, the concept named vendor.
• Definitions must follow established grammar and syntax rules. For example, definitions must apply correct capitalization and punctuation rules.
• Specific definition wording patterns must be applied based on the part of speech of the term. For example, definitions for nouns must start with an article (a, an, the).
• Specific definition wording patterns must be applied when defining concepts that are of the same type. For example, the definitions for documents, such as customer invoice, request for quotation, and sales order should start with the phrase A source document that documents.

We established wording patterns to align to the practice of software design, which applies patterns. And we established wording patterns because of the number of repeatable activities in our domain. We have numerous documents, such as source documents and business documents. Because they all document something, we established a wording pattern to maintain consistency in the wording for definitions that document the same concept type.

When a proposed glossary entry is submitted for consideration as an entry for inclusion in the Microsoft Dynamics AX glossary, I measure the proposed glossary entry against these and other established objective guidelines, and at the end of my review, I assign an objective mark—Approved or Denied—to the proposed glossary entry.

The key to my remaining objective is to ensure that the guidelines I authored are objective so that the evaluation questions that I pose can conclude with an objective, binary response: yes or no; 0 or 1. I need to be able to enumerate the evaluation questions and then count the number of yes responses and no responses, thereby determining a numeric result, such as yes=6; no=0; or yes=2; no=3. This numeric result would inform next steps for me (the terminology manager) or for the submission contributor.

The sum of the parts: Examples of how we measure the characteristics of quality

When a glossary entry is proposed for inclusion in the Microsoft Dynamics AX glossary, the accuracy, precision, and quality of it is verified and validated against established guidelines. The result of the evaluation determines the quality of the submission objectively—that is—with respect to conformance to established guidance. This result determines whether the entry is eligible for inclusion in the Microsoft Dynamics AX glossary or if the entry requires modification.

I offer two groups of glossary entries below that document types of concepts in Microsoft Dynamics AX, and for the purpose of example, I evaluate the quality of these contributions against only these two guidelines:

• Specific definition wording patterns must be applied based on the part of speech of the term.
• Specific definition wording patterns must be applied when defining concepts of the same type.

The specific questions that a reviewer might ask in evaluating the quality of glossary entry submissions with respect to these two guidelines might be:

• Does the definition apply the established wording pattern based on its part of speech? [Noun entries should start with an article.] Yes/No
• Does the definition apply the established wording pattern if a wording pattern exists for the concept type? Yes/No

Source document concepts

In the following proposed glossary entry submissions, I underline instances where the guidelines have been met as prescribed.

• product receipt (noun) – A source document that documents the receipt of products ordered, the receipt of products returned, or the receipt of products received on consignment.
• purchase agreement (noun) – A source document that documents the offer to buy products or the acceptance of an offer to sell products in exchange for payment.
• purchase order (noun) – A source document that documents the offer to buy products or the acceptance of an offer to sell products in exchange for payment.

Rule concepts

In the following proposed glossary entry submissions, I underlined instances where the guidelines have been met as prescribed.

• rule (noun) – A condition-action pair that prescribes the action taken when the condition is met.
• accounting rule (noun) – A rule in an accounting system that controls the principles, methods, and procedures for classifying, recording, and reporting the financial consequences of business events.
• recognition accounting rule (noun) – An accounting rule that prescribes the recognition of revenue and expenditure in accounts and on financial statements.
• revenue recognition accounting rule (noun) – A recognition accounting rule that prescribes the recognition of revenue in accounts and on financial statements.
• expenditure recognition accounting rule (noun) – A recognition accounting rule that prescribes the recognition of expenditure in accounts and on financial statements.

Both sets of examples offer proposed glossary entry submissions that meet the two objective guidelines (as well as others that are not noted here). As a result, these proposed entries were approved for inclusion in the Microsoft Dynamics AX glossary.

Microsoft Dynamics AX glossary

I invite you to peruse the Microsoft Dynamics AX glossary to see examples of other types of entries that met our terminology quality acceptance bar. And I welcome any feedback that you have to offer.