Emerging Guidance
Versioning Web Services
Introduction
Most software systems evolve over time, and it is always challenging to manage the associated complexity of their evolution. When the relationships between the parts of the software system become less coupled, the complexity increases because they can then evolve independently. And when the parts of the system are built by different teams, even more challenges present themselves. This scenario describes many Web service applications and highlights the need for a conscious versioning strategy to address these challenges.
It is important to understand that the complexities cannot be completely mitigated. This is partially why the groups who have tried to address these challenges through the use of industry standards and technology-specific approaches have not yet succeeded. An appropriate strategy manages and isolates the complexity by restricting it to specific parts of the solution based on certain factors, such as the characteristics, requirements, and constraints of the solution and the organizations involved.
The purpose of this topic is to provide pragmatic guidance you can apply to various stages of your service's evolution. This guidance is not only applicable to services built with the Service Factory, but because it is specific to the factors mentioned earlier, it is not necessarily applicable to all Web services and all scenarios. In addition to describing specific tasks listed in A Versioning Strategy, this topic also describes:
· Organizational and technical scenarios for Web services.
· Elements of Web services that evolve and that are considered in scope for this topic.
· Common terms and definitions related to versioning Web services.
· Core motivations and rationale that support this guidance and dissuade other approaches.
Emerging Practices
Emerging practices are a form of guidance provided when there is an opportunity to gather more evidence in an area before labeling it a proven practice. This topic represents an initial position on how to manage the evolution of many Web services.
An emerging practice differs from most patterns & practices deliverables, which are driven by proven customer and field experience. The patterns & practices team is aware of the success some organizations are having with the approach in this guidance, but the team is still in the process of collecting more evidence.
This type of guidance is subject to change and will eventually be formalized after more evidence is collected. The development team is looking for feedback that will help shape the final release of this guidance. Please provide feedback to the Service Factory Community site on your experiences and expect this guidance to evolve over time.
Versioning Concepts
The process of versioning a Web service may involve changing many different parts of the service. The following sections clarify the types of evolution a service may experience and the parts of the service may be affected.
Explicit Types of Evolution
When two things have a dependency on each other and one of those things change, there can be only one of two outcomes: either the expectations of that dependency remain intact, or the expectations do not remain intact. It will either be a breaking change or a non-breaking change. In terms of Web services, the Windows Communication Foundation (WCF) documentation describes this most succinctly as "A change is non-breaking if all messages that would have been processed successfully in the previous version are processed successfully in the new version. Any change that does not meet that criterion is a breaking change."
Non-breaking changes can be broken down into compatibility requirements that depend on the release schedules of the service and consumers. Backward compatibility requires a non-breaking change between a newer service and an older consumer. Forward compatibility requires a non-breaking change between a newer consumer and an older service. Figure 1 illustrates these requirements.
Figure 1
Forward and backward compatibility
When used in reference to Web services, the term "breaking" refers to major versions, and the term "non-breaking" refers to minor versions. It is always important to be explicit about these definitions, the actions performed for each type of change, and how the definitions and actions are communicated. For information about the actions, see A Versioning Strategy. For information about how the actions are communicated, see Relationship Scenarios.
A major revision explicitly represents a breaking change and should be denoted by changing the major version number (for example, from 2.x to 3.x). This type of change has no intentions of being compatible with anything that has a dependency on it. As a result, these types of changes are more expensive; therefore, they should be avoided whenever possible.
A minor revision explicitly represents a non-breaking change and should be denoted by changing the minor version number (for example, from x.3 to x.4). This type of change has every intention of being compatible with either existing or future dependencies. These changes have less of an impact to the relationship between the service and the consumer. Therefore, you should strive for minor revisions whenever possible.
In summary, minor revisions do not require new consumers be deployed, and major revisions do require new consumers be deployed.
Elements of Evolution
The relationship a consumer has with its service is complex, and many elements of that relationship can evolve. It is not the aim of this topic to address the evolution of all possible elements. However, the following list represents common elements that are in scope:
· Request and response messages. This includes adding and removing fields from messages. As you will see later, the most challenging of these scenarios is removing fields from response messages. Additional complexity is added when the messages are validated or stored for future processing.
· Number of operations available on the interface. This includes adding and removing operations from the service. The most challenging of these scenarios is removing operations, but the impact can be lessened with a subscription approach when it is possible.
· Method signatures. This includes adding and removing input parameters and return types on operations. The challenges can be lessened by using a conscious service design approach.
For more information about how to mitigate the challenges of evolving each of the preceding elements, see A Versioning Strategy.
The following changes are not as common and are considered out of scope for this topic:
· Message formatting. An example of this would be changing the format of a message from XML (SOAP) to a binary format.
· Message transport. An example of this would be changing the transport of a message from HTTP, to SSL or MSMQ.
· Underlying specification. This change refers to a revision of the specification. For example, if the service chose to support SOAP 1.2 instead of SOAP 1.1.
Changing the preceding elements most often results in a new second service that may or may not reside alongside the initial one.
Some changes are common, but due to their nature, cannot be as easily mitigated. The following are a couple examples:
· Semantic meaning of messages or operations. This includes both changes to the service description and functionality within the business components. It is very rare for these types of changes to be non-breaking. For example, if a response message from a service contains a total amount value that does not have the tax applied to it and after the change it does have a tax applied to it. The semantic meaning of that total amount value has changed.
· Policy changes. These changes include Quality of Service (QoS), Service Level Agreements (SLA), and similar nonfunctional requirements.
As you can see, some of these changes can be addressed with technology and design, but other changes require communication with team members. The following section explores both of these important scenarios in more depth.
Relationship Scenarios
The ability to apply an appropriate versioning strategy first requires a complete understanding of the scenario being addressed. The relationship scenarios describe two ways a service is coupled with its consumers: organizationally and technically. Resist any temptation you may have to gloss over the organizational section while you seek out the technical solution to these challenges. Change becomes less challenging with effective communication—frequently, people communicate better than software components.
Organizational Relationship
The organizational relationship scenario refers to the relationship between the team building the service and the team building the consumers of the service. This relationship is defined as the knowledge these teams have with each other, the level of influence these teams have with each other, and the effectiveness of the communication mechanisms used between them. These aspects are directly related to each other:
· The more knowledge the service team has of its consumers, the more able they are to increase the effectiveness of the communication with the consumer team.
· If the service team is communicating effectively with the consumer team, they are more likely to influence their decision making.
· When the service team can influence the consumer team, they are given more flexibility which, in turn reduces the complexity of the evolution.
There are a number of ways the service team can know more about the consumers of the service. The following relationship scenario descriptions are ordered from the best case to the worse case and how this relates to having knowledge of their service consumers:
· Single team. When the same set of people building the service is also building the consumers, the advantages are obvious; communication, planning, and decision making are dramatically improved. Teams in this scenario rarely experience the versioning challenges this guidance is attempting to address.
· Different team. It is common within an organization for one team to consume services developed by a different team, division, or business unit. Frequently, organizational policy can dictate who can consume certain services because service level agreements (SLA) and budget allocation transfers may be involved. In these cases, services gain knowledge of the consumers through a business justification agreement or similar means.
· Partners. Services consumed across organizational boundaries are almost always available only to consumers that the service knows about. Frequently, this is because of the security and contractual constraints of the business relationship. However, this scenario differs from the previous one in terms of communication and influence characteristics because it spans organizations.
· Subscription. The majority of the publicly available services today fall into this scenario. The nature of the subscription may vary, but the characteristics are consistent. At a minimum, the service team should be able to associate an e-mail address to the consumer. For information about ways to help communication in this, and other scenarios, see A Versioning Strategy.
· Free public. These services represent the most challenging services to evolve because of the lack of any substantial relationship between the service and consumer. The only way to mitigate these challenges is through broad and public communication because anyone may be a consumer.
Real-world scenarios do not always match only one of the preceding scenarios because they may share characteristics from several different scenarios. However, the list does represent the spectrum of scenarios.
After the service has knowledge of the consumers, gaining flexibility comes only through communicating the right information to the consumers. Versioning is about change. Therefore, the service's change control policy is the most important thing to communicate to parties who might be affected by any changes—the consumers. At a minimum, the policy should answer the following questions:
· Are changes scheduled and, if so, what is the schedule?
· What are the notification mechanisms to learn about changes?
· Where will the test server be located and how long is the testing window?
· Where is the change log and what level of granularity is provided (such as service description, bug fixes, or all details)?
· If the old service is being deprecated, what is the timeframe?
It is also important to establish other expectations and points of view. The answers to the following questions should also be communicated to the consumers:
· What represents a major and minor change?
· What are the goals for forward and backward compatibility?
· What should the service description be used for and not used for?
· What are the expectations around breaking and non-breaking changes?
The way this information is most effectively communicated depends on the relationship, but the following list explores some of the options to consider:
· Periodic conference calls
· Shared or common Web site
· E-mail distribution list
· Really Simple Syndication (RSS) feed
· Universal Description, Discovery, and Integration (UDDI) or similar registry that provides query capabilities
· Software development kit (SDK), which provides information and tools for receiving notifications
· Custom SOAP header in a response message
By clearly communicating policies, opinions, and expectations regarding the versioning strategy, the team building the consumers will be more likely to give the team building the service the flexibility they need to isolate the associated complexities. While no technical strategy is able to compensate for effective communication between teams, there are some technical considerations to keep in mind.
Technical Relationship
The technical relationship scenario refers to the level of coupling between the actual service and the software client application that is consuming the service. In contrast to the organizational relationship, the goal with the technical relationship is to loosen the coupling as much as possible.
The following list explores some of the ways a service is coupled with its consumers and how certain decisions can impact complexity:
· Message compatibility. Services and their consumers exchange messages and, as a result, there are associated expectations (complexity). Managing this complexity requires the important expectations to be explicitly defined while relaxing the less important expectations. For guidelines about how to accomplish this, see A Versioning Strategy.
· Shared components. Some Web services require that certain components exist as part of the consumer for the service to be used. These components may perform validation or represent portions of the domain model. Whatever the reason, they usually increase the coupling between the service and consumer in a way that prevents the service from evolving independently of the consumers. An effective versioning strategy avoids this level of coupling.
· Endpoint address resolution. Before the consumer can send the service a message, it must know where to send it. This address may be hard-coded, in a configuration file, in an external registry, such as UDDI or LDAP/ADSI, or the consumer may only know the address to a message broker that will decide which service should receive the consumer's message. Providing the means for a consumer to learn about a new endpoint address at run time gives the service additional flexibility.
· Deployment. The impact of the previous three bullets ultimately comes down to the deployment decision. Are the service and consumers deployed in a synchronized or disjointed fashion?
There are a few ways a deployment can be synchronized. In the single team scenario or different team scenario, they can plan to deploy new consumers at the same time the new service is deployed. In the partners scenario or subscription scenario, they can plan and coordinate the deployments at the same time, but this requires a lot of communication and is difficult. In the case of a smart client application, ClickOnce can be used to ensure the most current version of the client application is being executed.
Disjointed deployments are more common in the partners scenario and subscription scenario and are the only option in the free public scenario. In these scenarios, it is important to reduce the amount of coupling between the service and consumers and increase the communication as much as possible.
Principles and Motivations
This section represents the principles and motivations that provide the backdrop for the strategies in the next section. In addition to the concepts and scenarios in the previous sections, this section will help you understand the reasoning that went into the strategies.
Favor Evolution over Creation
Changes result in either a service evolving or a new service being created. Minor versions represent an evolution, and major versions effectively represent new services because they explicitly prevent any form of compatibility. This level of distinction is important for clearly communicating expectations; it can also influence decision making for specific changes.
Creating new services is expensive because of the added responsibilities in the following list:
· Each new endpoint address must be managed, monitored, secured, and in many cases, reported upon.
· Each new endpoint address must be documented and clearly communicated so the consumers do not run the risk of being confused by multiple (seemingly similar) services.
· Each new service requires the deployment of consumers of that service.
There is little to no evidence that suggests that creating new services minimizes complexity for either the team building the service or the team building the consumers.
Unfortunately, there are times when a graceful evolution is not an option. The most common scenario that mandates the service undergo a major version change is a change in a business requirement. These changes can undermine any technical strategy you put in place. For example, adding a new field to a request message usually does not dictate a major change from a technical standpoint. However, if a change in a federal policy or specification your business must comply with enacts a change that you must now collect information the consumers of your service are not currently sending, this will results in a new service that requires the deployment of new consumers.
Use Artifacts Appropriately
In order for a service to be used, the team building the service provides various documentation artifacts to its consumers. Examples are the service description (WSDL), policy files, registry/metadata information, and various documents that describe the API and service interaction patterns. Some of these artifacts will be read by humans and others will be acted upon by tools such as Visual Studio. As part of the communication to the service's consumers, it is important to include how these artifacts should be used and what the expectations are surrounding their use.
The WSDL document potentially contains a lot of information about the service and is typically only read by tools instead of humans. Historically, the WSDL has been used for multiple purposes:
· Code generation. This code can generate service-related code, but most of the time it is used to create proxy code for the consuming application.
· Communicate service requirements. Depending on the actual requirements being communicated, this may be an appropriate use of the WSDL. For example, the shape of the messages, the names of the operations, and binding information is all appropriate because it can be adequately and consistently interpreted by tools.
· Enforce service requirements. This is an inappropriate use because it cannot adequately or consistently be accomplished by tools. WSDL does include XML Schema documents to define the types that comprise the messages, but very few tools support translating the validation rules in a meaningful way. Also, when it comes to the broader topic of message validation, XML Schema validation is not as robust as other approaches.
The WSDL document describes the shape of the service instead of the requirements of the service. Because WSDL is only effective as a code generation artifact, it should only be used as such. Likewise, it should only be used to generate specific code (such as consumer's proxy, a stub of the service, and associated message classes) and the expectations for this code should also be communicated. Other, external documents are more appropriate to communicate the requirements of a service.
A Versioning Strategy
Specific tasks must be performed during service development to reduce the complexity of evolving the service. These tasks and design decisions are defined in the following sections and will be augmented with scenarios in a future version of this topic.
XML Namespaces
XML namespaces qualify XML elements and attributes within a document to prevent them from being interpreted as different nodes with the same local name. Distinct namespaces can be applied to various parts of a Web service, but in most cases only two kinds of namespaces need to be defined: one for the service interface and one for the reusable types defined in the messages.
The namespace defined for the service can be used across all parts of the service (for example, bindings, messages, or interfaces) because these parts usually are not deployed and versioned independently of each other.
Messages are made up of primitive types (for example, strings and integers) and/or custom types. Custom types can be used for only a single operation or they may have been defined as part of an enterprise schema or industry specification to be reused across operations or services. Because reusable types are used with other services, they should have a namespace different from the service interface.
An identifier that represents the major version of the interface and reusable types should be reflected in the namespace. The month and year of the release date is commonly used for this identifier (for example, https://globalbank.com/PaymentServices/PaymentTransferService/2006/12).
Message Design
Messages refer to the payload of data and metadata transferred between a service and consumer. A message can be a request or response and contains primitive and complex types. For more information about messages, see Message Design.
Messages are a common element of services that evolve over time. How they are designed influences how easily they evolve and how useful they are in other contexts. The goal of message design, as it relates to versioning, is to be as explicit as possible for the service description without losing the ability to apply minor revisions in the future.
These goals can be achieved by making all the properties optional for the messages and the complex types the messages contain. To appreciate this design, it is helpful to compare it to other message designs that are used:
· Strings and XML. Some messages are defined as a single string and this string may or may not contain XML data. Other messages have one type that is an XML node or element. Both of these designs share common advantages and disadvantages. They are very easy to evolve because there are no expectations on the messages, but because they do not set any expectations, they are useless from a code generation perspective. As a result of this, if a developer-friendly API is desired for the consumer or service, much more work is required.
· Strong typing. Some messages explicitly define each property and specify whether that value is required. This design results in a robust API for the service and its consumers, but it reduces the flexibility in how the messages evolve.
There are other advantages to this message design. Members of messages and custom types are optional by default in ASMX and WCF. The serialization engine used by ASMX and WCF services also handles optional members correctly.
Message Validation
Message validation is a common requirement for many services. The WSDL document or related XML schemas being used to define messages should not be used to accomplish this validation for reasons defined earlier in Principles and Motivations. Instead, validation should use external documents. These documents may be XML schemas, regular expression statements, or some other document. The validation rules and the means to apply them should be defined in the service’s documentation.
Decoupling the validation requirements from the service description also allow each of these items to independently evolve. Changes to the validation requirements may result in a breaking or non-breaking change. The same is true about a change to the service description.
Applying Changes
The following tasks should be completed when evolving a service:
· Minor revisions:
o Do not change the XML namespace of the message, type, or service being changed.
o Do not change the endpoint address of the service.
o New consumers of this service may be deployed, but the existing consumers will continue to operate as they did before the change.
o Publish new documentation, metadata, and notifications about the change in accordance with the change control process that has been communicated.
· Major revisions:
o Change the XML namespace so it reflects the new release date.
o Deploy the new service to a test environment so consumers can be tested.
o Change the endpoint address of the service.
o Test and deploy the new consumers of the service.
o Deprecate the old service in a manner and on a timeline in accordance with the change control policy that has been communicated.
Consumer Constraints
Frequently, constraints are placed on a service by some of its consumers. These constraints reduce the flexibility the service has to evolve. In this scenario, the service may choose to evolve differently for different consumers. For this to happen, the consumer must send some form of information that distinguishes the consumer in the message. The service interface then uses this information to dispatch the message to the appropriate implementation.
Another option to increase flexibility involves associating a consumer with the specific operations that consumer will use. This association can take place during the subscription of a service and can be updated at any time. These associations will allow the service to know exactly what consumers will be affected by specific changes to the service.
Summary
Services and their consumers are coupled in two ways: the relationship between the teams building them, and the technology and contracts used by the implementation. The focus of this guidance is to tighten the coupling between the teams and organizations and to loosen the technology coupling between the service and consumer applications. The relationship between the teams will become stronger through the use of documentation that explicitly clarifies terms, policies, and expectations. The service will have more flexibility to evolve by providing the minimum amount of detail necessary for the service and consumers to successfully communicate without the unnecessary burden of the developers.
Comments
Anonymous
January 12, 2007
Via Don : the December 2006 release of the Web Service Software Factory is now available! This is a majorAnonymous
January 15, 2007
Don, you seem to advocate to have two sets of XML schemas, one for definition (linked to the WSDL) and one for validation (at run time). We are having some trouble here to really understand the trade-off between the complexity introduced by this duality (SOA life is already compled enough :-() and the real value brought by this. Thanks for your help jacques.talbot@teamlog.comAnonymous
January 25, 2007
What is a service after all ? I could not find a concrete definition of service. Many authors give differentAnonymous
February 10, 2007
Versioning Web Services Don Smith, PM of Web Service Software Factory P&P team has an interestingAnonymous
February 12, 2007
Don, you say "Enforce service requirements. This is an inappropriate use [of WSDL]" ... But isn't the WSDL the service contract definition, and the linch-pin of the service/client agreement? What you are saying amounts to "WSDL contracts are not enforceable" -- now I'm not a lawyer, but unenforceable contracts seem fundamentally flawed ... ?Anonymous
February 22, 2007
Don, Thanks for your insightful blog - especially on versioning strategy. I am from Axis/XML/XQuery open source world. When SOA concepts such as document/literal and transformation externalization are applied in context of an Enterprise Service Bus - then it is a total different ball game. Primarily - all request and response types are well defined and all service requests go through a distributed ESB. So versioning strategy will change drastically - it will be moved away from clients into ESB and Metadata Governance (schema, transfromations, wsdl, policy, etc.) scope. Does MS have anything similar? How does then service versioning strategy change in this case. Thanks, Radu.Anonymous
February 23, 2007
The comment has been removedAnonymous
April 01, 2007
Versioning Web Services Don Smith, PM of Web Service Software Factory P&P team has an interestingAnonymous
September 30, 2007
Versioning Web Services Don Smith, PM of Web Service Software Factory P&P team has an interesting