BizTalk Integration Development Architecture: Artifact Composition

Article
1/17/2024

Introduction

This is the second in series of articles, the first article is "BizTalk Integration Development Architecture".

Artifact composition describes several aspects of the BizTalk Integration architecture:

Naming conventions
Folder structure
Shared Artifacts

Naming conventions

Naming Conventions are placed in a separate document. Please, use these Naming conventions as source information.

Folder Structure

The folder structure copycats the artifact hierarchy. Because the namespaces also copycat the artifact hierarchy, the folder structure looks like the namespace structure.

The namespace is the better formal representation of the hierarchy, so I copy the folder structure from the namespaces not the opposite way.

The main rule is:
Folder structure and folder names should be a copy of a namespace or the namespace pieces [a copy of the artifacts saved into the folders].

Seems complex, but it is simpler if we use pictures to demonstrate this rule.

A name of the project could be just Schemas, but the full project namespace as a name is better. Here two rules are “fighting” for dominance.

One rule is: names of a project, a project assembly, a project namespace, and a project folder should be the same.

The second rule is: Folder name should be equal to the correspondent part of the namespace.

The first rule is a winner.

So we use the folders names: GLD, Samples, and Template for all projects which namespace started with “GLD.Samples.Template”. But for the Schema project folder we use “GLD.Samples.Template.Schemas” not “Schemas”.

If we group several projects inside single solution, we follow the same rule for the additional group folder names.

For example, we group the projects inside the Shared.Schemas solution by a system name, which is CRM or Internal or SAP or Shipping. So we create subfolders for each group:

Do not forget to use the same rule for the solution folders inside Visual Studio:

This rule is very important for the big BizTalk Integration. Imagine a hundred applications and a hundred Visual Studio solutions. You are going to investigate and fix an error. All you know is the assembly name, it is GLD.Samples.Shared.Classes.Shipping. If you are a developer, you instantly know that you have to open the GLD.Samples.Shared.Classes Visual Studio solution to look to the code. If you are an administrator, you instantly know that you have to check the GLD.Samples.Shared.Classes BizTalk application. If you the source code manager, you instantly know where this code is saved on TFS or Git.

If you have ever searched an assembly GLD.Samples.Shared.Classes.Shipping and finally find out code in the GLD.Samples.Sap project which was placed in the GLD.Shipping solution, you completely spoil your time. If you start to investigate the assembly relations in such mess, you are completely doomed.

There are several aspects. We are considering this question from different points of view: development, deployment, operations.

Too many shared artifacts results in too complex relations between assemblies, projects and applications. Artifacts should be shared only for a good reason.

The BizTalk Server controls the relations between assemblies in very strict rules. One of this rule is: If we want to redeploy a shared assembly, we have to previously undeploy all assemblies which reference our shared assembly. That rule is super important in the big BizTalk Integration.

Development usually pushes us to share artifacts to use more compact and organized code.

Deployment, from the other side, pushes us to denormalize code, because shared code complicates deployment and redeployment.

Please, consider both sides before starting to implement a shared component.

If we share, we reuse, which couples code. Changes in shared artifacts could trigger changes in the sharing artifacts.

If we don’t share, we broke relation between artifacts. Changes in isolated artifacts do not change anything outside those artifacts.

It is possible to share without creating relation between artifacts, if we share template, pattern, idea, not the code, not the artifact itself.

Schemas

As discussed a schema is a very special artifact. It plays the contract role in the BizTalk applications. Services in SOA architecture do not share code, they share contracts. For BizTalk development this means we share schemas between systems, we share schemas between applications.

The schemas are good candidates for sharing. Quite few schemas are designed not to be shared and used only for internal purposes.

So, if schemas belong to an external system are generated by an [adapter] wizard, they are/will be shared in the most cases. Place them in a Shared.Schemas application under a separate project.

If schemas are canonical schemas, place them in a Shared.Schemas application in the Canonical project group under a separate project.

If schemas are shared only inside single application, they could be placed in a separate project of the current solution. If all application artifacts are placed in one project, the schemas also could be placed in this project.

Let’s see a real-life example.

Shared Schemas Architecture Example

Application AtoB was created the first. It transfers data between systems A and B. Next was the AtoC application and it exchanges data between systems A and C. Both applications use the Sch_A schema. BizTalk uses schema not in the application namespace but in the global namespace. That means BizTalk cannot recognize a schema if it was deployed in several applications. A schema should be deployed only once, if we want to use this schema in the ports. This rule protects us from severe design errors but it also forces us to share schemas.

For our example that means, we cannot deploy Sch_A into both applications. The naïve approach is to reference AtoB.Process assembly from the AtoC.Process. I mentioned this placing Sch_B in the brackets on the picture.

So far so good.

Then we found out a bug in AtoB application, not in the Sch_A but in Orch_1. We fixed it and want to deploy a new AtoB.Process assembly. No so good, because first we have to undeploy AtoC application because it references the AtoB. That is not good. Now AtoC is related to any modifications in AtoB not only to the Sch_A.

Let’s fix it. We extracted schema Sch_A into a separate project/assembly. Now we don’t have to redeploy AtoC.Process when we change something in Sch_B or in Orch_1, but only if we change Sch_A.

But it still doesn’t look right. From the design standpoint the Sch_A does not belong to the AtoB application. It does not belong to the AtoC application either. Both applications us it, but it is an independent of them. Only system A dictates how the Sch_A schema looks like. Owners of this system can change it, not the owners/developers of the AtoB and AtoC applications.

Let’s change our design to show this ownership. A new Shared.Schemas application was created, which holds all schemas that belong to the integrated systems. Our Sch_A was placed into a separate, independent projects. Now AtoB and AtoC both reference this Shared.Schemas.A assembly.

Isn’t is something wrong? How do our changes simplify our development, deployment, operations? Looks like now we have one more application and one more reference (from AtoB to the Shared.Schemas.A). How it could be simpler?

The key word here is “changes”. If our applications will never be changed we don’t need these “improvements”. Actually we don’t need AtoB and AtoC but just one application holding all artifact.

But when we start to modify our applications we immediately start to understand that “loose coupling” is not just funny words.

Next step is to ask a question “Why is the Sch_A schema so special and not the Sch_C and Sch_B schemas”? All those schemas belongs to the integrated systems not to the integrating applications.

Let’s change our design to fix this. Now schemas of all systems are placed into separate assemblies. Moreover, we found out these systems have (or could have) several interfaces, not only a single interface, hence we got several assemblies for each interface. I use an interface term here as a separate data contract/schema. For sure, if it is a two-way interface (as request-response), both the request and response schemas belong to the same assembly.

Again, the resulted design looks more complex but it is more appropriate for the real life. And it models the real relations.

Next step in our design is to mention that we integrated two systems, B and C, with one system A. What happens if we add one more system or replace one of the system? Seems the canonical data model fits here perfectly.

Let’s create a canonical schema for this interface (I_1 interface) and link the systems through this canonical schema. Now each application deals exactly with one system. Each application deals with implementation details of this single system. Changes in this system will not force us to modify another applications. Before that if system A does change its interface, we have to modify both AtoB and AtoC applications. Now, we change only X_to_Canonical1. I intentionally changed the A application name to the X, to show, that we can easily add Y, Z, etc. systems to the picture without changing any other application.

Canonical data model is not a universal, required design pattern. It works if we have one-to-many or many-to-many integration interfaces between systems. It doesn’t make sense to use it if we have one-to-one interface.

Here is a Shared.Schemas Application Template which can save us some precious development time.

.NET classes

.NET classes can be shared between the BizTalk applications. It doesn’t create the “redeployment hell” as it could happen with sharing other BizTalk artifacts: schemas, maps, orchestrations, pipelines, etc.

If we need to modify a .NET assembly, we just re-GAC it. This does not require to undeploy all related assemblies.

One recommendation: Do not share .NET class right away. Try it in one application as a local class for this application. Try it in the next application. When this class is stabilized, when you feel it will not be changed too much in the future, extract it in a shared project. Usually those shared assemblies are placed inside one Shared.Classes application.

Here is a Shared.Classes Application Template which can save us some precious development time.

Maps, Orchestrations, Pipelines, etc.

Other BizTalk artifacts, not the schemas, are not good for sharing.

We share the pipelines, maps, orchestrations, rules only in very special occasions, usually as the components of the BizTalk shared infrastructure. One example is a notification service that manages notifications: formats notifications, filters notifications, sents them as emails, SMS-s, twitts, etc. Always consider to share artifacts as a service.

Maps are usually totally local to the application. It is a really bad thing to share maps between applications from the design point of view.

If you want to share the orchestration, do not share the orchestration assembly between applications. Share it as a service.

If you call or start an orchestration from another application, you have to reference the orchestration assembly. So consider to use another architecture pattern, like direct binding. That means the calling orchestration just publishes (sends) messages and the called orchestrations subscribe to these messages. If you want to pass the additional parameters with message use the message context properties. The message context also can be used to create the “custom binding” when subscriptions have additional predicates which match for example the originator orchestration name.

The pipelines also should not be shared between applications. But the pipeline components can be shared.

Share via

BizTalk Integration Development Architecture: Artifact Composition

Introduction

Naming conventions