Windows Azure Bridges & Message Itineraries: an architectural insight

June 3, 2013 at 4:20 PM

In this second blog post on Windows Azure BizTalk Services, we dive a little bit deeper in a concept that is very important for our Codit Integration Cloud project.  All our message processing and integration capabilities is running over bridges. 

This post is not a step-by-step demo walkthrough (Richard Seroter just did a nice one on his blog).  This article intends however to describe some patterns and some architectural explanation.

 

 

Itinerary routing mechanism


A message itinerary is a definition of a flow (it was called the flow designer in the past).  Let’s take the following example of a more complex itinerary:

image

The arrows specify the path that a message can follow.  It is a visual flow representation that links different components together. 

 

Persistence of itineraries

An itinerary does not have any persistence points and executes 100% in memory.  (example: If an FTP destination in an itinerary is unavailable, then there are no retries or whatsoever.  the consumer will get a 500 exception in that case!)

First match unicast routing

There is no one-to-many routing in itineraries.  Routing always happens on the first-matching path!  At every connection point, the message checks the first connection (based on the Sequence Number).  If the condition on that link matches, the message will be routed over that connection and all the other connections are ignored. 

The sequence number gets defined automatically, based on the sequence of adding the connections.

If you want to change the Sequence Number, you need to select the bridge and then you can configure the routing table with sequences in the Route Ordering Table:

image

Implement multi-cast routing

Multi-cast routing is routing a message, so that one message can be routed to multiple (or none) destinations.  If this needs to be implemented, then it is needed to use a service bus Topic with multiple subscriptions.  However, there is a problem, because at this moment, there is no service bus messaging listener implemented yet (like an FTP listener, but on a service bus queue or subscription).

No matching subscription

When a message is not able to arrive on an actual destination node, then the client will get a 500 error from the bridge and in the tracking data, there will be an error with the following description: “No Filter matched for the message.”.

 

Different bridge types


After we discussed the message itinerary, we can have a look at the bridges in detail.

image

As you can see in the screenshot above, there are different types of bridges:

  • XML One-Way bridge: this is a bridge that processes incoming messages and passes them on to the next matching connection.
  • XML Request-Reply bridge: this is a bridge that processes incoming messages and their corresponding response message.
  • PassThrough bridge: this is a bridge that does not look at the message details and can parse every type of message.
  • EDI bridge: unfortunately, this bridge is not available to the Visual Studio designer and is shielded by the TPM portal.

I would have loved to see a PassThrough Request-Reply bridge and a customizable EDI bridge.  These are not part of this release, but I’m sure they will come in a future release.  (that is the advantage of a cloud service – it’s much easier to add functionality for Microsoft now)

 

 

The different stages of a bridge


A bridge (much like a BizTalk pipeline) has different stages.  Depending on the type of bridge, one or more stages can be missing for that bridge.  The first thing to know is that you can easily disable a stage in the property window.  (that is something you typically do when disabling schema validation, for example).  Below, you can see a screenshot of a Request-Reply bridge.

image

Message types

(only in XML bridges)

The first step in an XML bridge is the section where all accepted message types are being defined.  This is done by adding schemas from a project to the request or response (in a Request-Reply bridge) direction.

Important to know:

  • You can add multiple schemas to this section.  The schema that matches the incoming message will be chosen.
  • The schemas you add have to be part of the current project and can not be added from a referenced project.  This is something that hopefully will change shortly.  (we want to reuse schemas and not redefine them in every single project)

image

Decode stage

(only in One-Way XML bridge)

The decode stage will be used if the incoming message is a flat file schema.  Incoming messages will then be serialized to XML, using the flat file schema schema.  This schema contains all the annotations that the Flat File parser will use to create XML out of the incoming flat file message.

Important to know:

  • You can also add flat file schemas in a One-Way bridge (not in an Request-Reply bridge!)  This makes it difficult to provide flat file parsing as a service.
  • The flat file schemas (and the Flat File Schema editor extension) are exactly the same as in BizTalk Server.  This makes it very easy to move flat file parsing between BizTalk Server and Services.

Validate stage

(only in XML bridges)

The validate stage validates the message at that time against the schema that is assigned to it.  The validation can be easily disabled, by disabling this stage (Is Enabled property).

On the validate step, you can specify how warnings should be handled.

Enrich stage

(available in all bridges)

This stage allows to configure property promotion.  All properties can be defined here and can be extracted from the incoming message or the context of the bridge.  There are different locations where this stage gets executed (before / after mapping , request / reply direction)
To configure the stage, the property definitions window should be used.

image

Property values can be read from the following sources:

SOAP Action, MessageId, To, ReplyTo
HTTP Here the name of an incoming HTTP header can be extracted and promoted to the property
FTP FileName, ServerAddress, Folder
SFTP FileName, ServerAddress, Folder
System The following properties can be retrieved from the system context: RequestId (Guid), MessageReceivedTime (UTC time), SourceName, SourceType (Http, Ftp…)
XPath Here an xpath statement (of the long type) should be entered and the schema has to be selected.  If the incoming message matches that schema, the lookup value will be done and promoted.  If the Xpath is not found, the property does not get promoted.
Lookup Here, it is possible to do a SQL Azure database lookup and promote the result of the query to a configured property.  All the database configuration settings are stored in the LookupProviderConfigurations.xml file.
The following screenshot shows the configuration settings for this:
image

 

Important to know:

  • If a source property does not exist (for example: Ftp FileName in a Http bridge), there is no error raised, the property just does not get promoted (which is good)
  • With the Lookup configuration, it is not possible to provide multiple input fields to a table (as we do with our framework)

Transform stage

(only in XML bridges)

The transform stage allows to configure one or more mappings that will be executed against the incoming message.  The mappings and mapper will be discussed in a later post.

Important to know:

  • You can only select a mapping that matches with the schema, configured for that direction of the bridge.  In the receive side, only mappings with a source schema that matches the request schema will be enabled for selection.  In the reply side, only mappings with a destination schema that matches the response schema are enabled for selection.
  • You cannot select multiple mappings with the same source schema in one transformation selection.  In that case you get the compilation warning: “Two or more maps selected in the Request Transform stage have the same source message type.”.

image

Send reply stage

(only in XML Request-Reply bridges)

In the Send reply stage, it is possible to read message properties and write them to the specific destination protocol (like Soap or HTTP headers).  Message properties can be written to Http or Soap headers of the response.

Important to know:

  • Only properties that have been declared in the Enrich stages are visible here.  If you have promoted a property in a custom bridge component, it is not visible here.  To promoting custom bridge properties, it is needed to define them in an enrich stage first (and fill them with the Tracking Id, for example). 
  • When the destination of an itinerary is a Service Bus messaging entity, the promoted properties are automatically being written on the BrokeredMessage, there is no specific action needed to do this.

image

Encode stage

(only in XML One-Way bridges)

In the Encode stage, the flat file schema will be interpreted to convert the XML document to the flat file structure, if the message matches a Flat File schema.

 

Expose a pipeline as a service


You cannot deploy a Request-Reply bridge without connecting it to a destination endpoint.  (or you would get the compilation warning: “Bridge '<bridge>' needs to route to at least one connected entity.”).  That means you need to route it to a TwoWay destination endpoint (like a TwoWay External service endpoint). 

But when you just want to create a ‘pipeline as a service’, for example to provide transformation or property promotion as a service endpoint, you don’t want to route to an external service endpoint, because this would cause increased latency.

Luckily enough there is the possibility to use custom WCF bindings and update the configuration of the TwoWay endpoint to that custom binding.  And when you use the EchoMessageBinding that comes with the Samples in the SDK, the request message just gets returned as the response message to the bridge. 

How to configure the EchoMessageBinding

  • Add a TwoWay external service endpoint destination to the designer canvas.
  • Give it a friendly entity name (for example: LoopbackEP).
  • Add a reference to your project to the EchoMessageBinding

Update the LoopbackEP.config configuration file as the following:

<?xml version="1.0" encoding="utf-8" ?> 
<configuration> 
  <system.serviceModel> 
    <extensions> 
      <bindingElementExtensions> 
        <add name="EchoMessageBindingElement" type="EchoMessageBinding.EchoMessageTransportElement, EchoMessageBinding, Version=1.0.0.0, Culture=neutral, PublicKeyToken=d39b54ec599a4cea"/> 
      </bindingElementExtensions> 
    </extensions> 
    <bindings> 
      <customBinding> 
        <binding name="EchoMessageBindingElement" > 
          <EchoMessageBindingElement /> 
        </binding> 
      </customBinding> 
    </bindings> 
    <client> 
      <!-- The clear below makes sure any client endpoints defined in machine.config don't get added--> 
      <clear /> 
      <endpoint name="TwoWayExternalServiceEndpointReference1" address="http://TwoWayExternalServiceEndpointReference2" binding="basicHttpBinding" contract="System.ServiceModel.Routing.IRequestReplyRouter"/> 
      <endpoint name="LoopbackEP" address="http://EchoMessage" binding="customBinding" bindingConfiguration="EchoMessageBindingElement" 
        contract="System.ServiceModel.Routing.IRequestReplyRouter" /> 
    </client> 
  </system.serviceModel> 
</configuration>

Then make sure the destination is using the LoopbackEP endpoint and you’re all set.

 

Unfortunately, it is not possible to expose Flat File parsing as a service at this point in time, because there are no decode / encode stages in Request-Reply bridges.

Some things you might now know about bridges


You can have multiple bridges as source for an itinerary. 

Every bridge has a specific URI.  So this means that a Message Itinerary can have multiple public endpoint URI’s.

With the following example itinerary, there are 3 public HTTP endpoints that can be called separately from each other.

image

The following itinerary has multiple source bridges available.  Based on the source, other ‘pre processing’ logic can be defined.

image

You can have custom code in bridges

It is possible to configure custom components in bridges.  These are being described in a future blog post.  A component can be configured on a stage in a On Enter and On Exit event.

 

Sam Vanhoutte

Posted in: Azure | BizTalk | WABS

Tags:

Pingbacks and trackbacks (4)+

Add comment

  Country flag

biuquote
  • Comment
  • Preview
Loading