Microsoft BizTalk Server 2010 ships with some assemblies that assist you with the administration and deployment of your BizTalk environment.  During the development of an automated deployment plan, we were able to perform most of the deployment tasks at a BizTalk application level, by using these assemblies:

Microsoft.BizTalk.ExplorerOM

-       Create/delete applications

-       Control receive locations

-       Control orchestrations

-       Control send ports

-       Control send port groups

-       Control application references

Microsoft.BizTalk.ApplicationDeployment

-       Add/remove resources

Microsoft.BizTalk.Deployment

-       Import/export bindings

Microsoft.BizTalk.Operations

-       Query for active service instances         

Remark that all of the used objects can be initialized by three parameters:

      -    SqlServerName

-    BizTalkMgmtDbName

-    ApplicationName

The object "BizTalk Group" is defined by the SqlServerName and the BizTalkMgmtDbName.  A "BizTalk Application" is defined by its BizTalk Group and the ApplicationName.  However, these assemblies have totally no notion of the object "BizTalk Server".  They only support operations at a database level (BizTalk Group), but some deployment tasks need to be executed on a BizTalk Server level:

- Install/uninstall assemblies to the Global Assembly Cache

- Control host instances

For both actions we needed to be able to dynamically determine which BizTalk Servers are part of the already defined BizTalk Group. After some research, we found two options to implement this logic:

Windows Management Instrumentation

WMI has some support for BizTalk via the WMI namespace “root\\MicrosoftBizTalkServer”.  We are able to loop though all configured host instances, by executing a query on the MSBTS_HostInstance class.  As each host instance runs on a BizTalk Server, it’s easy to implement some logic that retrieves all BizTalk Servers.

private List<string> GetBizTalkServers()
{
     List<string> btsServers = new List<string>();

     EnumerationOptions wmiEnumerationOptions = new EnumerationOptions { ReturnImmediately = false };
     ObjectQuery wmiQuery = new ObjectQuery("SELECT * FROM MSBTS_HostInstance");
     using (ManagementObjectSearcher wmiSearcher = new ManagementObjectSearcher("root\\MicrosoftBizTalkServer", wmiQuery.QueryString, wmiEnumerationOptions))
     {
          ManagementObjectCollection hostInstanceCollection = wmiSearcher.Get();
          foreach (ManagementObject hostInstance in hostInstanceCollection.Cast<ManagementObject>())
          {
                string btsServer = hostInstance["Name"].ToString().Split(' ').Last();
                if (btsServers.Contains(btsServer) == false)
                {
                     btsServers.Add(btsServer);
                }
          }
      } 

      return btsServers;
}

Custom SQL Server Query

Another option is to retrieve the information directly from the BizTalkMgmtDb.  The table adm_Server actually contains a list of all BizTalk Servers that are part of the BizTalk Group.  So this simple .NET code is able to return us the needed information:

private List<string> GetBizTalkServers()
{
     List<string> btsServers = new List<string>();

     SqlConnection sqlConnection = new SqlConnection(String.Format(CultureInfo.CurrentCulture, "Server={0};Database={1};Integrated Security=SSPI;", SqlServerName, MgmtDatabase));
     SqlCommand sqlCommand = new SqlCommand("SELECT Name FROM [dbo].[adm_Server]", sqlConnection);

     sqlConnection.Open();

     SqlDataReader sqlDataReader = sqlCommand.ExecuteReader();

     while (sqlDataReader.Read())
     {
          btsServers.Add(sqlDataReader["Name"].ToString());
     }

     sqlConnection.Close();

     return btsServers;
}

Conclusion

It's pretty easy to retrieve a list of all BizTalk Servers that are part of a BizTalk Group.  A limitation of the WMI approach is that this will only work when it's executed on one of the BizTalk Servers.  So I prefer the SQL approach, because this will also work for remote execution.

Toon Vanhoutte

Recently, I had a project where full side-by-side deployment was required.  In a first test phase of the automated deployment, we used MSBuild together with the Codeplex MSBuild Extension Pack. We tried deploying assemblies side-by-side, but BizTalk threw errors when trying to deploy a new version of our Blog.Codit.BizTalk.Pipelines assembly:

Failed to add resource(s). Change requests failed for some resources. BizTalkAssemblyResourceManager failed to complete end type change request. SQL exception: "Violation of UNIQUE KEY constraint 'UQ__bts_pipe__DD425CAE7908F585'. Cannot insert duplicate key in object 'dbo.bts_pipeline'. The duplicate key value is (D6C53CD9-8466-4520-A85B-7BD887A57455)." 

I received this error when trying to deploy from Visual Studio, from MSBuild (using BTSTask.exe) and also when trying to deploy it manually via the Admin Console (Add Resource).  So the error is actually independent from the deployment source.  As BizTalk returned a SQL exception, I decided to run SQL Profiler and detected this call:

exec btm_CreatePipeline 
   @PipelineID='D6C53CD9-8466-4520-A85B-7BD887A57455',
   @Category=1,
   @Name=N'Blog.Codit.BizTalk.Pipelines.rcvSalesOrder',
   @FullyQualifiedName=N'Blog.Codit.BizTalk.Pipelines.rcvSalesOrder, Blog.Codit.BizTalk.Pipelines, Version=1.2.0.0, Culture=neutral, PublicKeyToken=xxx',
   @IsStreaming=0,
   @AssemblyID=265,
   @Release=2,
   @ID=@p8output

So the deployment logic tries to create a new entry in the bts_pipeline table, but it uses exactly the same (unique) ID of the pipeline of the older version.  This entry was already present in the bts_pipeline table: 

D6C53CD9-8466-4520-A85B-7BD887A57455 Blog.Codit.BizTalk.Pipelines.rcvSalesOrder, Blog.Codit.BizTalk.Pipelines, Version=1.1.0.0.  

We opened a case at Microsoft Support for this issue.  They forwarded the issue to the BizTalk Product Team and we received this answer:

The pipeline GUID is stored in the rcvSalesOrder.btp.cs  file. The GUID gets regenerated when either of the following occurs:
 
- The pipeline is saved from within Visual Studio.
- When REBUILD the BizTalk project.

As we executed a BUILD, instead of a REBUILD, a small change to our MSBuild script solved the issue:

<MSBuild Projects="@(ProjectsToBuild)" Targets="Rebuild" />

This behavior is rather inconsistent, because it only occurs when your assembly contains a pipeline object. 

Big thanks to Microsoft for the fast and accurate support.

Toon Vanhoutte

This blog post contains some guidelines for consuming web services, with a focus on exception handling.  We will both handle pure messaging scenario and web service calls from within an orchestration.  I also make a distinguishment between asynchronous and synchronous flows.  Synchronous means that the flow needs to return a message to the original caller.  So in case of troubles, a SoapFault needs to be returned to the calling application.  In an asynchronous flow, you just need to make sure that your web service call is resumable. 

Messaging Scenario – Asynchronous

When an error occurs during the web service call, the default behavior of BizTalk retry 3 times and will suspend the request message on the BizTalk Send Port and this instance will be resumable.  This is the perfect behavior for this asynchronous process.

Messaging Scenario - Synchronous

If you are in a synchronous process, a suspended Send Port will lead to a timeout on your Receive Port.  In this scenario you need to enable "Propagate Fault Message" on the Send Port and a SoapFault will be returned to the original caller.  Also think about the Retry Count and Retry Interval settings, because a bad configuration can also lead to timeouts.

Orchestration - Asynchronous

When you consume a web service from within an orchestration, it's getting more complex.  If the consumed web service throws an exception, BizTalk creates by default 2 suspended service instances.  Resuming them does not always have the expected behavior.

-  The Orchestration is suspended.  A resume will not change anything to the state of the Orchestration.
-  The Send Port is suspended.  A resume will resume the web service call as expected, but the Orchestration will not be able to continue processing afterwards.

How can we now achieve a resumable web service call?

First of all enable "Propagate Fault Message" on the Send Port.

Also modify the calling orchestration:

-  You need to create a loop.  This loop continues as long as the Boolean bSuccess equals false.

-  Modify the logical Orchestration Send Port and add a Fault Message.  To keep it general, you can choose the SoapFault (which is part of the Microsoft.BizTalk.GlobalPropertySchemas).

-  Add an exception handler to the web service call. Catch the Fault Message that was returned.  Add a suspend shape for human intervention. Note that it’s a good practice to also add an extra exception handler which catches a System.Exception.

-  If the web service call fails: set bSuccess == false in the exception handler.  If the call succeeds, set bSuccess == true, so the orchestration will exit the loop.

The result should look like this:

Now we've created a resumable web service call from within an orchestration.  When testing the orchestration in a failure scenario, you will see that there are still two suspended service instances:

-  The Orchestration is suspended.  This is the one that we've made resumable.

-  The Send Port is suspended.  To avoid this second service instance to be suspended, you can enable "Routing for Failed Messages".  This will generate an ErrorReport, instead of suspending the Send Port.  Subscribe on the ErrorReport with the filter ErrorReport.ErrorType == “FailedMessage”.

Orchestration - Synchronous

In this scenario, you will also need to catch the SoapFault (or other exception) and make sure that you don’t end up with suspended send port instances. See “Orchestration – Asynchronous”

The extra functionality that you’ll need to add is returning a SoapFault to the original caller.

- Add a Fault Message to the Request/Response receive port of your orchestration.

- You can choose to return a typed Fault or, to keep it general, a SoapFault (which is part of the Microsoft.BizTalk.GlobalPropertySchemas).

- Make sure you return the catched SoapFault (or the created SoapFault in case of other exception) in case of errors.

- Remark that you’ll need to add a decide shape to determine you’re in a success/failure scenario. Otherwise, the compiler will fail (complaining about unconstructed messages).

Again, think about the Retry Count and Retry Interval settings on the send port.

Toon Vanhoutte

Recently, we've encountered some problems with SCOM 2007 R2 monitoring of a BizTalk 2010 environment. The SQL Agent Job "Backup BizTalk Server (BizTalkMgmtDb)" failed, but we didn’t receive any SCOM alert about that.  The first reflex was to look at the BizTalk Management Pack.

BizTalk Server Management Pack

The BizTalk Server Management Pack contains a rule, called “CRITICAL ERROR: A SQL Server agent job failed - Backup BizTalk Server”, which is responsible for alerting us in case of troubles.  The rule is disabled by default, but we had already enabled the rule with an override.

Why didn’t we receive an alert?  We had a closer look at the default rule and discovered that this is the way of identifying the backup job failure:

-    It’s a rule that subscribes on the event log [TypeID="Windows!Microsoft.Windows.EventProvider"]
-    It looks for event log entries with ID = 208 [EventDisplayNumber equals 208] in the Application log
-    The computer name is set to $Target/Property[Type="Microsoft.BizTalk.Server.2010.ServerRole"]/ComputerName$

When the error occurred, we indeed discovered event log entries with ID 208 on the SQL server.  But this rule is subscribing on the event log of the BizTalk server.  Conclusion: the default rule will only work on a single box installation (SQL server and BizTalk server have the same event log).

To solve this issue, we’ve created a custom rule, based on the default one.  The custom rule had exactly the same configuration, except for the computer name that we’ve changed to:
-    “$Target/Property[Type="Microsoft.BizTalk.Server.2010.BizTalkGroup"]/MgmtDbServerName$”

Now the rule was subscribing on the event log of the SQL Server (running the ManagementDb), and we received an alert when the backup job failed.  This worked fine, until we deployed this rule against a multi-server BizTalk environment, containing a SQL Server cluster.  Suddenly we’ve received SCOM errors, complaining that the SCOM Agent couldn’t access the event log of the MgmtDbServer.  The cause of this is that the BizTalk Management Pack actually discovers the virtual SQL Server Cluster name.  As this is not a physical server name, it’s normal that the SCOM Agent can’t access the event log.

To overcome this issue, we could have created a new custom rule, which uses the discovered MgmtDbServer and performs a query on the [msdb].[dbo].[sysjobhistory] system table.  The sysjob tables contain all needed information to detect SQL Agent Job failures.  As this is too complex, we decided to have a look at the SQL Server Management Pack.

SQL Server Management Pack

As it is actually a SQL process that fails, the SQL Server Management Pack should be responsible for alerting us.  But why didn’t that happen?  There are actually two reasons for this:

-    The discovery of SQL Agent Jobs is disabled by default.
-    The alerting for SQL Agent Job Failures is disabled by default.

We enabled the discovery and alerting, tested the solution and everything went smooth.

Conclusion

SQL Agent Jobs should be monitored by the SQL Server Management Pack.  This Management Pack should always be installed in a BizTalk environment, as SQL Server is the core of BizTalk.  Please, keep in mind that the alerting of SQL Agent Jobs is disabled by default.

It’s actually a strange approach of Microsoft to try to include the monitoring of the SQL Agent Jobs in the BizTalk Server Management Pack.  Certainly because the default implementation only works for a single box installation.  On MSDN you can find a vague description on this subject.

Toon Vanhoutte

Host Integration Server 2010 comes with a BizTalk data conversion pipeline component.  This enables the copybook parsing functionality, withou the usage of the HostApps adapters. Great stuff when you need to parse copybook files while using a custom WCF binding for the mainframe connectivity.

During a POC, we were glad to discover that the import of a copybook definition goes a lot smoother than in the previous HIS versions.  The copybook definition import results in a HIS data conversion assembly.  This assembly contains the logic to parse the copybook file to XML and vice versa.  We configured the by default installed HIS send pipeline, “DataConversionSendPipeline”, with the generated HIS data conversion assembly and everything went fine. 

Afterwards, we’ve changed the POC scenario and needed to add some extra functionality to the send pipeline.  Therefore we’ve created a custom send pipeline and configured it with the HIS data conversion pipeline component.  This custom pipeline had actually the same setup and configuration of the by default installed HIS send pipeline, “DataConversionSendPipeline”.  However we’ve encountered suddenly a bunch of conversion errors. 

After many hours of troubleshooting, Sam discovered the following pipeline name dependency in the data conversion pipeline component via Reflector:

if (pc.PipelineName.IndexOf("DataConversionSendPipeline") != -1)

{ this._toHost = true; }

else

{ this._toHost = false; }

The _toHost property is actually used to determine if the component needs to execute receive pipeline conversion functionality (FF to XML) or send pipeline conversion logic (XML to FF).  The value of this property is set, depending on the name of the send pipeline it belongs to.  A dependency between a pipeline component and a pipeline name is NOT a good practice.  So if you want the component to work correctly in a custom send pipeline, the pipeline name should contain “DataConversionSendPipeline”.  It would be more straightforward to set the _toHost property in the component, depending on the determined pipeline stage at runtime. 

We’ve raised this bug via Microsoft Support and we hope to see a fix in the next cumulative update pack.

In some cases, other applications want to make use of the flat file parsing functionality of BizTalk. These applications need for example a conversion between their .NET object and a flat file, or vice versa. The functional context can be EDI, HL7, Copybook or custom flat file parsing.

The most performant way to expose this BizTalk pipeline functionality as a two-way WCF service, is by routing the transformed request message automatically back to the send pipeline of the two-way receive port. This pure messaging scenario only involves one pub/sub mechanism to the MsgBox.  We have tested this on a single box development machine, arriving at only 60ms roundtrips for our flat file test schema.

This can be achieved by adding a custom pipeline component at the end of the receive pipeline. This pipeline component promotes the necessary properties to route back the request directly to the subscribing send pipeline. The Execute method of this component is where the magic happens:

string ns = http://schemas.microsoft.com/BizTalk/2003/system-properties;

inmsg.Context.Promote("RouteDirectToTP", ns, inmsg.Context.Read("IsRequestResponse", ns));
inmsg.Context.Promote("EpmRRCorrelationToken", ns, inmsg.Context.Read("EpmRRCorrelationToken", ns));

return inmsg;

This logic can also be used in custom disassembler pipeline components, which need to generate a response message that needs to be routed back to the original caller.

Toon Vanhoutte