integrating your business

A common requirement in many development scenarios is caching.  In BizTalk implementations, this can be the case, mainly for performance reasons.

As a test I wrote 2 pipelinecomponents that handle 2 common issues with caching inside BizTalk.  These sample components are performing similar tasks to two components of the Codit implementation framework.  This is a framework we use at a lot of our customers.

1. Code table mappings : limit access to SQL database to perform Code table mappings.  In our framework, this is similar to the Transco component.
2. Duplicate message : stop messages that come into BizTalk multiple times within a specific timeframe.  In our framework, this is similar to the Double checker component.

CacheHelper

Because both pipelinecomponents use the AppFabric cache, I wrote a small class that takes care of this.

public class CacheHelper : IDisposable
    {
        private string _cacheName = "default";
        private string _region;

        public CacheHelper(string region)
        {
            _region = region;
            CreateRegion(_cacheName,region);
        }

        /// 
        /// Creates a Region in a specified cache.
        /// 
        /// Cache name
        /// Region name
        private void CreateRegion(string cacheName,string region)
        {
            DataCacheFactory dcf=ConnectToCache();

            if (dcf != null)
            {
                DataCache dc=dcf.GetCache(cacheName);
                dc.CreateRegion(region);
            }
        }

        /// 
        /// Connect to a Cache server
        /// 
        /// The Datacache
        private DataCacheFactory ConnectToCache()
        {
            //This can also be kept in a config file
            var config = new DataCacheFactoryConfiguration();
            config.SecurityProperties = new DataCacheSecurity(DataCacheSecurityMode.None, DataCacheProtectionLevel.None);
            config.Servers = new List
            {
                new DataCacheServerEndpoint(Environment.MachineName, 22233)
            };

            return new DataCacheFactory(config);
        }

        ~CacheHelper()
        {
            Dispose(false);
        }

        public void Dispose()
        {
            Dispose(true);
            GC.SuppressFinalize(this);
        }

        protected virtual void Dispose(bool disposing)
        {
        }

        /// 
        /// Gets a value from the cache in the specified region (class level)
        /// 
        /// Key linked to the data
        /// The found data. If null --> not found in the cache
        public string GetLookUpCacheData(string keyValue)
        {
            DataCacheFactory dcf = ConnectToCache();

            var cache = dcf.GetCache(_cacheName);
            string data = cache.Get(keyValue,_region) as string;

            return data;
        }

        /// 
        /// Store a value in the cache
        /// 
        /// Key
        /// Data
        public void StoreLookUpCacheData(string keyValue, object value)
        {
            DataCacheFactory dcf = ConnectToCache();

            var cache = dcf.GetCache(_cacheName);
            cache.Add(keyValue, value, _region);
        }

        /// 
        /// Stores a value in the cache for a specified amount of time
        /// 
        /// Key
        /// Data
        /// Time to keep the data in the cache
        public void StoreLookUpCacheData(string keyValue, object value,TimeSpan expires)
        {
            DataCacheFactory dcf = ConnectToCache();

            var cache = dcf.GetCache(_cacheName);
            cache.Add(keyValue, value, expires, _region);
        }        
    }

This is a very simple implementation that will store values in the default cache and in a specified region.

CodeTable Mapper

Codetable mapping is a very common scenario in BizTalk implementations. In my example we will be translating countrycodes to the country name.
The values are stored in a SQL table. But every time we get a value, we are going to save it to the AppFabric cache.
When we want to get the same value again, we are not going to the database but we will get the stored value from the AppFabric Cache.

public IBaseMessage Execute(IPipelineContext pContext, IBaseMessage pInMsg)
        {
            // Set variables
            biztalkMessage = pInMsg;
            XmlReader reader = XmlReader.Create(pInMsg.BodyPart.Data);
            XPathCollection xpaths = new XPathCollection();
            //For this example we are going to use 1 xpath expression
            xpaths.Add(this.XPath);

            ValueMutator vm = new ValueMutator(handleXpathFound);
            pInMsg.BodyPart.Data = new XPathMutatorStream(reader, xpaths, vm);
            return pInMsg;
        }

        private void handleXpathFound(int matchIdx, XPathExpression matchExpr, string origVal, ref string finalVal)
        {
            CacheHelper ch = new CacheHelper("Countries");
            string data = ch.GetLookUpCacheData(origVal);
            if (data == null)
            {
                finalVal = GetCountryFromDB(origVal);
                ch.StoreLookUpCacheData(origVal, finalVal);
            }
            else
                finalVal = ch.GetLookUpCacheData(origVal);
        }

        private string GetCountryFromDB(string countryCode)
        {
            string country = string.Empty;
            SqlConnection conn = null;
            SqlCommand comm = null;

            try
            {
                //Connect to look up database and retrieve the names of the products.
                conn = new SqlConnection("Data Source=(local);Initial Catalog=CacheTest;Integrated Security=SSPI;");
                conn.Open();

                comm = new SqlCommand();
                comm.Connection = conn;
                comm.CommandText = string.Format("SELECT Country FROM Countries WHERE CountryCode='{0}'", countryCode);
                comm.CommandType = CommandType.Text;

                country = (string)comm.ExecuteScalar();
                if(string.IsNullOrEmpty(country))
                    throw new Exception(string.Format("No country found for code {0}",countryCode));
            }
            catch (Exception e)
            {
                throw new Exception(e.Message + e.StackTrace);
            }
            finally
            {
                comm.Dispose();
                conn.Close();
                conn.Dispose();
            }


            return country;
        }

Duplicate Message Checker

As a sample scenario I took one I read about a few months ago. BizTalk had to stop messages that come in multiple times within 2 minutes.
So if there are more then 2 minutes between the messages, they should continue.

Normally this would involve a SQL table to store some information and some job to do the cleanup of this table.
But for my example I use AppFabric cache. There you have the option to store something in the cache for a certain timespan.
It is automatically deleted after this period.

public Microsoft.BizTalk.Message.Interop.IBaseMessage Execute(IPipelineContext pContext, Microsoft.BizTalk.Message.Interop.IBaseMessage pInMsg)
        {
            //Create hash
            VirtualStream input = new VirtualStream(pInMsg.BodyPart.GetOriginalDataStream());
            MD5 md5 = MD5.Create();
            byte[] hash = md5.ComputeHash(input);
            string hashString = Convert.ToBase64String(hash);

            //check Cache
            CacheHelper ch = new CacheHelper("DuplicateMessages");
            string date=ch.GetLookUpCacheData(hashString);
            if (string.IsNullOrEmpty(date))
            {
                //If not in cache yet, store it --> lifetime is 2 minutes
                ch.StoreLookUpCacheData(hashString, DateTime.Now.ToString(), new TimeSpan(0, 2, 0));
            }
            else
            {
                //Throw error
                throw new ApplicationException(string.Format("Duplicate Message. Already received at {0}",date));
            }

            //Put stream back to beginning
            input.Seek(0, SeekOrigin.Begin);
            return pInMsg;
        }

This makes the implementation very easy and you will not need a SQL table or anything to store the information.
You could say that you can do this with a custom caching solution as well. But what about HA environments with multiple BizTalk servers?
AppFabric is a distributed cache. So it doesn't mather on which server the message is processed. It will end up in the same cache and will be accessible on all the servers.

Conclusion

As you see, AppFabric caching has some advantages in BizTalk as well. The API is very easy to use and I got this to work quite quickly.

Tim D'haeyer, CODit

A few weeks ago I had to create a flow that receives messages from SAP with a specific IDoc release number. The SAP system was a 620 release and the IDoc was 46C release.
So to start with, I generated the schema for the 46C release and deployed that on my BizTalk box.

When we sent an IDoc to BizTalk, we got following error :

The segment or group definition E2EDP01006 was not found in the IDoc metadata. The UniqueId of the IDoc type is: IDOCTYP/3/ORDERS05//620. For Receive operations, the SAP adapter does not support unreleased segments.

So this message stated that SAP was sending a 620 IDoc. After a discussion with an SAP developer, we found out that the DOCREL field in the EDI_DC40 header had the value 620 while we where expecting to receive 46C.

The way that the new SAP adapter works in BizTalk is that when he receives an IDoc from SAP, he will check all the segments that are sent to see if it is a valid IDoc. Since the segment E2EDP01006 is not present in IDoc release 620, this is a normal error. This validation occurs in the adapter, so even before the message reaches BizTalk. So even if you have the correct schema deployed (like in my case) you will still get the error.

While searching for a solution we found 3 ways to solve this.

1.  Always use the latest IDoc release

This is the most obvious one off course. Just ask an SAP developer to send the latest release of the IDoc to BizTalk and you will be able to receive it. (The latest release will be the same as the system release)

This is also the option we choose at the time. It was the fastest solution we could both support.
But we still spent a little bit time to investigate the other options.

2.    Receive “untyped” flat file IDocs.

This means that you need to generate the flat file schemas for the IDoc instead of the typed xml schema. You have to make sure that you set the GenerateFlatfileCompatibleIdocSchema field to true in the “Consume adapter service” window.(If you want to receive multiple IDocs via the same receive port, make sure you install the KB977528 hotfix)

This has the limitation that you need to use the flat file assembler in your receive pipeline. If you want to receive alot of different types of IDocs with the same receive pipeline, you will have to put a flat file disassembler in there for every IDoc type.

3.    Change the SAP output module

SAP has 2 ways of exchaning IDocs with other systems.

1. Application Link Enabling (ALE)

ALE was created to exhange IDocs between SAP R/3 systems and/or R/3 and external systems.

1. Electronic Data Interchange (EDI)

This interface is used to communicate with non R/3 systems.

Since BizTalk can act as an R/3 system, the standard way we received IDocs was via ALE. But when you use ALE, the DOCREL field will always contain the system release number.So if you use ALE as the output module, you can only receive the latest IDoc release know inside the system.The EDI interface works in a different way, that makes sure that the DOCREL field is filled in correctly. Since we already chose to go for the first solution, we did not test this at runtime.
But The SAP developer told us that this should do the trick as well.

Tim D'haeyer, CODit

Yesterday we discovered something about the use of custom extension objects in custom XSLT.
Imagine that you need a counter value in your mapping. Since variables in XSLT can only be set once, you need to use custom code for this.
In our case we wrote a small helper class that has a counter variable.

public class Counter
{
    private int _counter = 0;

    public int GetCounter()
    {
       return ++_counter;
    }
}

I just called this in my map and tested it. It worked perfectly.

But then I fired a few concurrent calls to our BizTalk process, resulting in the map being executed multiple times in a very small time frame.
We noticed that the counter value wasn't correct anymore. After some analisys it seemed like the instance of the Counter class was shared over multiple executions of the map.
Our assumption that BizTalk creates a new instance of the class every time the map is called, was wrong.

We came up with a quite simple solution for this. Instead of just having 1 counter, we created a Dictionary that can hold multiple counter values.
The key for this Dictionary is a Guid (as string) and the value is the actual counter. Each map has it's own Guid that makes it unique.

public class Counter
    {
        private Dictionary _counter;

        public Counter()
        {
            _counter=new Dictionary();
        }

        public string NewGuid()
        {
            return Guid.NewGuid().ToString();
        }

        public int GetCounter(string guid)
        {
            return ++_counter[guid];
        }
    }

So this is how our class looks like with this Dictionary. At the top of your map you just create a variable that holds the Guid, and you use it every time you call the GetCounter method.

<xsl:variable name="Guid" select="Counter:CreateGuid()"/>

<xsl:value-of select="Counter:GetCounter($Guid)"/>

 We are still not quite sure how long BizTalk keeps this in memory. But to be sure we added a method to the class that deletes one row in the Dictionary at the end of a map.

 Tim D'haeyer, CODit