Great news for the Azure community.  The Windows Azure SDK and Windows Azure Tools for Microsoft Visual Studio (November 2010) have been released!  This release contains a lot of new features and most of them were announced at PDC.

The release can be downloaded here.

The overview of new features:

• Virtual Machine (VM) Role (Beta):Allows you to create a custom VHD image using Windows Server 2008 R2 and host it in the cloud.
• Remote Desktop Access: Enables connecting to individual service instances using a Remote Desktop client.
• Full IIS Support in a Web role: Enables hosting Windows Azure web roles in a IIS hosting environment.
• Elevated Privileges: Enables performing tasks with elevated privileges within a service instance.
• Virtual Network (CTP): Enables support for Windows Azure Connect, which provides IP-level connectivity between on-premises and Windows Azure resources.
• Diagnostics: Enhancements to Windows Azure Diagnostics enable collection of diagnostics data in more error conditions.
• Networking Enhancements: Enables roles to restrict inter-role traffic, fixed ports on InputEndpoints.
• Performance Improvement: Significant performance improvement local machine deployment.

This blog shows a first highlight of the new features that are part of the new SDK.

Extra small instance size

One of the nice features that many people have been asking for is the introduction of a new, lightweight, VM size: the Extra small instance.  They come with a small amount of memory (786 MB) and a low CPU speed (1Ghz), but at a very low cost (5ct / compute hour).  Configuring this is very easy, through the Role Configuration tab.

Remote desktop features

It is now possible to connect to an Azure instance, by using a Remote Desktop Connection.  The publishing wizard has been updated to enable this easily.  When deploying an application, you can provide a user name and a password for a remote desktop user, together with a certificate that has to be uploaded to the Azure portal, after exporting it with the private key.  (Export certificate - Browse to the portal – Your project – Installed Certificates – Manage – Upload certificate)

In the new Azure management portal, it is now possible to connect to a specific instance of an Azure Role, by clicking on the Connect button.

Public ports in endpoints

In previous SDK’s, it was unknown until at runtime at what actual port the role endpoints were hosted.  This can now be fixed and configured in the endpoints tab page on an Azure role:

Using Azure Connect

The Azure Connect feature is a very interesting feature that allows to set up a virtual network between Azure roles and the local network.  The property page of the Azure role allows to configure the specific token that is need for the Azure Connect feature.  This feature will be opened for a beta program soon, so we will be able to blog about it as soon as we are allowed in the beta program.

Sam Vanhoutte, Codit

Another great announcement on PDC was the Virtual Machine Role feature.  This feature is added to Azure with the primary goal to move existing applications to the cloud. 

The feature allows us to upload our own VHD (virtual hard disk) with the Operating System on it (Windows 2008 R2 Enterprise).  This machine could have your application pre-packaged and installed.  After doing this, you are now able to spin up multiple instances of that machine.

BizTalk on VM Role?

Being a BizTalk Server architect that is highly interested in the Azure platform, I immediately thought about a scenario where I could have my own BizTalk Server in the cloud, on Azure.  But, knowing some of the limitations of the Azure platform, I knew I would have a lot of potential issues. 

I listed these issues and added the various workaround or solutions for it.  This post is a post, based on the PDC information and can contain incorrect information.  Consider it as some early thinking and considerations.

No MSDTC support with SQL Azure

• Problem description
  • BizTalk Server relies heavily on SQL Server and uses MSDTC to keep transactional state across databases and with the adapters.
  • SQL Azure does not support distributed transactions and also introduces more latency to the database queries.
• Solution
  • SQL Server will need to be installed on the VHD image locally
• Consequences
  • It won’t be possible to build a multi-machine BizTalk Server Group through the VM role.

The OS image is not durable

• Problem description
  • All changes that are being made after a Virtual Machine instance is started will be lost, once the instance shuts down or fails over.  (there is only one VHD, but multiple instances are possible –> concurrency issues)
  • This means all database state (configuration, tracking, process state) will be lost if an instance fails.
• Consequences
  • It won’t be possible to have a stateful BizTalk Server configured or to host long running processes on a VM Role BizTalk Server
  • We will need to expose BizTalk Server capabilities as services to a stateful engine (Workflow?)

The Virtual Machine name will be assigned by the Windows Azure fabric controller

• Problem description
  • Since it is possible to have multiple instances of a VM running, these instances will get a specific Computer Name assigned by the Azure Fabric controller. 
  • It is very hard to change the computer name of a BizTalk Server machine
• Solution
  • We will need to automate the BizTalk Server configuration, using a Silent Install, once the VM is initiating.
• Consequences
  • One of the biggest painpoints in setting up BizTalk Server in a VM role will be to configure the BizTalk Server instance on the fly as a startup task.
  • Starting / restarting a BizTalk VM Role instance will take a considerable amount of time

Licenses are needed

• Problem description
  • BizTalk Server and SQL Server licenses are needed for each instance that is running
• Solution
  • Since everything will be installed on a single box, we could use a standard edition of BizTalk & SQL
• Consequences
  • There is no big pricing advantage, except for the operational cost
  • Only 5 applications will be supported , when using the standard edition

If we succeed in setting up BizTalk Server on VM Role at all, it will be a BizTalk Server with the following limitations:

• No support for long running transactions
• Single box machine
• Stateless BizTalk box

Considering that integration as a service is on the roadmap of Microsoft (see session at PDC), we should only consider it as a temporary solution to have BizTalk Server configured on a VM Role.  If we do this, then we should just see it as a BizTalk Server that exposes its various ‘short running / isolated’ capabilities as a service.  (flat file parsing, transformation, pub/sub, connectivity, EDI)

Sam Vanhoutte, Codit

Early Thinking

This post describes some new Azure features and how they might help us in leveraging the power of the web/app.config files again.  The content is this article is written, before any CTP/Beta versions of these features are available, so the details might be incorrect and there might be some wrong assumptions.  I will verify all of these parts, once the features are made available.

Current situation

Until now, an important difference between deploying a web application (or web service) on a local or hosted server and deploying a web application (or web service) to Windows Azure was the usage of the web.config file.

In a standard web application (hosted on IIS), the web.config can easily be changed to update configuration settings.  When changing this file, the web application gets refreshed and the new settings are being applied.

On Azure, there was a total different story.  The web.config was compiled with the deployment package and to change it, a new version of the packaged needed to be uploaded, resulting in a lot of work/time before the new settings got applied.  Luckily, Windows Azure provided a nice alternative: the serviceconfiguration files (.cscfg).  This file allowed to specify configuration values that could be loaded at runtime through the following code:

RoleEnvironment.GetConfigurationSettingValue("myValue"); code.

Painpoints with the Azure serviceconfiguration files.

The idea is great, having service configuration files managed on Role level and not on instance level.  But there are some big disadvantages in the following scenarios:

WCF Configuration settings

WCF has tons of configuration settings and I am a big fan of having my service endpoints configured through the .config files.  This allows for easy change and full declarative configuration, without having the settings (binding/behaviors…) compiled with the program.  And since the introduction of the WCF Router service, even more configuration (the pub/sub logic) gets in the .config file.

Having all these settings in the Azure .cscfg file, would be very hard to maintain, or we’d need to switch to custom configuration files.

Custom configuration settings

In most of the applications that are developed at Codit, we use custom configuration sections to have a better structured configuration experience.  These configuration sections are complex types and no standard name/value configuration pairs.  Migrating these applications to Azure, would mean our configuration logic would need to be totally redesigned.

ASP.NET settings

ASP.NET also include a lot of settings that might need to be changed at runtime.

PDC announcements

At PDC a lot of stuff was announced that can help us here.  This article described the various announcements and their impact on the configuration issues I described above.

Remote desktop capabilities

Through the updated Azure management portal (didn’t that look a 100 times nicer than the current one?), we will soon be able to have Remote Desktop access to our web.  This will be done by adding the following Modules in the service definition file (.csdef)

<Imports>

<Import moduleName="RemoteAccess" />

<Import moduleName="RemoteForwarder" />

</Imports>

This allows us to connect to a running instance of an Azure role.  This way we can alter the web.config file and have everything updated as on a regular web application.

The biggest disadvantages of this approach are the following:

• When having multiple instances (which is required to have High Availability), we need to remote in on every running instance of our web role.
• Our changes will be undone when our instance restarts or gets redeployed, because the machine gets ‘prepped’ again, resulting in a fresh installation of our Azure package.

The new Azure VM Role

Through the announced Windows Azure VM Role, we will be able to customize our own Virtual Machine, by uploading a pre-built virtual hard disk (VHD) to a blob storage account.

The steps to do this are the following :

• Install Windows Server 2008 R2 Enterprise on the VHD
• Add any Additional software you might need on the VM Role instances
• Install Windows Azure Integration Components that include the following components
  • Agent
  • Runtime interface (topo, config, shutdown notification)
  • Remote Desktop configurator
  • Windows Azure Drives driver
• Generalize the VHD is recommended, by using sysprep  (machine name will be assigned by Azure.)

By using a difference VHD, it might be easy to upload a new version of that VHD with the updated configuration file on it and have the changes applied.  Besides that, the same approach can be taken as described in the previous section (remote desktop capabilities).

The biggest disadvantages of this approach are the following:

• To upload a new version of the VHD difference disk, the role might need to be reinitiated (to be tested, once the VM Role is available for CTP)
• Cumbersome way of changing the web.config.

Startup tasks

As shown in the CloudCover show of two weeks ago (watch episode), there is a possibility of executing startup tasks, when the role instance starts up.  This looks like the best suited feature to solve our config file problem.

I am making one crucial assumption here (that I’ll check once this is made available):  I believe the start up tasks are being executed prior to the Role Instance initiation – so before the web application is started.  This means we can still change the web.config without having the web application being recycled automatically.

Leveraging the startup tasks will allow us to take the web.config from a configurable blob container and write it to the application directory on the Azure instances.  Therefore, we just need to write a script that performs this logic.  (code will be in a future post).  Then we just need to configure that script in the Startup element in the csdef configuration file:

<Startup>
     <Task commandline="updateconfig.cmd" executionContext="elevated" 
               taskType="simple/foreground/background" />
</Startup>

This approach looks like the best approach to make our web.config and app.config files configurable again.

Conclusion

A lot of uncertainties, because there is no bit available yet to test this with.  So, please expect a reviewed post, once it has been made available. 
But one thing is for sure: the new Azure features will make Azure application development much more flexible again.

Sam Vanhoutte, Codit

A common requirement in many development scenarios is caching.  In BizTalk implementations, this can be the case, mainly for performance reasons.

As a test I wrote 2 pipelinecomponents that handle 2 common issues with caching inside BizTalk.  These sample components are performing similar tasks to two components of the Codit implementation framework.  This is a framework we use at a lot of our customers.

1. Code table mappings : limit access to SQL database to perform Code table mappings.  In our framework, this is similar to the Transco component.
2. Duplicate message : stop messages that come into BizTalk multiple times within a specific timeframe.  In our framework, this is similar to the Double checker component.

CacheHelper

Because both pipelinecomponents use the AppFabric cache, I wrote a small class that takes care of this.

public class CacheHelper : IDisposable
    {
        private string _cacheName = "default";
        private string _region;

        public CacheHelper(string region)
        {
            _region = region;
            CreateRegion(_cacheName,region);
        }

        /// 
        /// Creates a Region in a specified cache.
        /// 
        /// Cache name
        /// Region name
        private void CreateRegion(string cacheName,string region)
        {
            DataCacheFactory dcf=ConnectToCache();

            if (dcf != null)
            {
                DataCache dc=dcf.GetCache(cacheName);
                dc.CreateRegion(region);
            }
        }

        /// 
        /// Connect to a Cache server
        /// 
        /// The Datacache
        private DataCacheFactory ConnectToCache()
        {
            //This can also be kept in a config file
            var config = new DataCacheFactoryConfiguration();
            config.SecurityProperties = new DataCacheSecurity(DataCacheSecurityMode.None, DataCacheProtectionLevel.None);
            config.Servers = new List
            {
                new DataCacheServerEndpoint(Environment.MachineName, 22233)
            };

            return new DataCacheFactory(config);
        }

        ~CacheHelper()
        {
            Dispose(false);
        }

        public void Dispose()
        {
            Dispose(true);
            GC.SuppressFinalize(this);
        }

        protected virtual void Dispose(bool disposing)
        {
        }

        /// 
        /// Gets a value from the cache in the specified region (class level)
        /// 
        /// Key linked to the data
        /// The found data. If null --> not found in the cache
        public string GetLookUpCacheData(string keyValue)
        {
            DataCacheFactory dcf = ConnectToCache();

            var cache = dcf.GetCache(_cacheName);
            string data = cache.Get(keyValue,_region) as string;

            return data;
        }

        /// 
        /// Store a value in the cache
        /// 
        /// Key
        /// Data
        public void StoreLookUpCacheData(string keyValue, object value)
        {
            DataCacheFactory dcf = ConnectToCache();

            var cache = dcf.GetCache(_cacheName);
            cache.Add(keyValue, value, _region);
        }

        /// 
        /// Stores a value in the cache for a specified amount of time
        /// 
        /// Key
        /// Data
        /// Time to keep the data in the cache
        public void StoreLookUpCacheData(string keyValue, object value,TimeSpan expires)
        {
            DataCacheFactory dcf = ConnectToCache();

            var cache = dcf.GetCache(_cacheName);
            cache.Add(keyValue, value, expires, _region);
        }        
    }

This is a very simple implementation that will store values in the default cache and in a specified region.

CodeTable Mapper

Codetable mapping is a very common scenario in BizTalk implementations. In my example we will be translating countrycodes to the country name.
The values are stored in a SQL table. But every time we get a value, we are going to save it to the AppFabric cache.
When we want to get the same value again, we are not going to the database but we will get the stored value from the AppFabric Cache.

public IBaseMessage Execute(IPipelineContext pContext, IBaseMessage pInMsg)
        {
            // Set variables
            biztalkMessage = pInMsg;
            XmlReader reader = XmlReader.Create(pInMsg.BodyPart.Data);
            XPathCollection xpaths = new XPathCollection();
            //For this example we are going to use 1 xpath expression
            xpaths.Add(this.XPath);

            ValueMutator vm = new ValueMutator(handleXpathFound);
            pInMsg.BodyPart.Data = new XPathMutatorStream(reader, xpaths, vm);
            return pInMsg;
        }

        private void handleXpathFound(int matchIdx, XPathExpression matchExpr, string origVal, ref string finalVal)
        {
            CacheHelper ch = new CacheHelper("Countries");
            string data = ch.GetLookUpCacheData(origVal);
            if (data == null)
            {
                finalVal = GetCountryFromDB(origVal);
                ch.StoreLookUpCacheData(origVal, finalVal);
            }
            else
                finalVal = ch.GetLookUpCacheData(origVal);
        }

        private string GetCountryFromDB(string countryCode)
        {
            string country = string.Empty;
            SqlConnection conn = null;
            SqlCommand comm = null;

            try
            {
                //Connect to look up database and retrieve the names of the products.
                conn = new SqlConnection("Data Source=(local);Initial Catalog=CacheTest;Integrated Security=SSPI;");
                conn.Open();

                comm = new SqlCommand();
                comm.Connection = conn;
                comm.CommandText = string.Format("SELECT Country FROM Countries WHERE CountryCode='{0}'", countryCode);
                comm.CommandType = CommandType.Text;

                country = (string)comm.ExecuteScalar();
                if(string.IsNullOrEmpty(country))
                    throw new Exception(string.Format("No country found for code {0}",countryCode));
            }
            catch (Exception e)
            {
                throw new Exception(e.Message + e.StackTrace);
            }
            finally
            {
                comm.Dispose();
                conn.Close();
                conn.Dispose();
            }


            return country;
        }

Duplicate Message Checker

As a sample scenario I took one I read about a few months ago. BizTalk had to stop messages that come in multiple times within 2 minutes.
So if there are more then 2 minutes between the messages, they should continue.

Normally this would involve a SQL table to store some information and some job to do the cleanup of this table.
But for my example I use AppFabric cache. There you have the option to store something in the cache for a certain timespan.
It is automatically deleted after this period.

public Microsoft.BizTalk.Message.Interop.IBaseMessage Execute(IPipelineContext pContext, Microsoft.BizTalk.Message.Interop.IBaseMessage pInMsg)
        {
            //Create hash
            VirtualStream input = new VirtualStream(pInMsg.BodyPart.GetOriginalDataStream());
            MD5 md5 = MD5.Create();
            byte[] hash = md5.ComputeHash(input);
            string hashString = Convert.ToBase64String(hash);

            //check Cache
            CacheHelper ch = new CacheHelper("DuplicateMessages");
            string date=ch.GetLookUpCacheData(hashString);
            if (string.IsNullOrEmpty(date))
            {
                //If not in cache yet, store it --> lifetime is 2 minutes
                ch.StoreLookUpCacheData(hashString, DateTime.Now.ToString(), new TimeSpan(0, 2, 0));
            }
            else
            {
                //Throw error
                throw new ApplicationException(string.Format("Duplicate Message. Already received at {0}",date));
            }

            //Put stream back to beginning
            input.Seek(0, SeekOrigin.Begin);
            return pInMsg;
        }

This makes the implementation very easy and you will not need a SQL table or anything to store the information.
You could say that you can do this with a custom caching solution as well. But what about HA environments with multiple BizTalk servers?
AppFabric is a distributed cache. So it doesn't mather on which server the message is processed. It will end up in the same cache and will be accessible on all the servers.

Conclusion

As you see, AppFabric caching has some advantages in BizTalk as well. The API is very easy to use and I got this to work quite quickly.

Tim D'haeyer, CODit

A couple of weeks ago, I had to create a custom EDI party trough code.
I noticed that the information provided on this topic was rather small and that other people were experiencing the same problem as well.
After some research I did on this topic, I discovered it’s actually rather easy to do these kinds of modifications through code.
Below you can find an example on how to create and modify your EDI Party.

Prerequisites

You will need to add references to following assemblies in order to create and modify an EDI Party trough code.

These assemblies can be found in the BizTalk installation folder.

Add new Party

The first step is to add a new party to your BizTalk Group.
Below is an example how to add it through code, but it’s perfectly possible to add this through the BizTalk Administration Console.

Modify the Party Settings

The party you just created will be used to create a new partner.
After that you can tweak and modify every setting needed for your Party/Partner.
There are tons of possibilities to adjust, but below you can see some basic modifications I did.

The result

When we open the BizTalk Administration console, we can view the result.

The source code of my example can be found here.
EDIPartyManagement.zip (18.43 kb)

Glenn Colpaert, CODit

WCF has some weak spots when it comes to contract first development. Creating a service starting from a given XSD or WSDL is a scenario that is common in enterprise integration.

One option is using svcutil.exe to generate a service interface for a WSDL file.

Today I found a tool on CodePlex that has much more capabilities: WSCF.blue tool.

WSCF comes with a lot of useful features (quote CodePlex):

• A WSDL Wizard that allows the developer to step through the creation of a WSDL from one or more XSDs.
• A Data Contract Generator (similar to XSD.exe, XSDObjectGen.exe and Svcutil.exe) that generates the . NET equivalent of the XSD types.
• A Service/Endpoint Stub (SVC) Generator and
• A Client Proxy Generator.
• A Generate Data Contract Code feature that supports the selection of multiple XSD/WSDL source files. (More Information)

Tools like this makes life much easier if you have to be compatible with a predefined WSDL file…

Peter Borremans