Yesterday, I gave a presentation at Amstelveen, NL, on the messaging enhancements in the AppFabric Service Bus.  This CTP was released last week and I did the following demos:

• Service Bus Tracer (using a client side WPF application that listens on trace information that gets broadcasted from Azure roles over the service bus eventrelaybinding)
• AppFabric queues & transactions: sending a batch of messages in one transaction to a queue.
• Sessions: splitting large messages in portions and send them in a session-enabled queue.
• Duplicate checks: preventing delivery of duplicate items, using deduplication on queues.
• Pub/sub demo: showing the usage of advanced filters and ad-hoc subscriptions on topics.

I uploaded the slide deck on SlideShare and the source code can be downloaded here.

Earlier this week, the first CTP of the Service Bus enhancements was released; the May CTP.  This CTP was announced at TechEd and on the AppFabric Team Blog.  One week prior to that, Codit has been accepted in the Technology Adoption Program (TAP) of the Windows Azure AppFabric team.  In this program, we will work closely together with the product team to build our cloud integration offering.

Being BizTalk experts since our inception, we are highly interested in this release of the AppFabric Service Bus, since this one provides real ‘hardcore’ messaging capabilities, just like BizTalk has.  These capabilities make asynchronous, loosely coupled scenarios possible on a cloud scale service bus.

This post dives into one of the most interesting features towards flexible routing and decoupling of messages: topics & subscriptions. 

Terminology

Queues

AppFabric service bus queues can be compared with the storage queues or with the message buffers of AppFabric V1, but they offer much more complex and rich functionality (sessions, dead letter queues: DLQ, deferring of messages, duplicate checking, transactions…)

Some enhancements over the azure storage queues:

• Large message sizes (currently: 256KB – storage queues: 8KB)
• Unlimited TTL (time to live – storage queues: max 7d)
• Correlation, request/reply
• Grouping of messages

Topics & subscriptions

Topics & subscriptions allow to implement real publish/subscribe patterns on the Windows Azure AppFabric platform.  A sender/publisher submits his messages to a topic.  (this means he does not need to know anything about any potential subscribers/destinations).  And linked with a topic, you can add multiple Subscriptions. 
A subscription will register on messages that are matching the filter for that subscription.  A filter is defined as a SQL92 or LINQ expression and that filter is matched against the properties of a message (BrokeredMessage).  If the filter is matched, a message will be marked for delivery on that subscription.
A new concept is the use of filter actions.  These can be configured on a subscription to update the properties of a message. 

The following schema shows an overview of topics and subscriptions. 

Someone familiar with BizTalk will immediately get this picture.  I would make the following analogy with BizTalk:

• AppFabric Topic = BizTalk Receive Agent (Receive Port)
• AppFabric Subscription = BizTalk Subscription
• AppFabric Filter Action= A very lightweight BizTalk pipeline (only influencing message properties)
• Message properties = Message Context

Implementing a pub/sub sample.

In this blog post, I am implementing a scenario where a client application (WPF app – the publisher block in the above diagram) is sending messages to a topic.  These messages are objects of a custom class: Material.  On these messages, the client application adds some properties.  These properties will be used to match the filters against.  The client allows to send multiple messages during a long period.  This will allow us to test creating subscriptions on the fly.

At the other end of the Service Bus, I will have multiple receivers that each either listen on an existing subscription, or that will create a subscription on the fly and receive from it.  I can start up different receivers and messages will be delivered to them, as long as the message properties match the filter of the subscription.

The code

Registering the Service Bus namespace

The blog post by David Ingham of the AppFabric team shows how you can register on the AppFabric labs portal and how you can download the SDK. You will need the name of the service bus namespace, an issuer name (owner is default, but should not be used in production) and the corresponding issuer secret key.

Creating the client application

• Create a new WPF Forms application.
• Add the Microsoft.ServiceBus and the Microsoft.ServiceBus.Messaging assemblies to the project.  Make sure you use the v2.0.0.0, because the v1.0.0.0 still references the old (?) version.
• I designed the client form like displayed further down this article.
• When clicking the submit button, I check if an interval is required between sending messages.  If that is the case, I send messages in a for loop with a Thread.Sleep between each send.  If there is no interval needed, I do a Parallel.For, so that all messages are send in parallel to the topic.
• When staring up the application, I initialize two important objects: the ServiceBusNamespaceClient (which will be used for management of the namespace) and the MessagingFactory (which will be used to send and receive messages).

             private void Initialize()        
             {
                          // Read issuer name & secret
                          string issuerName = ConfigurationManager.AppSettings["issuerName"];
                          string issuerKey = ConfigurationManager.AppSettings["issuerKey"];
                          string sbNamespace = ConfigurationManager.AppSettings["sbnamespace"];

                          // Create credentials
                          SharedSecretCredential sbCredentials = TransportClientCredentialBase.CreateSharedSecretCredential(issuerName, issuerKey);

                          // Namespace client
                          Uri sbAddress = ServiceBusEnvironment.CreateServiceUri("sb", sbNamespace, "");
                          sbClient = new ServiceBusNamespaceClient(sbAddress, sbCredentials);
                          msgFactory = MessagingFactory.Create(sbAddress, sbCredentials);
             }

• The implementation for sending the message is like this.  The object that is being sent is of type Material.  It will be serialized, using the standard Serializer.  I create a TopicClient for that specific topic and in that client, I create a Sender.

            private void SubmitToTopic(Material material, string action, int sequence)
            {
                        TopicClient topicClient = null;
                        lock (msgFactory)
                        {
                                    topicClient = msgFactory.CreateTopicClient(TOPICNAME);
                        }           
                        var sender = topicClient.CreateSender();
                        BrokeredMessage message = BrokeredMessage.CreateMessage(material);
                        message.Properties["Type"] = material.MaterialType;
                        message.Properties["Number"] = material.Number;
                        message.Properties["Region"] = material.Region;
                        message.Properties["Action"] = action;
                        message.Properties["Sequence"] = sequence;
                        sender.Send(message);
            }

Creating the subscribers

• In the subscribing application, I also add the initialization code as in the client app.
• After collecting some information from the user, through the console, I start listening on the subscription, using the following code

var subscriptionClient = msgFactory.CreateSubscriptionClient(topicName, subscriptionName);
var receiver = subscriptionClient.CreateReceiver(ReceiveMode.ReceiveAndDelete);
bool dontstop = true;
while (dontstop)
{
            BrokeredMessage message = null;
            while (receiver.TryReceive(TimeSpan.FromSeconds(10), out message))
            {
                        var material = message.GetBody<Material>();
                        Console.WriteLine("Material received from subscription " + subscriptionName);
                        Console.WriteLine(material);
             }
            Console.WriteLine("If you want to continue polling, enter X");
            dontstop = Console.ReadLine().ToLower() == "x";
}

• In the subscribing application, I also give the user the ability to create a subscription and immediately start listening on it. Therefore, I am executing the following code, prior to the above mentioned code.  I ask the user to enter a SQL expression that I am using to define the Filter for the Subscription.

Console.WriteLine("Please enter your new subscription name.");
string subscriptionName = Console.ReadLine();
Console.WriteLine("Please enter your filter expression.");
string filter = Console.ReadLine();
sbClient.GetTopic(topicName).AddSubscription(subscriptionName, new SqlFilterExpression(filter));

Creating the queues and topics

At this moment there is no out of the box tooling to create queues, topics and subscriptions.  Therefore, I wrote a simple WPF application to list, created, delete these objects in an AppFabric namespace. 

The application at work

The following screenshots show a simple test where I am sending a message with the Region set to EU (left) and I am listening on the EUSubscription with filter (Region = ‘EU’) on the right.

Filters

The filter expressions that can be applied are very rich.  A SQL syntax is being used to create these filters.  (the underlying store of the service bus is SQL Azure).  I tried out different types of filters.
These filters have been successfully tested as SqlFilterExpressions:

• Region = ‘EU’
• Sequence > 3 AND Sequence < 7
• Number LIKE ‘LG%’
• Region IN (‘US’, ‘EMEA’, ‘EU’)

Conclusions

This first release of the Service Bus Enhancements is very promising.  This will form the basis for the rest of the entire middleware platform of the future.  This release of the Service Bus provides real rich messaging capabilities, allowing to build decoupled, asynchronous connections.

Things we might (not sure yet) expect over time are:

• Pipelines (like in BizTalk) on topics, queues and subscriptions.
• Advanced Filter Actions
• Content based filtering (through Filter Actions?)
• Tooling

Sam Vanhoutte, Codit

In this post I will highlight what the different pain points are when hosting a WCF Service in Windows Azure with Service Bus Endpoints (e.g. using HttpRelayBinding)

Including Microsoft.ServiceBus in your deployment

As many people know already, the Windows Azure AppFabric SDK is not installed on Windows Azure. Which means that you won’t have the Microsoft.ServiceBus.dll once you deploy your solution to Windows Azure. This first problem is easily fixed by setting the ‘Copy Local’ property of this reference to true.

HttpRelayBinding error

Now chances are you have a web.config to accompany your svc file that uses one of the relay bindings that come with the AppFabric SDK.

<service name="test">
          <endpoint name="PublishServiceEndpoint"
                      contract="WCFServiceWebRole1.IService1"
                      binding="basicHttpRelayBinding" />
</service>

Once you deploy and navigate to your svc you will get an error saying ….

Of course you can modify your web.config to include the necessary extensions manually, as specified in this post, but it won’t work anymore on your dev machine, since installing the AppFabric SDK already updated your machine.config with all the extensions. I wanted a solution where the machine.config was updated on Windows Azure prior to executing my code.

Using Startup Tasks

Luckily the AppFabric SDK comes with a tool called RelayConfigurationInstaller.exe which installs the Machine.config settings necessary for the Service Bus bindings to be supported in App.config. Now this just screams Startup Task.

So I went ahead and created a new solution and added the RelayConfigurationInstaller.exe to the project along with a RelayConfigurationInstaller.exe.config file which looks like this:

?xml version ="1.0"?>
<configuration>
          <startup>
                    <requiredRuntime safemode="true" imageVersion="v4.0.30319"           
                              version="v4.0.30319"/>
          </startup>
</configuration>

I also added a Startup.cmd file which simply calls RelayConfigurationInstaller.exe /i. Then went into the ServiceDefinition.csdef and added the Startup Task with the call to Startup.cmd. This is what my project looks like so far:

Deploy…waiting 15 minutes… and no difference, I still get the same error, what went wrong?

Installing Microsoft.ServiceBus in the GAC

I found out that executing my startup task actually resulted in an error:

Which means that the Microsoft.ServiceBus.dll should be installed in the GAC prior to calling the executable.

Now this presents a problem, there is no Windows SDK installed on Windows Azure, so we can’t execute a gacutil to install the assembly. However I discovered that you can use the System.EnterpriseServices library to publish a dll into the GAC. Time to include powershell in my startup task.

I’ve created this ps script that installs the Microsoft.ServiceBus.dll in the GAC and added it to my project:

BEGIN {
         $ErrorActionPreference = "Stop"
 
         if ( $null -eq ([AppDomain]::CurrentDomain.GetAssemblies() |? { $_.FullName -eq "System.EnterpriseServices, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" }) ) 
         {
                  [System.Reflection.Assembly]::Load("System.EnterpriseServices, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a") | Out-Null 
         }

         $publish = New-Object System.EnterpriseServices.Internal.Publish
}
PROCESS {
         $dir = [Environment]::CurrentDirectory=(Get-Location -PSProvider FileSystem).ProviderPath
         $assembly = Join-Path ($dir) "Microsoft.ServiceBus.dll"

         if ( -not (Test-Path $assembly -type Leaf) ) 
         {
                  throw "The assembly '$assembly' does not exist."
         }
         
         if ( [System.Reflection.Assembly]::LoadFile($assembly).GetName().GetPublicKey().Length -eq 0 ) 
         {
                  throw "The assembly '$assembly' must be strongly signed."
         }
         Write-Output "Installing: $assembly"
         $publish.GacInstall($assembly) 
}

next I’ve modified the startup.cmd file to first call the ps script and then call the RelayConfigurationInstaller.exe to modify the machine.config with the needed bindings. This is what the final cmd file looks like:

 1: powershell.exe Set-ExecutionPolicy RemoteSigned -Force

 2: powershell.exe .\Startup\gacutil.ps1

 3: Startup\RelayConfigurationInstaller.exe /i

Caveats

1. Make sure you’ve selected osFamily=”2” in your ServiceConfiguration.cscfg file, since this gives you Windows Server 2008 R2, needed for caveat 2.
2. If you want to execute powershell scripts, you have to set the ExecutionPolicy to RemoteSigned (see line 1 of startup.cmd)

Wouter Seye, Codit

Lately, a lot of CTP’s came available on the Windows Azure platform.  My goal is to try them all out.  I blogged on two of these technologies already:

• Using Windows Azure VM Role
• Azure connect - VPN as a service

This time, we’re getting on the data side, with SQL Azure Data Sync.  This data synchronization service is built on the Microsoft Sync Framework technologies. It provides bi-directional data synchronization and data management capabilities allowing data to be easily shared across SQL Azure databases within multiple data centers.

A very good and detailed overview on Data Sync can be found on the TechNet Wiki: http://social.technet.microsoft.com/wiki/contents/articles/sql-azure-data-sync-overview.aspx

Scenarios

This SQL Azure Data Sync service is a very promising and interesting feature that opens a lot of great scenarios:

• For development/testing purposes: provide data in your local database (on development) and synchronize with the Azure database to feed the cloud instances.
• Synchronizing configuration or master data across databases in a very easy way.
• Use SQL Azure Reporting Services on ‘local’ data.

Registration for CTP

Since this feature is still in CTP, you need to register first to get an invitation code.  To do all this, just browse to https://datasync.azure.com and sign in with your live id.  After this, you can enter your e-mail and other details and you should receive an invitation code some time later.  With that registration code, you can log on to the full features.

Concepts

The following concepts are important in setting up Data Sync:

• Sync group: a sync group is a group of databases that can be synchronized together
• Database: a database that gets registered for SQL Azure Data Sync
• Agent: an agent is a Windows Service that performs and orchestrates the actual synchronization

Configuring data synchronization

Step 1: download the Sync Agent

A Sync agent is only required when synchronizing data from an on-premise database.  If the synchronization is set up from Cloud to Cloud, then it is not required to download the Agent, since the synchronization logic will be run in the Cloud.

For this example, I will synchronize a local database with a SQL Azure database.

On the Data Sync portal, click the Agents tab page and download the Agent installer at the bottom of the page.  After running the installer (and providing credentials for the Data Sync windows service), you can open a local configuration tool, through the Start menu: SQL Azure Data Sync Agent CTP2.

This tool allows you to configure local databases for synchronization.  The following screenshot shows the default configuration window, after installation.  To join this agent to a Windows Azure subscription, it is important to configure the Agent Key first.  This can be done by clicking the Edit Agent Key button and providing the key that can be copied from the Data Sync Portal. 

Once this is configured, it should be possible to Ping the Azure Sync Service, by clicking the Ping Sync Service button.

Step 2: Add your local database

Now we have the local agent installed, we will add a local (on premise) database to the configuration tool, so that it can be synchronized later.

In this example, I am using a custom database with 3 tables: Customer, OrderLines, OrderEvents.

• I will add this database in the SQL Azure Data Sync Agent tool, by clicking the ‘Add Member’ button at the left.  This pops up a configuration window where the server, database and the authentication method need to be selected. (1) 
• The Data Sync services have some limitations on supported data types, etc.  It is possible to check the database for issues, by clicking the ‘Check Member Schema’ button. (2).
• I added a field with an unsupported data type (geography) to the customer table and the Schema validation provided a warning that indicated that this column would be skipped. (3) This is because geography is not supported in Windows Azure at this point in time.
• It is very important to start the Windows Service of the Azure Data Sync Agent, in order to register the client database with the Windows Azure subscriptions.

Step 3: Add the SQL Azure database

Adding a SQL Server database to the Data Sync subscription is much easier and can be done on the Data Sync management portal, by clicking the Databases tab and clicking the Add button.  In the pop up, you just need to provide the server, database and credentials, before saving. 

In my example, I am just adding a new empty database, here.

Step 4: Set up the synchronization, through a Sync Group

• In the management portal, click the New Sync Group button in the Sync Groups tab. And add all the databases you want to synch together to the database list(1) and click Next.
• In the next screen, you can add the tables, per database that you want to synchronize.  For this demo, I want to synchronize everything, except the events table. (2)  You can also enable a schedule for the synchronization (expressed in minutes).
• Once the group is created, you can synchronize the group easily.

Testing the data synchronization

Now I have the databases configured for synchronization, it’s time to play around with it a bit.  To execute the synchronization, you can either rely on the synchronization schedule , when configured on the Sync group, or you can manually trigger the synchronization.

1st synchronization: Creating tables in the cloud.

The first synchronization I did created the two synchronized tables on my empty Cloud database and added the data there.  One thing to notice is that the Location column (with the spatial data type) was not created on the SQL Azure database. 

2nd synchronization: Adding data locally, synchronizing to the cloud.

In this test, I added two customers and some order lines to the local database and synchronized with the Cloud database, to find out that everything was copied without problems.

3rd synchronization: Adding data in the cloud, synchronizing locally.

In this test, I added a customer to the cloud database and synchronized with the on premise database, to find out that everything was copied without problems.

4th synchronization: Adding data on premise and in the cloud, synchronizing bi-directionally

In this test, I added customers in both databases before synchronization to find out that the data was synchronized correctly. 

Data conflicts

Now, I wanted to simulate some conflicts to find out how the Data Sync would handle them.

Adding or updating data in both databases, with the same primary key.

I added two different customers, but with the same primary key in both databases.  But, surprisingly, the synchronization happened without any issue, but my local customer was overridden by the cloud customer, resulting in lost data. 

• Both records (local + cloud) were normally marked as new, so it should be possible to detect this.
• I was looking to find out if the ‘most recent’ record won, but in all scenarios, the data from the cloud database survived the synchronization.  (probably, because this database was added first)

Maybe something for a next CTP?

Deleting data that is being referenced by a non-synchronized table

Another interesting test was to delete an order line on my cloud database.  But this order line was being referenced by a record in my local database (OrderEvent).  Knowing that the OrderEvent table is not being synchronized, this should result in a conflict.

Here I did not receive an exception, but I also noticed that my record on my local database still existed, where my cloud record was deleted.  So here my data was out of synch. 

Maybe something for a next CTP?

Adding a third database

The last test I did, was adding a new empty (cloud) database to the sync group and after the synchronization, everything seemed to work automagically.  Great!

Underlying design

• When configuring a table for synchronization, triggers are being created for the three actions: insert, update, delete.  They have the following name structure: [TableName]_dss_[action]_trigger.
• These triggers add the data in new tables that are being created during configuration of the sync group.  For every data table, a sync table is being added with the following name: DataSync.[TableName]_dss_tracking
• Next to that, we can also see that a lot of new stored procedures are getting created

Conclusion

This CTP looks already very stable and the sychronization (between multiple databases) is very smooth.  I am just hoping that there will be a better view or configuration for synchronization exceptions (like explained in the conflicts section).

Sam Vanhoutte, Codit

Apparently, I was lucky to be in the first batch of beta participants for the 3 new Azure beta programs:

• I received the confirmation for the extra small instances rather fast
• Wednesday evening, I received confirmation for the Azure Connect beta program.  You can read my post on that here: Azure Connect: VPN as a Service, a quick introduction.
• And later that night, I received confirmation for the VM Role beta program.

This post will demonstrate how to upload a virtual hard disk to the Azure storage and how to configure it to run it as your own instance(s) in the cloud.

Creating the VHD

Microsoft Hyper-V on server, or VMWare workstation on laptop?

I was very happy with Windows Virtual PC.  It was lightweight, free and pretty stable.  It allowed us, developers, to use it on our laptops on the road, at our customers, in the office.  But there is one important thing: Virtual PC does not support a 64-bit OS on a guest system.  And that became a big problem with the introduction of Windows Server 2008 R2; that operating system is only available in 64-bit, which makes sense. 

Because of this, we needed an alternative:

• Hyper-V: free, stable, Microsoft-product, but not running on Windows 7
• VMWare Workstation: not free, stable, non-Microsoft product, but running on Windows 7

So, I decided to use VMWare for the creation of my virtual machines for Azure. 

To leverage the full capabilities of the VM Roles, it would be better to use Hyper-V though, since that allows the usage of differencing disks, which would be a big timesaver for incremental updates to a virtual machine image.  But for now, I’ve used VMWare, since I didn’t have a physical Hyper-V server available.

Preparing the Virtual Machine

I took the following steps to create the Virtual machine.

• Created a virtual machine
• Installed Windows Server 2008 R2 Enterprise edition on it
• I enabled the Application Server and Web Role (and therefore also the .NET Framework 3.5.1 features)
• I installed all updates, but disabled the auto update feature.

Installing the Windows Azure Integration Components

• The Windows Azure Integration Components are a set of components that are needed to be able to communicate with the Windows Azure Fabric controller.  They need to be installed on the virtual machine, prior to uploading the machine to the Azure storage.
• Load the iso file that is found in the Azure SDK (iso directory - wavmroleic.iso) as a DVD/IDE in your virtual machine
• Start the installer on the virtual machine, by double clicking WaIntegrationComponents-x64.msi
• Follow the installation screen (Next-next-finish)

Please make sure that the firewall windows service is enabled (see ‘Things to know’ at the end of this post)

Sysprepping the Virtual machine

There are some specificities with VM Roles that are different from a typical hosting/IaaS solution.  I discussed some of these in my previous post: Hosting BizTalk Server on Azure VM Role.  One of the important things to know is that the machine name of the Virtual Machine is changed and assigned at runtime by the Azure Fabric controller.  This makes sense, since it is possible to configure multiple instances for a VM Role.

To make the virtual machine ready for these actions, the machine needs to be ‘sysprepped’.  This is a typical action that is taken when creating a base image for a machine.  It provides a different SID and name for the machine.

• Run the %windir%\system32\sysprep\sysprep.exe tool
• In the popup windows, select the “Enter System Out-of-Box experience (OOBE)” for the System Cleanup Action
• Make sure that Generalize is selected and select Shutdown in the Shutdown options.
• OK

[Only with VMWare] : Convert to VHD file

• This step is only needed when using VMWare. 
• I used the tool VMDK to VHD Converter and that allowed me to convert my VMWare virtual disk to a VHD.

Uploading the Virtual Hard disk (VHD)

The VHD file now needs to be uploaded.  For this, the Azure SDK 1.3 comes with a command-line tool: CSUPLOAD.  Using this tool, we can upload the VHD to our Azure subscription.

• Open the Azure SDK Command prompt
• Type the following command to link the context to your current subscription:
  csupload Set-Connection “SubscriptionId=[SubscriptionId];CertificateThumbprint=[tp]”
  The subscription id can be found on the management portal in the property grid, when selecting a subscription.
  The certificate thumbprint needs to be a private certificate that is installed on the local machine and that is linked to the management certificates on the Azure portal.  On the Azure portal, you can easily copy the thumbprint from the property grid.
• Now the upload command needs to be done
  csupload Add-VMImage –LiteralPath [localpathtovhd] –Location “West Europe” –Name [cloudname]
  Passing in the localpath to the VHD is important.  Next to that, the location (data center) needs to be defined where the page blob will be created.  And optionally you can specify a friendly name that will be used on the Azure management portal.
  
• When this is done, a verification window pops up.  This indicates that the image will be mounted to verify some settings (like Operating System, etc).  This verification is important because it can prevent you from uploading a VHD of several GBs, and then finding out there is an incorrect configuration.
  If the operating system you’re uploading the VHD from does not support mounting a VHD, you can use the parameter –SkipVerify of the CSUpload tool to skip this step.
  
• After the verification, the upload to the Azure data center starts and that seems to be a good moment to have a sleep (like I did), take a break, or do something else.  The upload typically will take some hours and therefore, the csupload tool is recoverable from connection errors.
  
• As soon as the upload process starts, you can already look at the Azure portal and see the status of the VHD as pending.  The VHD is created as a page blob.
  
• After the upload is completed, you can see the status of the VHD to committed on the Azure portal.  When the connection breaks, it is possible to re-execute the command and the upload will resume where it stopped.  (see ‘Things to know’)

Configure the Azure Role Service

Create Visual Studio Project

• In Visual Studio (after installing the Azure SDK 1.3 and the VMRole Beta unlocker), you can create a new Visual Studio Cloud project and add a VMRole to the Cloud project.
• On the property page of the VMRole project, you can specify the credentials and the VHD that is uploaded to the Azure platform.
  
• On the Configuration tab page, you can specify the number of instances and the size of the instances (notice that I can use the Extra small instance size here, part of the beta program)
  
• It is also required to open the endpoints when needed (like when IIS needs to be used, you’ll typically create an HTTP endpoint on port 80 or 443)

Publishing the project to Azure

• Now it is straightforward to publish this project to Azure, using Visual Studio, or by uploading it on the Azure management portal.
• After the upload is done, you can see on the management portal the process of the Virtual machine.
• The roles take the following statuses
• Initializing
• Waiting for host
• Setting up Windows for first use
• Starting Windows
• Starting Traffic
• Ready

Things to know & troubleshooting

Don’t disable the Windows Firewall service

On my virtual machine (locally), I had disabled the Windows Firewall service.  Because of that, the installation of the Windows Azure Integration Components failed with the following exception (copying it here for bing/google search):
Product: Windows Azure VM Role Integration Components -- Error 1720. There is a problem with this Windows Installer package. A script required for this install to complete could not be run. Contact your support personnel or package vendor. Custom action InstallScriptAction script error -2147023143, : Line 81, Column 5,

After I enabled the Windows firewall service again, things worked smooth again.

Resuming the upload

• Just before the upload finished, I received a timeout exception on the csupload command.  So, I decided to find out if the resumability of the CSUpload tool works as announced.
• I re-executed the command the 1st time and I noticed the upload started again at 3,2%…  Knowing that I’ve seen the upload pending at 98%, the last time I checked, I decided to retry it again.  (ctrl+c to exit the command)
• The second time, the resume started at 87,3%.  So I tried again.
• And, third time lucky!, now I immediately received a confirmation of the upload.  And the management portal also reflected this well.

Don’t install the Windows Azure SDK on the Virtual Machine

It is not supported to install the Windows Azure SDK inside a VM intended for use with VM role. The Windows Azure SDK and the Integration Components are mutually exclusive.

No support for startup tasks

Unfortunately there seems to be no support to use Startup tasks in VMRoles.  So startup tasks will have to be ‘embedded’ in the Virtual Machine startup process.  This might change after the beta, off course.

Conclusion

Again, just as with the Azure Connect, I was very surprised by the simplicity of the process of getting things to work.  Off course, it takes more time, because of the preparation and the upload, but besides that, things looks solid and stable.

Things that I hope to see soon:

• Base images on Azure that can be downloaded, so that only the differencing disks need to be uploaded.  This would save a lot of time and bandwidth.  (download is cheaper/faster than upload)
• Start up tasks.

Great stuff.

Sam Vanhoutte, Codit

I just received a notification that my request to join the Beta Program for Azure Connect was approved.  And that immediately got me starting to test it out.  Things look very straightforward.

I am using the phrase VPN as a Service for this feature, since it really explains it all.

Virtual Network Configuration of Windows Azure Role

Portal settings

• After logging in on the Azure Portal, you can click the Virtual Network button in the left corner at the bottom of the screen:
• After this, it is possible to enable the Virtual Network features for a specific subscription
• When selecting a subscription, you can get the Activation Token from the portal, by clicking the ‘Get Activation Token’ button.  That allows to copy the activation token to the clipboard for later use.
  

Visual Studio project settings

• In Visual Studio, when having the SDK 1.3 installed, it is possible to copy the activation token to the properties of an Azure role in the property pages:
  
• Now you can deploy the role to the Windows Azure portal.

Adding an ‘on-premise’ server to the Virtual Cloud Network

Installing the Azure Connect Client software agent

• On the local servers, it is now possible to install the ‘Local endpoint’, by clicking the correct button.
• This shows a link to download the software on the machine (on premise).  This link is only active for a while.
  
• The installation package is very easy to install, by selecting the correct language and clicking Next-Next-Finish.  After the endpoint software is installed, be sure to open the TCP 443 outbound port.
• As expected, the local endpoint agent runs as a Windows Service:
  

Adding a local endpoint to an Azure Endpoint group

• An Azure Endpoint group can be created, by clicking the “Create Group” button in the ribbon of the management portal.
• This pops up a wizard where you can provide a name for the group and where you can add the local endpoints and Azure roles that should be part of the group.  You can also indicate if the local endpoints are “interconnected” or not.  This specifies if the endpoints can reach each other. 
  (be careful: in some multi-tenant situations, this can be seen as a risk!)
• I could immediately see my local computer name in the Local Endpoint list and in the Role list, I could only see the role that was configured with the activation token for this Connect group. 
  
• That’s the only required actions we need to take and now we have IP/Network connectivity between my local machine and my Azure role in the Cloud.

Testing the connectivity

Since I had added the Remote Desktop Connectivity support to my Azure role (see my previous blog post: Exploring the new Azure property pages in Visual Studio), I am now able to connect to my Role instance in the cloud and connect to it.

• After logging in on my machine, I was immediately able to connect to my local machine, using my machine name.  I had a directory shared on my local machine and I was able to connect to it.
  
• For a nice test, I added a nice ‘cloud picture’ on my local share and selected it to be my desktop background in the cloud.  (the picture was on top of a Mountain in the French Alps, with the Mount Blanc in the background, for those wondering) 
• A part of my cloud desktop is here:
  

Conclusion

This was a very simple post, highlighting the way to set up the configuration between a Cloud app and local machines.  It really only took me about 5 minutes to get this thing working, knowing that I had never seen or tested this functionality before (only heard about it).

Some nice scenarios can now be implemented:

• Making your Azure roles part of your Active Directory
• Network connectivity between Cloud and Local (including other protocols, like UDP)

Definitely more to follow soon.

Sam Vanhoutte, Codit

Great news for the Azure community.  The Windows Azure SDK and Windows Azure Tools for Microsoft Visual Studio (November 2010) have been released!  This release contains a lot of new features and most of them were announced at PDC.

The release can be downloaded here.

The overview of new features:

• Virtual Machine (VM) Role (Beta):Allows you to create a custom VHD image using Windows Server 2008 R2 and host it in the cloud.
• Remote Desktop Access: Enables connecting to individual service instances using a Remote Desktop client.
• Full IIS Support in a Web role: Enables hosting Windows Azure web roles in a IIS hosting environment.
• Elevated Privileges: Enables performing tasks with elevated privileges within a service instance.
• Virtual Network (CTP): Enables support for Windows Azure Connect, which provides IP-level connectivity between on-premises and Windows Azure resources.
• Diagnostics: Enhancements to Windows Azure Diagnostics enable collection of diagnostics data in more error conditions.
• Networking Enhancements: Enables roles to restrict inter-role traffic, fixed ports on InputEndpoints.
• Performance Improvement: Significant performance improvement local machine deployment.

This blog shows a first highlight of the new features that are part of the new SDK.

Extra small instance size

One of the nice features that many people have been asking for is the introduction of a new, lightweight, VM size: the Extra small instance.  They come with a small amount of memory (786 MB) and a low CPU speed (1Ghz), but at a very low cost (5ct / compute hour).  Configuring this is very easy, through the Role Configuration tab.

Remote desktop features

It is now possible to connect to an Azure instance, by using a Remote Desktop Connection.  The publishing wizard has been updated to enable this easily.  When deploying an application, you can provide a user name and a password for a remote desktop user, together with a certificate that has to be uploaded to the Azure portal, after exporting it with the private key.  (Export certificate - Browse to the portal – Your project – Installed Certificates – Manage – Upload certificate)

In the new Azure management portal, it is now possible to connect to a specific instance of an Azure Role, by clicking on the Connect button.

Public ports in endpoints

In previous SDK’s, it was unknown until at runtime at what actual port the role endpoints were hosted.  This can now be fixed and configured in the endpoints tab page on an Azure role:

Using Azure Connect

The Azure Connect feature is a very interesting feature that allows to set up a virtual network between Azure roles and the local network.  The property page of the Azure role allows to configure the specific token that is need for the Azure Connect feature.  This feature will be opened for a beta program soon, so we will be able to blog about it as soon as we are allowed in the beta program.

Sam Vanhoutte, Codit

Another great announcement on PDC was the Virtual Machine Role feature.  This feature is added to Azure with the primary goal to move existing applications to the cloud. 

The feature allows us to upload our own VHD (virtual hard disk) with the Operating System on it (Windows 2008 R2 Enterprise).  This machine could have your application pre-packaged and installed.  After doing this, you are now able to spin up multiple instances of that machine.

BizTalk on VM Role?

Being a BizTalk Server architect that is highly interested in the Azure platform, I immediately thought about a scenario where I could have my own BizTalk Server in the cloud, on Azure.  But, knowing some of the limitations of the Azure platform, I knew I would have a lot of potential issues. 

I listed these issues and added the various workaround or solutions for it.  This post is a post, based on the PDC information and can contain incorrect information.  Consider it as some early thinking and considerations.

No MSDTC support with SQL Azure

• Problem description
  • BizTalk Server relies heavily on SQL Server and uses MSDTC to keep transactional state across databases and with the adapters.
  • SQL Azure does not support distributed transactions and also introduces more latency to the database queries.
• Solution
  • SQL Server will need to be installed on the VHD image locally
• Consequences
  • It won’t be possible to build a multi-machine BizTalk Server Group through the VM role.

The OS image is not durable

• Problem description
  • All changes that are being made after a Virtual Machine instance is started will be lost, once the instance shuts down or fails over.  (there is only one VHD, but multiple instances are possible –> concurrency issues)
  • This means all database state (configuration, tracking, process state) will be lost if an instance fails.
• Consequences
  • It won’t be possible to have a stateful BizTalk Server configured or to host long running processes on a VM Role BizTalk Server
  • We will need to expose BizTalk Server capabilities as services to a stateful engine (Workflow?)

The Virtual Machine name will be assigned by the Windows Azure fabric controller

• Problem description
  • Since it is possible to have multiple instances of a VM running, these instances will get a specific Computer Name assigned by the Azure Fabric controller. 
  • It is very hard to change the computer name of a BizTalk Server machine
• Solution
  • We will need to automate the BizTalk Server configuration, using a Silent Install, once the VM is initiating.
• Consequences
  • One of the biggest painpoints in setting up BizTalk Server in a VM role will be to configure the BizTalk Server instance on the fly as a startup task.
  • Starting / restarting a BizTalk VM Role instance will take a considerable amount of time

Licenses are needed

• Problem description
  • BizTalk Server and SQL Server licenses are needed for each instance that is running
• Solution
  • Since everything will be installed on a single box, we could use a standard edition of BizTalk & SQL
• Consequences
  • There is no big pricing advantage, except for the operational cost
  • Only 5 applications will be supported , when using the standard edition

If we succeed in setting up BizTalk Server on VM Role at all, it will be a BizTalk Server with the following limitations:

• No support for long running transactions
• Single box machine
• Stateless BizTalk box

Considering that integration as a service is on the roadmap of Microsoft (see session at PDC), we should only consider it as a temporary solution to have BizTalk Server configured on a VM Role.  If we do this, then we should just see it as a BizTalk Server that exposes its various ‘short running / isolated’ capabilities as a service.  (flat file parsing, transformation, pub/sub, connectivity, EDI)

Sam Vanhoutte, Codit

Early Thinking

This post describes some new Azure features and how they might help us in leveraging the power of the web/app.config files again.  The content is this article is written, before any CTP/Beta versions of these features are available, so the details might be incorrect and there might be some wrong assumptions.  I will verify all of these parts, once the features are made available.

Current situation

Until now, an important difference between deploying a web application (or web service) on a local or hosted server and deploying a web application (or web service) to Windows Azure was the usage of the web.config file.

In a standard web application (hosted on IIS), the web.config can easily be changed to update configuration settings.  When changing this file, the web application gets refreshed and the new settings are being applied.

On Azure, there was a total different story.  The web.config was compiled with the deployment package and to change it, a new version of the packaged needed to be uploaded, resulting in a lot of work/time before the new settings got applied.  Luckily, Windows Azure provided a nice alternative: the serviceconfiguration files (.cscfg).  This file allowed to specify configuration values that could be loaded at runtime through the following code:

RoleEnvironment.GetConfigurationSettingValue("myValue"); code.

Painpoints with the Azure serviceconfiguration files.

The idea is great, having service configuration files managed on Role level and not on instance level.  But there are some big disadvantages in the following scenarios:

WCF Configuration settings

WCF has tons of configuration settings and I am a big fan of having my service endpoints configured through the .config files.  This allows for easy change and full declarative configuration, without having the settings (binding/behaviors…) compiled with the program.  And since the introduction of the WCF Router service, even more configuration (the pub/sub logic) gets in the .config file.

Having all these settings in the Azure .cscfg file, would be very hard to maintain, or we’d need to switch to custom configuration files.

Custom configuration settings

In most of the applications that are developed at Codit, we use custom configuration sections to have a better structured configuration experience.  These configuration sections are complex types and no standard name/value configuration pairs.  Migrating these applications to Azure, would mean our configuration logic would need to be totally redesigned.

ASP.NET settings

ASP.NET also include a lot of settings that might need to be changed at runtime.

PDC announcements

At PDC a lot of stuff was announced that can help us here.  This article described the various announcements and their impact on the configuration issues I described above.

Remote desktop capabilities

Through the updated Azure management portal (didn’t that look a 100 times nicer than the current one?), we will soon be able to have Remote Desktop access to our web.  This will be done by adding the following Modules in the service definition file (.csdef)

<Imports>

<Import moduleName="RemoteAccess" />

<Import moduleName="RemoteForwarder" />

</Imports>

This allows us to connect to a running instance of an Azure role.  This way we can alter the web.config file and have everything updated as on a regular web application.

The biggest disadvantages of this approach are the following:

• When having multiple instances (which is required to have High Availability), we need to remote in on every running instance of our web role.
• Our changes will be undone when our instance restarts or gets redeployed, because the machine gets ‘prepped’ again, resulting in a fresh installation of our Azure package.

The new Azure VM Role

Through the announced Windows Azure VM Role, we will be able to customize our own Virtual Machine, by uploading a pre-built virtual hard disk (VHD) to a blob storage account.

The steps to do this are the following :

• Install Windows Server 2008 R2 Enterprise on the VHD
• Add any Additional software you might need on the VM Role instances
• Install Windows Azure Integration Components that include the following components
  • Agent
  • Runtime interface (topo, config, shutdown notification)
  • Remote Desktop configurator
  • Windows Azure Drives driver
• Generalize the VHD is recommended, by using sysprep  (machine name will be assigned by Azure.)

By using a difference VHD, it might be easy to upload a new version of that VHD with the updated configuration file on it and have the changes applied.  Besides that, the same approach can be taken as described in the previous section (remote desktop capabilities).

The biggest disadvantages of this approach are the following:

• To upload a new version of the VHD difference disk, the role might need to be reinitiated (to be tested, once the VM Role is available for CTP)
• Cumbersome way of changing the web.config.

Startup tasks

As shown in the CloudCover show of two weeks ago (watch episode), there is a possibility of executing startup tasks, when the role instance starts up.  This looks like the best suited feature to solve our config file problem.

I am making one crucial assumption here (that I’ll check once this is made available):  I believe the start up tasks are being executed prior to the Role Instance initiation – so before the web application is started.  This means we can still change the web.config without having the web application being recycled automatically.

Leveraging the startup tasks will allow us to take the web.config from a configurable blob container and write it to the application directory on the Azure instances.  Therefore, we just need to write a script that performs this logic.  (code will be in a future post).  Then we just need to configure that script in the Startup element in the csdef configuration file:

<Startup>
     <Task commandline="updateconfig.cmd" executionContext="elevated" 
               taskType="simple/foreground/background" />
</Startup>

This approach looks like the best approach to make our web.config and app.config files configurable again.

Conclusion

A lot of uncertainties, because there is no bit available yet to test this with.  So, please expect a reviewed post, once it has been made available. 
But one thing is for sure: the new Azure features will make Azure application development much more flexible again.

Sam Vanhoutte, Codit

A common requirement in many development scenarios is caching.  In BizTalk implementations, this can be the case, mainly for performance reasons.

As a test I wrote 2 pipelinecomponents that handle 2 common issues with caching inside BizTalk.  These sample components are performing similar tasks to two components of the Codit implementation framework.  This is a framework we use at a lot of our customers.

1. Code table mappings : limit access to SQL database to perform Code table mappings.  In our framework, this is similar to the Transco component.
2. Duplicate message : stop messages that come into BizTalk multiple times within a specific timeframe.  In our framework, this is similar to the Double checker component.

CacheHelper

Because both pipelinecomponents use the AppFabric cache, I wrote a small class that takes care of this.

public class CacheHelper : IDisposable
    {
        private string _cacheName = "default";
        private string _region;

        public CacheHelper(string region)
        {
            _region = region;
            CreateRegion(_cacheName,region);
        }

        /// 
        /// Creates a Region in a specified cache.
        /// 
        /// Cache name
        /// Region name
        private void CreateRegion(string cacheName,string region)
        {
            DataCacheFactory dcf=ConnectToCache();

            if (dcf != null)
            {
                DataCache dc=dcf.GetCache(cacheName);
                dc.CreateRegion(region);
            }
        }

        /// 
        /// Connect to a Cache server
        /// 
        /// The Datacache
        private DataCacheFactory ConnectToCache()
        {
            //This can also be kept in a config file
            var config = new DataCacheFactoryConfiguration();
            config.SecurityProperties = new DataCacheSecurity(DataCacheSecurityMode.None, DataCacheProtectionLevel.None);
            config.Servers = new List
            {
                new DataCacheServerEndpoint(Environment.MachineName, 22233)
            };

            return new DataCacheFactory(config);
        }

        ~CacheHelper()
        {
            Dispose(false);
        }

        public void Dispose()
        {
            Dispose(true);
            GC.SuppressFinalize(this);
        }

        protected virtual void Dispose(bool disposing)
        {
        }

        /// 
        /// Gets a value from the cache in the specified region (class level)
        /// 
        /// Key linked to the data
        /// The found data. If null --> not found in the cache
        public string GetLookUpCacheData(string keyValue)
        {
            DataCacheFactory dcf = ConnectToCache();

            var cache = dcf.GetCache(_cacheName);
            string data = cache.Get(keyValue,_region) as string;

            return data;
        }

        /// 
        /// Store a value in the cache
        /// 
        /// Key
        /// Data
        public void StoreLookUpCacheData(string keyValue, object value)
        {
            DataCacheFactory dcf = ConnectToCache();

            var cache = dcf.GetCache(_cacheName);
            cache.Add(keyValue, value, _region);
        }

        /// 
        /// Stores a value in the cache for a specified amount of time
        /// 
        /// Key
        /// Data
        /// Time to keep the data in the cache
        public void StoreLookUpCacheData(string keyValue, object value,TimeSpan expires)
        {
            DataCacheFactory dcf = ConnectToCache();

            var cache = dcf.GetCache(_cacheName);
            cache.Add(keyValue, value, expires, _region);
        }        
    }

This is a very simple implementation that will store values in the default cache and in a specified region.

CodeTable Mapper

Codetable mapping is a very common scenario in BizTalk implementations. In my example we will be translating countrycodes to the country name.
The values are stored in a SQL table. But every time we get a value, we are going to save it to the AppFabric cache.
When we want to get the same value again, we are not going to the database but we will get the stored value from the AppFabric Cache.

public IBaseMessage Execute(IPipelineContext pContext, IBaseMessage pInMsg)
        {
            // Set variables
            biztalkMessage = pInMsg;
            XmlReader reader = XmlReader.Create(pInMsg.BodyPart.Data);
            XPathCollection xpaths = new XPathCollection();
            //For this example we are going to use 1 xpath expression
            xpaths.Add(this.XPath);

            ValueMutator vm = new ValueMutator(handleXpathFound);
            pInMsg.BodyPart.Data = new XPathMutatorStream(reader, xpaths, vm);
            return pInMsg;
        }

        private void handleXpathFound(int matchIdx, XPathExpression matchExpr, string origVal, ref string finalVal)
        {
            CacheHelper ch = new CacheHelper("Countries");
            string data = ch.GetLookUpCacheData(origVal);
            if (data == null)
            {
                finalVal = GetCountryFromDB(origVal);
                ch.StoreLookUpCacheData(origVal, finalVal);
            }
            else
                finalVal = ch.GetLookUpCacheData(origVal);
        }

        private string GetCountryFromDB(string countryCode)
        {
            string country = string.Empty;
            SqlConnection conn = null;
            SqlCommand comm = null;

            try
            {
                //Connect to look up database and retrieve the names of the products.
                conn = new SqlConnection("Data Source=(local);Initial Catalog=CacheTest;Integrated Security=SSPI;");
                conn.Open();

                comm = new SqlCommand();
                comm.Connection = conn;
                comm.CommandText = string.Format("SELECT Country FROM Countries WHERE CountryCode='{0}'", countryCode);
                comm.CommandType = CommandType.Text;

                country = (string)comm.ExecuteScalar();
                if(string.IsNullOrEmpty(country))
                    throw new Exception(string.Format("No country found for code {0}",countryCode));
            }
            catch (Exception e)
            {
                throw new Exception(e.Message + e.StackTrace);
            }
            finally
            {
                comm.Dispose();
                conn.Close();
                conn.Dispose();
            }


            return country;
        }

Duplicate Message Checker

As a sample scenario I took one I read about a few months ago. BizTalk had to stop messages that come in multiple times within 2 minutes.
So if there are more then 2 minutes between the messages, they should continue.

Normally this would involve a SQL table to store some information and some job to do the cleanup of this table.
But for my example I use AppFabric cache. There you have the option to store something in the cache for a certain timespan.
It is automatically deleted after this period.

public Microsoft.BizTalk.Message.Interop.IBaseMessage Execute(IPipelineContext pContext, Microsoft.BizTalk.Message.Interop.IBaseMessage pInMsg)
        {
            //Create hash
            VirtualStream input = new VirtualStream(pInMsg.BodyPart.GetOriginalDataStream());
            MD5 md5 = MD5.Create();
            byte[] hash = md5.ComputeHash(input);
            string hashString = Convert.ToBase64String(hash);

            //check Cache
            CacheHelper ch = new CacheHelper("DuplicateMessages");
            string date=ch.GetLookUpCacheData(hashString);
            if (string.IsNullOrEmpty(date))
            {
                //If not in cache yet, store it --> lifetime is 2 minutes
                ch.StoreLookUpCacheData(hashString, DateTime.Now.ToString(), new TimeSpan(0, 2, 0));
            }
            else
            {
                //Throw error
                throw new ApplicationException(string.Format("Duplicate Message. Already received at {0}",date));
            }

            //Put stream back to beginning
            input.Seek(0, SeekOrigin.Begin);
            return pInMsg;
        }

This makes the implementation very easy and you will not need a SQL table or anything to store the information.
You could say that you can do this with a custom caching solution as well. But what about HA environments with multiple BizTalk servers?
AppFabric is a distributed cache. So it doesn't mather on which server the message is processed. It will end up in the same cache and will be accessible on all the servers.

Conclusion

As you see, AppFabric caching has some advantages in BizTalk as well. The API is very easy to use and I got this to work quite quickly.

Tim D'haeyer, CODit