A small tale of bringing push notifications to Windows Azure

Some time ago I got more and more problems on my shared hosting because it couldn’t handle the amount of push notifications I wanted to send. I heard my hosting provider tell me that I was quite often taking more than 90% of the CPU on the server. So I thought to give Windows Azure a try.

I wanted to make optimal use of Windows Azure so I designed my solution to make use of Windows Azure Table Storage to store registered devices and pushchannels. When making use of Azure Table Storage it’s important to choose your RowKey and PartitionKey carefully.

To start with the PartitionKey, I chose to put the ApplicationName in there. I want to send push notifications to different applications I created, like Fokke & Sukke and iBood. So far I haven’t found a reason I made a wrong decision.

And now the RowKey, something that I need to use more carefully. The combination of PartitionKey and RowKey needs to be unique. So I wanted to put the DeviceId of the Device that should receive the PushNotification in there, that combined with the platform identifier. As far as I know there’s nothing that guarantees that the DeviceId is unqiue over different platforms. So I prefix the DeviceId with “WP|” for Windows Phone and “RT|” for Windows 8. The rest was just the copy of the DeviceId. I tested this using the emulator, and everything seems to work fine.

Windows Phone app trouble

After a while I notices reviews telling me that Push Notification don’t work, even further, it didn’t work on my own Windows Phone. After searching for many different reasons for this trouble, I found the source, more or less.

When saving an entity to the Azure Table Storage, every now and then a StorageException occurred. There aren’t many details in the exception, so after attaching Fiddler to my Nokia Lumia 920, I saw interesting stuff happening on the line.

The DeviceId contained special characters. I didn’t notice this when using the emulator, because the DeviceId on the emulator didn’t contain any special characters. So in total I had a percentage of users that could never register because of the StorageException, I still have no idea how large that percentage is, DeviceIds at least regularly contain the ‘/’ character.

Lesson learned, make sure the RowKey and PartitionKey don’t contain special characters: /, \, #, ?

Windows Style app trouble

Besides the Windows Phone issues, I had a very strange behavior on Windows 8 as well. It happened that I was sending a push notification to my local (installed through Visual Studio) app that did not appear. For example I sent a BadgeNotification with value 1 and sometimes the value 17 appeared. I have been trying to find the reason behind it, I never found it. Because when I tried to debug it explicitly with a value like 4 it did show 4. I never got feedback about issues with the push notifications on Windows 8 since my move to Windows Azure, but my dev-machine had troubles.

So after a couple of weeks trying to find causes for the problem I did something that was my final call. I did uninstall the app, and installed the app from the Store. What happened? The pushnotifications started behaving correctly. I have no understanding about the differences between the apps, but it’s good to be aware that there seem to be differences between the app installed from the store and the app installed by Visual Studio.

Cancel a Thread.Sleep uh Task.Delay the right way

To wait, or not to waitIn one of my Windows Azure Workers I’m using a back-off algorithm to expand the time between checking the queue for new messages. In my solution the time between checks can become as large as 5 minutes. While that helps in making sure I’m not creating unnecessary data transactions and limit the cpu-usage, it won’t help me when I’m deploying a new update. When a new deployment is done the old-running deployment will be gracefully shutdown. Meaning it waits until all processes are done. In the case of a sleep of 5 minutes, it will wait until it’s ready with that sleep instead of cancelling that sleeping. I thought that to be strange, so I was looking into a nice solution.

The solution is easy as long as you have a CancellationToken available to pass to the Task.Delay method. You can create a CancellationToken by using the CancellationTokenSource. I’ve always been using the overload that just takes one argument, but the overload that takes two arguments allows cancellation. The below sample application explains how to use it.

internal class Program
{
    private static void Main(string[] args)
    {
        var tokenSource = new CancellationTokenSource();
        Task consoleReadTask = Task.Run(
            () =>
                {
                    Console.WriteLine("Press the enter key to cancel execution of tasks.");
                    Console.ReadLine();
                    tokenSource.Cancel();
                });

        Task workerTask = Task.Run(
            async () =>
                        {
                            CancellationToken token = tokenSource.Token;
                            try
                            {
                                while (!token.IsCancellationRequested)
                                {
                                    Console.WriteLine("Task output @ {0}", DateTime.Now);
                                    await Task.Delay(60000, token);
                                }
                            }
                            catch (OperationCanceledException)
                            {
                                Console.WriteLine("Cancelled @ {0}", DateTime.Now);
                            }
                        });

        Task.WaitAll(consoleReadTask, workerTask);
        Console.WriteLine("Press the enter key to exit.");
        Console.ReadLine();
    }
}

Running multiple workers inside one Windows Azure Worker Role

Even the smallest Azure Worker Role, the Extra Small Instance, can be too large. At least too large for running just one piece of work on it. I’m currently running a couple small processes on one worker role and even now, the amount of CPU usage is below 1%. Yes you’re reading this correctly, I’m running multiple small processes on one worker role, and I’m planning to add more. I’m only using a small percentage of the resources on the worker role, so why should I add a worker role for every small process? I think we should be careful in answering this question, because there could be reasons enough, why you want to separate things. One reason could be that you never want any influence of one process on the other. This is something that’s unavoidable when running multiple processes on one worker role, but still not every process has such requirements.

Let’s start with my influence for the solution I’m currently using. It all started with this article: Running Multiple Threads on Windows Azure. It’s an excellent article, and that solution works really well. Until you start making use of async/await. Because their solution is based on threads that are monitored it’s hard or even impossible to make this work with async/await. Still this article was of great help, without it my solution would not have been possible.

First of all it’s important to know that I’m not going to paste all of the code in this blog post. I’m just going to explain the important bits, and the full solution can be downloaded for you to experiment with.

When you’re looking for a solution to run a Worker Role inside a Web Role you should definitely take a look at the solution Marcel Meijer created.

Independent workers

Let’s start with the easy part, the independent workers. You can have one, two, or ten. It doesn’t really matter that much to the solution, though you should monitor of course of your production Worker Role can handle all the load.

There’s a Start method and a Run method, and yes, there’s no Stop method. The stopping is done through a CancellationToken that’s signaled. You know best when you’re able to stop your worker, not in the middle of sequence of steps of course, so you should check if the CancellationToken is set at the place you think is the best place in your workers code. I’m usually checking at the beginning and the end of a sequence.

public class ExampleWorker1 : WorkerEntryPoint
{
    protected CancellationToken CancellationToken { get; set; }

    public override async Task<bool> OnStart(CancellationToken cancellationToken)
    {
        CancellationToken = cancellationToken;
        return await Task.FromResult(true);
    }

    public override async Task Run()
    {
        try
        {
            while (true)
            {
                ContinueOrCancel();
                Trace.TraceInformation("ExampleWorker1:Run");
                ContinueOrCancel();
                const int sleepTime = 5*1000;
                await Task.Delay(sleepTime);
            }
        }
        catch (OperationCanceledException)
        {
            Trace.TraceInformation("ExampleWorker1:Cancelled by cancellation token.");
        }
        catch (SystemException)
        {
            throw;
        }
        catch (Exception ex)
        {
            Trace.TraceError("ExampleWorker1:Run Exception", ex.ToString());
        }
    }

    private void ContinueOrCancel()
    {
        if (CancellationToken.IsCancellationRequested)
            CancellationToken.ThrowIfCancellationRequested();
    }
}

 

You can create multiple workers like the one above. To get started use the above template and implement your own logic at the highlighted line.

A very simple worker role

Of course you want to combine the multiple workers in the worker role. That’s easy, in the below example I’m just including 2 workers, but there could be more. You can even put multiple instances of the same type of worker in the list, as long as the workers support running simultaneously.

public class WorkerRole : TasksRoleEntryPoint
{
    public override void Run()
    {
        var workers = new List<WorkerEntryPoint>
                            {
                                new ExampleWorker1(),
                                new ExampleWorker2(),
                            };

        Run(workers.ToArray());
    }
}

Behind the scenes of the TasksRoleEntryPoint

All the magic is happening in the TasksRoleEntryPoint. When a worker is ended, outside of a cancellation, it will be automatically restarted.

/// <summary>
///     Called from WorkerRole, bringing in workers to add to tasks
/// </summary>
/// <param name="arrayWorkers">WorkerEntryPoint[] arrayWorkers</param>
public async void Run(WorkerEntryPoint[] arrayWorkers)
{
    try
    {
        _workers = arrayWorkers;

        foreach (WorkerEntryPoint worker in _workers)
        {
            await worker.OnStart(_tokenSource.Token);
        }

        foreach (WorkerEntryPoint worker in _workers)
        {
            _tasks.Add(worker.ProtectedRun());
        }

        int completedTaskIndex;
        while ((completedTaskIndex = Task.WaitAny(_tasks.ToArray())) != -1 && _tasks.Count > 0)
        {
            _tasks.RemoveAt(completedTaskIndex);
            //Not cancelled so rerun the worker
            if (!_tokenSource.Token.IsCancellationRequested)
            {
                _tasks.Insert(completedTaskIndex, _workers[completedTaskIndex].ProtectedRun());
                await Task.Delay(1000);
            }
        }
    }
    catch (Exception e)
    {
        Trace.TraceError(e.Message);
    }
}

 

And when the OnStop is called on the WorkerRole, it will fire Cancel on the CancelTokenSource and wait for all tasks to stop nicely.

public override void OnStop()
{
    try
    {
        _tokenSource.Cancel();
        Task.WaitAll(_tasks.ToArray());
    }
    catch (Exception e)
    {
        Trace.TraceError(e.Message);
    }
    base.OnStop();
}

 

Complete source

Because pasting all the code in this blogpost doesn’t make too much sense, I created a fully downloadable source-package. Hope this helps others as much as it helped me.

Book review: Microsoft Silverlight 5 and Windows Azure Enterprise Integration

Disclaimer: I received a copy of this book from Packt Publishing.

The author, David Burela, though I didn’t hear about him before seems to be well known in the Australian community.

When you look at the contents of the book it looks like it’s covering the main integration parts with Windows Azure.

- Hosting Silverlight Applications in Azure
- Using Azure Queues with Silverlight
- Accessing Azure Blob Storage from Silverlight
- Storing Data in Azure Table Storage from Silverlight
- Relational Data with SQL Azure and Entity Framework
- RIA Services and SQL Azure
- Exposing OData to Silverlight Applications
- Application Authentication

The author expects you to already have some Silverlight knowledge and thus won’t explain the Silverlight parts in too much detail. My experience with Windows Azure is small, which makes me probably part of the target audience. The introduction helps you understand what Azure is, a good refresher for me. After that, chapter after chapter there’s an explanation on how to integrate Silverlight with the backend, running on Azure. All in all it stays very basic and a couple of times it refers to some MSDN pages for the details. Though I agree that copying information that’s already on the web is probably not a good thing, I was reading the book on the bench and this means, no details for me at the time. I would rather have a book that is complete and puts these kind of additional parts in an Appendix.

In the end I didn’t feel satisfied. Most of the integration was more like, you integrate with WCF and that WCF will integrate with Windows Azure. This doesn’t have anything to do with the book itself, the content is good. However is a book like this really needed? You have to judge that yourself. It is an engaging book that will introduce Windows Azure to Silverlight developers.

Microsoft DevDays 2009 – I’m attending!

DevDays 2009 This year will be another year I’m attending the Microsoft DevDays. DevDays 2009 will be held in The Hague, The Netherlands. This year my focus will be largely to attend Silverlight sessions, but also Windows Azure and ASP.NET MVC are part of the sessions I’m interested in.

Sessions I’m attending on day 1 – Thursday 28th of May:

My Sessions for DevDays 2009 Day 1

Sessions I’m attending on day 2 – Friday 29th of May:

My Sessions for DevDays 2009 Day 2

Are you also attending the Microsoft DevDays? And what sessions are you going to attend?