Running multiple workers inside one Windows Azure Worker Role

Even the smallest Azure Worker Role, the Extra Small Instance, can be too large. At least too large for running just one piece of work on it. I’m currently running a couple small processes on one worker role and even now, the amount of CPU usage is below 1%. Yes you’re reading this correctly, I’m running multiple small processes on one worker role, and I’m planning to add more. I’m only using a small percentage of the resources on the worker role, so why should I add a worker role for every small process? I think we should be careful in answering this question, because there could be reasons enough, why you want to separate things. One reason could be that you never want any influence of one process on the other. This is something that’s unavoidable when running multiple processes on one worker role, but still not every process has such requirements.

Let’s start with my influence for the solution I’m currently using. It all started with this article: Running Multiple Threads on Windows Azure. It’s an excellent article, and that solution works really well. Until you start making use of async/await. Because their solution is based on threads that are monitored it’s hard or even impossible to make this work with async/await. Still this article was of great help, without it my solution would not have been possible.

First of all it’s important to know that I’m not going to paste all of the code in this blog post. I’m just going to explain the important bits, and the full solution can be downloaded for you to experiment with.

When you’re looking for a solution to run a Worker Role inside a Web Role you should definitely take a look at the solution Marcel Meijer created.

Independent workers

Let’s start with the easy part, the independent workers. You can have one, two, or ten. It doesn’t really matter that much to the solution, though you should monitor of course of your production Worker Role can handle all the load.

There’s a Start method and a Run method, and yes, there’s no Stop method. The stopping is done through a CancellationToken that’s signaled. You know best when you’re able to stop your worker, not in the middle of sequence of steps of course, so you should check if the CancellationToken is set at the place you think is the best place in your workers code. I’m usually checking at the beginning and the end of a sequence.

public class ExampleWorker1 : WorkerEntryPoint
{
    protected CancellationToken CancellationToken { get; set; }

    public override async Task<bool> OnStart(CancellationToken cancellationToken)
    {
        CancellationToken = cancellationToken;
        return await Task.FromResult(true);
    }

    public override async Task Run()
    {
        try
        {
            while (true)
            {
                ContinueOrCancel();
                Trace.TraceInformation("ExampleWorker1:Run");
                ContinueOrCancel();
                const int sleepTime = 5*1000;
                await Task.Delay(sleepTime);
            }
        }
        catch (OperationCanceledException)
        {
            Trace.TraceInformation("ExampleWorker1:Cancelled by cancellation token.");
        }
        catch (SystemException)
        {
            throw;
        }
        catch (Exception ex)
        {
            Trace.TraceError("ExampleWorker1:Run Exception", ex.ToString());
        }
    }

    private void ContinueOrCancel()
    {
        if (CancellationToken.IsCancellationRequested)
            CancellationToken.ThrowIfCancellationRequested();
    }
}

 

You can create multiple workers like the one above. To get started use the above template and implement your own logic at the highlighted line.

A very simple worker role

Of course you want to combine the multiple workers in the worker role. That’s easy, in the below example I’m just including 2 workers, but there could be more. You can even put multiple instances of the same type of worker in the list, as long as the workers support running simultaneously.

public class WorkerRole : TasksRoleEntryPoint
{
    public override void Run()
    {
        var workers = new List<WorkerEntryPoint>
                            {
                                new ExampleWorker1(),
                                new ExampleWorker2(),
                            };

        Run(workers.ToArray());
    }
}

Behind the scenes of the TasksRoleEntryPoint

All the magic is happening in the TasksRoleEntryPoint. When a worker is ended, outside of a cancellation, it will be automatically restarted.

/// <summary>
///     Called from WorkerRole, bringing in workers to add to tasks
/// </summary>
/// <param name="arrayWorkers">WorkerEntryPoint[] arrayWorkers</param>
public async void Run(WorkerEntryPoint[] arrayWorkers)
{
    try
    {
        _workers = arrayWorkers;

        foreach (WorkerEntryPoint worker in _workers)
        {
            await worker.OnStart(_tokenSource.Token);
        }

        foreach (WorkerEntryPoint worker in _workers)
        {
            _tasks.Add(worker.ProtectedRun());
        }

        int completedTaskIndex;
        while ((completedTaskIndex = Task.WaitAny(_tasks.ToArray())) != -1 && _tasks.Count > 0)
        {
            _tasks.RemoveAt(completedTaskIndex);
            //Not cancelled so rerun the worker
            if (!_tokenSource.Token.IsCancellationRequested)
            {
                _tasks.Insert(completedTaskIndex, _workers[completedTaskIndex].ProtectedRun());
                await Task.Delay(1000);
            }
        }
    }
    catch (Exception e)
    {
        Trace.TraceError(e.Message);
    }
}

 

And when the OnStop is called on the WorkerRole, it will fire Cancel on the CancelTokenSource and wait for all tasks to stop nicely.

public override void OnStop()
{
    try
    {
        _tokenSource.Cancel();
        Task.WaitAll(_tasks.ToArray());
    }
    catch (Exception e)
    {
        Trace.TraceError(e.Message);
    }
    base.OnStop();
}

 

Complete source

Because pasting all the code in this blogpost doesn’t make too much sense, I created a fully downloadable source-package. Hope this helps others as much as it helped me.

Gravatar