The journey towards resumable downloads on Windows Phone

First of all, most of you would advise me to use the Background Transfers that are available in Windows Phone. And you are right! Yes you are.

But we can’t initiate a Background Transfer from a Background Agent. I still want do download files in the Background Agent. I know the standard Background Agents won’t help, because they can’t be run any longer than 25 seconds. However we still have the Resource Intensive Agents. And yes the constraints for Resource Intensive are even harder to meet. But when the constrains are met, it can run for 10 minutes. For your information these are the hard constraints to meet.

  • External power is required
  • A WiFi connection, or connected to the internet through a pc
  • The device’s battery level is greater than 90%
  • The screen is locked
  • No active phone call

 

Yes I know, in practice these constraints are only met when the phone is on charger during the night. But still it will happen. So I want to download large files, sometimes the download won’t even complete in the 10 minutes that are available. Which could be cause by a slow or medium WiFi/Internet connection in combination with very large files. So I would like to continue the unfinished downloads the next time the Resource Intensive agent gets to run. So I’ve been working on a solution.

Resumable Downloads

Yes I found a solution, you can find it at the end of this post, but you can read my journey as well. There are interesting parts in my journey, but that’s up to you to read or ignore.

Cheapest request to know the size of the download

I started with a little bit of experimentation, just a simple console app. I know that’s not completely equal to a Windows Phone app, specially a Windows Phone Background Agent. But it’s easy experimenting and because I wanted to use the HttpClient, I expected them to be largely equal. They were equal for the API parts, but not the implementation. But back to my idea, I want to know the size of the download I’m about to start. The cheapest way to get to know this, is by a HTTP HEAD request, instead of a HTTP GET. The HEAD request is identical to GET, but it doesn’t contain the body, being the file in my case.

So how do we do a HTTP Head request, and get the Content Length Header? It’s as easy as the following piece of code.

public static async Task<long?> GetRemoteSize(HttpClient client, Uri uri)
{
    var headRequest = new HttpRequestMessage(HttpMethod.Head, uri);
    HttpResponseMessage response = await client.SendAsync(headRequest);
    return response.Content.Headers.ContentLength;
}

 

It’s interesting code, however for Windows Phone you can immediately forget this. It works in the Command line app, but the ContentLength header is somehow not available for Windows Phone when doing a HEAD request, it’s always returning 0. This could be caused by the HttpClient implementation, or it’s maybe at a lower level, I don’t know.

If I modify the method above, to a HTTP GET request where I ask the method to complete as early as when the headers are read. I don’t want to wait for the whole 80 MiB file to be downloaded, just to know the size of the Download.

private async Task<long?> GetRemoteSize(HttpClient client, Uri uri)
{
    var headRequest = new HttpRequestMessage(HttpMethod.Get, uri);
    HttpResponseMessage response = await client.SendAsync(headRequest, HttpCompletionOption.ResponseHeadersRead);
    return response.Content.Headers.ContentLength;
}

That’s working, nice! So I can at least get the length of the download when I start the download. That will enable me to check if they bytes I’ve written to the file are the complete download. I will not make this call as a separate method call, like the method above. This was just experimentation.

The Range Header, or better the Header that enables a partial download aka resume

Let’s say we’ve been downloading for 10 minutes, but the download did not finish yet. We’ve already downloaded 45 MiB of the in total 80 MiB file. We want to continue with the download as soon as our Resource Intensive Agent gets run. So we need to tell the remote server which of the bytes we want from them. Here we have the HTTP Range header to help us out. It’s not available via method or properties, but we can set it very easily. So in below example I wanted to download all the content starting from 45 MiB. The HttpRequestMessage would read like this.

var readRequest = new HttpRequestMessage(HttpMethod.Get, originToDownload);
readRequest.Headers.Add("Range", string.Format("bytes={0}-", 45*1024*1024));

 

The range that you add to the Range header can end with a dash when you want to get all the remaining bytes. This works fine on the Console application, but it fires an InvalidOperationException telling “Nullable object must have a value” on Windows Phone. Sounds like the upper range being empty isn’t supported on Windows Phone.

I did investigate it a little bit, but not too much. I now fill the upper part of the range with long.MaxValue and that seems to work.

var readRequest = new HttpRequestMessage(HttpMethod.Get, originToDownload);
readRequest.Headers.Add("Range", string.Format("bytes={0}-{1}", 45*1024*1024, long.MaxValue));

Be aware that the ContentLength header you’re getting in the response will contain the actual length of bytes that are sent. So in the above case it will show the amount of bytes remaining 45 MiB done of the 80 MiB, means 35 MiB in the Content and 36700160 (35*1024*1024) as ContentLength.

The Solution

If you combine the Range header with the knowledge we gained about getting the ContentLength we can write a method like this:

private async Task<bool> Download(HttpClient client, Uri originToDownload, string targetToDownloadTo)
{
    var readRequest = new HttpRequestMessage(HttpMethod.Get, originToDownload);

    StorageFile fileToDownloadTo = await ApplicationData.Current.LocalFolder
        .CreateFileInPathAsync(targetToDownloadTo, CreationCollisionOption.OpenIfExists);
    using (Stream writeStream = await fileToDownloadTo.OpenStreamForWriteAsync())
    {
        writeStream.Seek(0, SeekOrigin.End);
        long currentLength = writeStream.Length;
        if (currentLength > 0)
            readRequest.Headers.Add("Range",
                string.Format("bytes={0}-{1}", currentLength, long.MaxValue));
        using (HttpResponseMessage response =
            await client.SendAsync(readRequest, HttpCompletionOption.ResponseHeadersRead))
        {
            long? remoteSize = response.Content.Headers.ContentLength;
            long totalRead = 0L;
            if (remoteSize == 0)
            {
                //All bytes have already been read.
                return true;
            }
            using (Stream stream = await response.Content.ReadAsStreamAsync())
            {
                var buffer = new byte[1024*64];
                int bytesRead;
                do
                {
                    bytesRead = stream.Read(buffer, 0, buffer.Length);
                    writeStream.Write(buffer, 0, bytesRead);
                    await writeStream.FlushAsync();
                    totalRead += bytesRead;
                } while (bytesRead != 0);
            }
            if (remoteSize == totalRead)
            {
                return true;
            }
            return false;
        }
    }
}

If you want to copy paste, you will need the helper method to CreateFileInPathAsync.

public static async Task<StorageFile> CreateFileInPathAsync(this StorageFolder parentFolder, string path,
                                                            CreationCollisionOption collisionOption)
{
    string folderPath = path.Substring(0, path.LastIndexOf('/'));
    string fileName = path.Substring(path.LastIndexOf('/') + 1);
    StorageFolder targetFolder = await EnsureFolderExistsAsync(parentFolder, folderPath);
    if (targetFolder != null)
    {
        return await targetFolder.CreateFileAsync(fileName, collisionOption);
    }
    return null;
}

public static async Task<StorageFolder> EnsureFolderExistsAsync(this StorageFolder parentFolder, string path)
{
    StorageFolder currentFolder = parentFolder;
    var pathElements = path.Trim('/').Split('/');
    foreach (string name in pathElements)
    {
        currentFolder = await currentFolder.CreateFolderAsync(name, CreationCollisionOption.OpenIfExists);
    }
    return currentFolder;
}

Memory Usage

It’s very nice we now have a way to download large files, and when required resume them. However be careful around memory usage. In my application I already have the SQLite db-engine in memory, so there’s not much memory left. So even though you can download multiple files after each other the memory usage increases. I stop downloading more files when I’ve less than 1.5 MiB of memory left. Not sure if this will be the sweet spot for your app, I’ve tried keeping that thresshold smaller but got Out of Memory exceptions quite often. So you’ve been warned.

Final notes

Now I created a method to download large files and even resume downloads, I can’t recommend to use it in every place. Use it where appropriate and where the better alternative Background Transfers can’t be used. I can’t think of any place other than a background agent to use this. But there might be others.

In the end this is something that can be used in any other .NET application, with a small set of modifications maybe to suit the targeted platform.

  • Gravatar MSicc December 27th, 2013 at 22:54
    Great Article Mark! I already have an idea for using this as download indicator for even small requests like downloading data from Twitter or Azure etc. ;)
  • Gravatar Mark Monster December 27th, 2013 at 22:58
    Hi Marco,

    Love the way others come with ideas I didn't think off at all.

    Have fun!

    Best,

    Mark Monster
  • Gravatar Matt Lacey December 29th, 2013 at 20:53
    Hi Mark.
    A couple of years ago I built a WP7 app that downloaded very large movie files (0.7 to 1.6GB in size). In doing this I used the Range header to download the files in 64KB "chunks". This meant that we could easily measure progress and where we should resume from. It also meant it was easy to identify any problems in transmission (corruption on the wire). With large files it's good to not have to re-download the whole file if there's a problem with part of it. (This really matters if it's a large file and a slow connection.) Knowing that the same chunks of a file will also be downloaded means that caching can be improved and you only need to calculate check sums for specific chunks once. ;)

    Be aware that with this approach there are a couple of potential gotchas that others may need to be aware of.
    - Not all servers support range headers, check this if you're using a shared host.
    - There are still some old proxy servers around that don't support these headers and may strip them.
    In both the above situations the app needs to be able to handle downloading the whole file in one go. The whole file is returned in such instances.


    Great article though. Just wanted to help add to it.

    Matt.
  • Gravatar Mark Monster December 30th, 2013 at 08:42
    Hi Matt,

    Thanks for the additional information! This really adds up to the article. Specially the support for Range Headers on the server is very important.

    Interesting approach to do 64KB chunks.

    Best,

    Mark Monster
  • Gravatar rob February 13th, 2014 at 19:01
    Thanks a lot!

    Was banging my head for hours why the content length was always 0. Your solution works fine!
  • The celebration featured the Governor General laying the cornerstone of the Confederation
    Building, and the inauguration of the Carillon in the Peace Tower.
    Some firms charge depending on the mileage covered.
    So, if you are planning to hire such a company in
    Montreal, you will be overwhelmed with the wide selection of
    companies.
  • Gravatar Francesco April 15th, 2014 at 16:56
    Hello, if i use your code receive this error

    read is not supported on the main thread when buffering is disabled
Gravatar