Extending ASP.NET Core response compression with support for Brotli

The amount of transferred data matters. On one hand it often contributes to the cost of running a service and on the other a lot of clients doesn't have as fast connections as we would like to believe. This is why response compression is one of key performance mechanisms in web world.

There is a number of compression schemas (more or less popular) out there, so clients advertise the supported ones with Accept-Encoding header.

Above screenshot shows result of a request from Chrome to the simplest possible ASP.NET Core application.

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
    }

    public void Configure(IApplicationBuilder app, IHostingEnvironment env)
    {
        app.Run(async (context) =>
        {
            await context.Response.WriteAsync("-- Demo.AspNetCore.ResponseCompression.Brotli --");
        });
    }
}

As we can see the browser has advertised four different options of compressing the response but none has been used. This shouldn't be a surprise as ASP.NET Core is modular by its nature and leaves up to us picking the features we want. In order for compression to be supported we need to add a proper middleware.

Enabling response compression

The support for response compression in ASP.NET Core is available through ResponseCompressionMiddleware from Microsoft.AspNetCore.ResponseCompression package. After referencing the package all that needs to be done is registering middleware and related services.

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddResponseCompression();
    }

    public void Configure(IApplicationBuilder app, IHostingEnvironment env)
    {
        app.UseResponseCompression()
            .Run(async (context) =>
            {
                if (!StringValues.IsNullOrEmpty(context.Request.Headers[HeaderNames.AcceptEncoding]))
                    context.Response.Headers.Append(HeaderNames.Vary, HeaderNames.AcceptEncoding);

                context.Response.ContentType = "text/plain";
                await context.Response.WriteAsync("-- Demo.AspNetCore.ResponseCompression.Brotli --");
            });
    }
}

One thing to remember is setting Content-Type as compression is enabled only for specific MIME types (there is also a separated setting for enabling compression over HTTPS). Additionally I'm adding Vary: Accept-Encoding header to the response so any cache along the way knows the response needs to be cached per compression type (future version of middleware will handle this for us).

Below screenshot shows result of the same request as previously, after modifications.

Now the response has been compressed using gzip. Gzip compression is the only one supported by the middleware, which is "ok" in most cases as it has the widest support among clients. But the web world is constantly evolving and compression algorithms are no different. The latest-greatest seems to be Brotli which can shrink data by an additional 20% to 25%. It would be nice to use it in ASP.NET Core.

Extending response compression with Brotli

The ResponseCompressionMiddleware can be extended with additional compression algorithms by implementing ICompressionProvider interface. The interface is pretty simple, it has two properties (providing information about encoding token and if flushing is supported) and one method (which should create a stream with compression capabilities). The true challenge is the actual Brotli implementation. I've decided to use a .NET Core build of Brotli.NET. This is in fact a wrapper around original implementation, so some cross-platform issues might appear and force recompilation. The wrapper exposes the original implementation through BrotliStream which makes it very easy to use in context of ICompressionProvider.

public class BrotliCompressionProvider : ICompressionProvider
{
    public string EncodingName => "br";

    public bool SupportsFlush => true;

    public Stream CreateStream(Stream outputStream)
    {
        return new BrotliStream(outputStream, CompressionMode.Compress);
    }
}

The custom provider needs to be added to ResponseCompressionOptions.Providers collection as part of services registration.

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddResponseCompression(options =>
        {
            options.Providers.Add<BrotliCompressionProvider>();
        });
    }

    ...
}

Now the demo request can be done once again - it should show that Brotli is being used for compression.

Not every browser (and not always) supports Brotli

Lets take a quick look at compression support advertised by different browsers:

IE11: Accept-Encoding: gzip, deflate
Edge: Accept-Encoding: gzip, deflate
Firefox: Accept-Encoding: gzip, deflate (HTTP), Accept-Encoding: gzip, deflate, br (HTTPS)
Chrome: Accept-Encoding: gzip, deflate, sdch, br
Opera: Accept-Encoding:gzip, deflate, sdch, br

So IE and Edge don't support Brotli at all and Firefox supports it only over HTTPS. Checking more detailed information at caniuse we will learn that couple more browsers don't support Brotli (but Edge already has it in preview, although it is rumored that the final support will be only over HTTPS). The overall support is about 57% which means that we want to keep gzip around as well. In order to do so it needs to be added to ResponseCompressionOptions.Providers collection too (the moment we start manually registering providers the default one is gone).

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddResponseCompression(options =>
        {
            options.Providers.Add<BrotliCompressionProvider>();
            options.Providers.Add<GzipCompressionProvider>();
        });
    }

    ...
}

If we test this code against various browsers we will see that chosen compression always ends up being gzip. The reason for that is the way in which middleware chooses the provider. It takes the advertised compressions, sorts them by quality if present and chooses the first one for which provider exists. As browser generally don't provide any quality values (in another words they will be equally happy to accept any of the supported ones) the gzip always wins because it is always first on advertised list. Unfortunately the middleware doesn't provide an option for defining server side preference for such cases. In order to work around it I've decided to go the hacky way. If the only way to control provider selection is through quality values, they need to be adjusted before the response compression middleware kicks in. I've put together another middleware to do exactly that. The additional middleware would inspect the request Accept-Encoding header and if there are no quality values provided would adjust them.

public class ResponseCompressionQualityMiddleware
{
    private readonly RequestDelegate _next;
    private readonly IDictionary<string, double> _encodingQuality;

    public ResponseCompressionQualityMiddleware(RequestDelegate next, IDictionary<string, double> encodingQuality)
    {
        _next = next;
        _encodingQuality = encodingQuality;
    }

    public async Task Invoke(HttpContext context)
    {
        StringValues encodings = context.Request.Headers[HeaderNames.AcceptEncoding];
        IList<StringWithQualityHeaderValue> encodingsList;

        if (!StringValues.IsNullOrEmpty(encodings)
            && StringWithQualityHeaderValue.TryParseList(encodings, out encodingsList)
            && (encodingsList != null) && (encodingsList.Count > 0))
        {
            string[] encodingsWithQuality = new string[encodingsList.Count];

            for (int encodingIndex = 0; encodingIndex < encodingsList.Count; encodingIndex++)
            {
                // If there is any quality value provided don't change anything
                if (encodingsList[encodingIndex].Quality.HasValue)
                {
                    encodingsWithQuality = null;
                    break;
                }
                else
                {
                    string encodingValue = encodingsList[encodingIndex].Value;
                    encodingsWithQuality[encodingIndex] = (new StringWithQualityHeaderValue(encodingValue,
                        _encodingQuality.ContainsKey(encodingValue) ? _encodingQuality[encodingValue] : 0.1)).ToString();
                }

            }

            if (encodingsWithQuality != null)
                context.Request.Headers[HeaderNames.AcceptEncoding] = new StringValues(encodingsWithQuality);
        }

        await _next(context);
    }
}

This "adjusting" middleware needs to be registered before the response compression middleware and configured with tokens for which a preference is needed.

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddResponseCompression(options =>
        {
            options.Providers.Add<BrotliCompressionProvider>();
            options.Providers.Add<GzipCompressionProvider>();
        });
    }

    public void Configure(IApplicationBuilder app, IHostingEnvironment env)
    {
        app.UseMiddleware<ResponseCompressionQualityMiddleware>(new Dictionary<string, double>
            {
                { "br", 1.0 },
                { "gzip", 0.9 }
            })
            .UseResponseCompression()
            .Run(async (context) =>
            {
                if (!StringValues.IsNullOrEmpty(context.Request.Headers[HeaderNames.AcceptEncoding]))
                    context.Response.Headers.Append(HeaderNames.Vary, HeaderNames.AcceptEncoding);

                context.Response.ContentType = "text/plain";
                await context.Response.WriteAsync("-- Demo.AspNetCore.ResponseCompression.Brotli --");
            });
    }
}

Now the tests in different browsers will give different results. For example in case of Edge the response will be compressed with gzip but in case of Chrome with Brotli, which is the desired effect.