The amount of transferred data matters. On one hand it often contributes to the cost of running a service and on the other a lot of clients doesn't have as fast connections as we would like to believe. This is why response compression is one of key performance mechanisms in web world.

There is a number of compression schemas (more or less popular) out there, so clients advertise the supported ones with Accept-Encoding header.

Chrome Network Tab - No Response Compression

Above screenshot shows result of a request from Chrome to the simplest possible ASP.NET Core application.

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
    }

    public void Configure(IApplicationBuilder app, IHostingEnvironment env)
    {
        app.Run(async (context) =>
        {
            await context.Response.WriteAsync("-- Demo.AspNetCore.ResponseCompression.Brotli --");
        });
    }
}

As we can see the browser has advertised four different options of compressing the response but none has been used. This shouldn't be a surprise as ASP.NET Core is modular by its nature and leaves up to us picking the features we want. In order for compression to be supported we need to add a proper middleware.

Enabling response compression

The support for response compression in ASP.NET Core is available through ResponseCompressionMiddleware from Microsoft.AspNetCore.ResponseCompression package. After referencing the package all that needs to be done is registering middleware and related services.

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddResponseCompression();
    }

    public void Configure(IApplicationBuilder app, IHostingEnvironment env)
    {
        app.UseResponseCompression()
            .Run(async (context) =>
            {
                if (!StringValues.IsNullOrEmpty(context.Request.Headers[HeaderNames.AcceptEncoding]))
                    context.Response.Headers.Append(HeaderNames.Vary, HeaderNames.AcceptEncoding);

                context.Response.ContentType = "text/plain";
                await context.Response.WriteAsync("-- Demo.AspNetCore.ResponseCompression.Brotli --");
            });
    }
}

One thing to remember is setting Content-Type as compression is enabled only for specific MIME types (there is also a separated setting for enabling compression over HTTPS). Additionally I'm adding Vary: Accept-Encoding header to the response so any cache along the way knows the response needs to be cached per compression type (future version of middleware will handle this for us).

Below screenshot shows result of the same request as previously, after modifications.

Chrome Network Tab - Gzip Compression

Now the response has been compressed using gzip. Gzip compression is the only one supported by the middleware, which is "ok" in most cases as it has the widest support among clients. But the web world is constantly evolving and compression algorithms are no different. The latest-greatest seems to be Brotli which can shrink data by an additional 20% to 25%. It would be nice to use it in ASP.NET Core.

Extending response compression with Brotli

The ResponseCompressionMiddleware can be extended with additional compression algorithms by implementing ICompressionProvider interface. The interface is pretty simple, it has two properties (providing information about encoding token and if flushing is supported) and one method (which should create a stream with compression capabilities). The true challenge is the actual Brotli implementation. I've decided to use a .NET Core build of Brotli.NET. This is in fact a wrapper around original implementation, so some cross-platform issues might appear and force recompilation. The wrapper exposes the original implementation through BrotliStream which makes it very easy to use in context of ICompressionProvider.

public class BrotliCompressionProvider : ICompressionProvider
{
    public string EncodingName => "br";

    public bool SupportsFlush => true;

    public Stream CreateStream(Stream outputStream)
    {
        return new BrotliStream(outputStream, CompressionMode.Compress);
    }
}

The custom provider needs to be added to ResponseCompressionOptions.Providers collection as part of services registration.

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddResponseCompression(options =>
        {
            options.Providers.Add<BrotliCompressionProvider>();
        });
    }

    ...
}

Now the demo request can be done once again - it should show that Brotli is being used for compression.

Chrome Network Tab - Brotli Compression

Not every browser (and not always) supports Brotli

Lets take a quick look at compression support advertised by different browsers:

  • IE11: Accept-Encoding: gzip, deflate
  • Edge: Accept-Encoding: gzip, deflate
  • Firefox: Accept-Encoding: gzip, deflate (HTTP), Accept-Encoding: gzip, deflate, br (HTTPS)
  • Chrome: Accept-Encoding: gzip, deflate, sdch, br
  • Opera: Accept-Encoding:gzip, deflate, sdch, br

So IE and Edge don't support Brotli at all and Firefox supports it only over HTTPS. Checking more detailed information at caniuse we will learn that couple more browsers don't support Brotli (but Edge already has it in preview, although it is rumored that the final support will be only over HTTPS). The overall support is about 57% which means that we want to keep gzip around as well. In order to do so it needs to be added to ResponseCompressionOptions.Providers collection too (the moment we start manually registering providers the default one is gone).

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddResponseCompression(options =>
        {
            options.Providers.Add<BrotliCompressionProvider>();
            options.Providers.Add<GzipCompressionProvider>();
        });
    }

    ...
}

If we test this code against various browsers we will see that chosen compression always ends up being gzip. The reason for that is the way in which middleware chooses the provider. It takes the advertised compressions, sorts them by quality if present and chooses the first one for which provider exists. As browser generally don't provide any quality values (in another words they will be equally happy to accept any of the supported ones) the gzip always wins because it is always first on advertised list. Unfortunately the middleware doesn't provide an option for defining server side preference for such cases. In order to work around it I've decided to go the hacky way. If the only way to control provider selection is through quality values, they need to be adjusted before the response compression middleware kicks in. I've put together another middleware to do exactly that. The additional middleware would inspect the request Accept-Encoding header and if there are no quality values provided would adjust them.

public class ResponseCompressionQualityMiddleware
{
    private readonly RequestDelegate _next;
    private readonly IDictionary<string, double> _encodingQuality;

    public ResponseCompressionQualityMiddleware(RequestDelegate next, IDictionary<string, double> encodingQuality)
    {
        _next = next;
        _encodingQuality = encodingQuality;
    }

    public async Task Invoke(HttpContext context)
    {
        StringValues encodings = context.Request.Headers[HeaderNames.AcceptEncoding];
        IList<StringWithQualityHeaderValue> encodingsList;

        if (!StringValues.IsNullOrEmpty(encodings)
            && StringWithQualityHeaderValue.TryParseList(encodings, out encodingsList)
            && (encodingsList != null) && (encodingsList.Count > 0))
        {
            string[] encodingsWithQuality = new string[encodingsList.Count];

            for (int encodingIndex = 0; encodingIndex < encodingsList.Count; encodingIndex++)
            {
                // If there is any quality value provided don't change anything
                if (encodingsList[encodingIndex].Quality.HasValue)
                {
                    encodingsWithQuality = null;
                    break;
                }
                else
                {
                    string encodingValue = encodingsList[encodingIndex].Value;
                    encodingsWithQuality[encodingIndex] = (new StringWithQualityHeaderValue(encodingValue,
                        _encodingQuality.ContainsKey(encodingValue) ? _encodingQuality[encodingValue] : 0.1)).ToString();
                }

            }

            if (encodingsWithQuality != null)
                context.Request.Headers[HeaderNames.AcceptEncoding] = new StringValues(encodingsWithQuality);
        }

        await _next(context);
    }
}

This "adjusting" middleware needs to be registered before the response compression middleware and configured with tokens for which a preference is needed.

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddResponseCompression(options =>
        {
            options.Providers.Add<BrotliCompressionProvider>();
            options.Providers.Add<GzipCompressionProvider>();
        });
    }

    public void Configure(IApplicationBuilder app, IHostingEnvironment env)
    {
        app.UseMiddleware<ResponseCompressionQualityMiddleware>(new Dictionary<string, double>
            {
                { "br", 1.0 },
                { "gzip", 0.9 }
            })
            .UseResponseCompression()
            .Run(async (context) =>
            {
                if (!StringValues.IsNullOrEmpty(context.Request.Headers[HeaderNames.AcceptEncoding]))
                    context.Response.Headers.Append(HeaderNames.Vary, HeaderNames.AcceptEncoding);

                context.Response.ContentType = "text/plain";
                await context.Response.WriteAsync("-- Demo.AspNetCore.ResponseCompression.Brotli --");
            });
    }
}

Now the tests in different browsers will give different results. For example in case of Edge the response will be compressed with gzip but in case of Chrome with Brotli, which is the desired effect.

In my previous post I've shown how HttpClient can be extended with payload encryption capabilities by providing support for aes128gcm encoding. In this post I'm going to extend Aes128GcmEncoding class with decoding capabilities.

Decoding at the high level

It shouldn't be a surprise that decoding is mostly about doing the opposite of encoding. This is why the DecodeAsync method is very similar to EncodeAsync.

public static class Aes128GcmEncoding
{
    public static async Task DecodeAsync(Stream source, Stream destination, Func<string, byte[]> keyLocator)
    {
        // Validation skipped for brevity
        ...

        CodingHeader codingHeader = await ReadCodingHeaderAsync(source);

        byte[] pseudorandomKey = HmacSha256(codingHeader.Salt, keyLocator(codingHeader.KeyId));
        byte[] contentEncryptionKey = GetContentEncryptionKey(pseudorandomKey);

        await DecryptContentAsync(source, destination,
            codingHeader.RecordSize, pseudorandomKey, contentEncryptionKey);
    }
}

The keyLocator parameter is a simple way for delegating the key management responsibility to the caller, the implementation expects a method for retrieving it based on key identifier without going into any further details. I have also decided to introduce a class for the coding header properties in order to make the code more readable.

Retrieving the coding header

As we already know the coding header contains three fields with constant length (salt, record size and key identifier length) and one with variable length (zero in extreme case). They can be retrieved one by one. The important thing is to validate the presence and size of every field, for this purpose I've split the reading in several smaller methods. Also the record size must be additionally validated as this implementation support smaller value than allowed by specification.

public static class Aes128GcmEncoding
{
    private static async Task<byte[]> ReadCodingHeaderBytesAsync(Stream source, int count)
    {
        byte[] bytes = new byte[count];
        int bytesRead = await source.ReadAsync(bytes, 0, count);
        if (bytesRead != count)
            throw new FormatException("Invalid coding header.");

        return bytes;
    }

    private static async Task<int> ReadRecordSizeAsync(Stream source)
    {
        byte[] recordSizeBytes = await ReadCodingHeaderBytesAsync(source, RECORD_SIZE_LENGTH);

        if (BitConverter.IsLittleEndian)
            Array.Reverse(recordSizeBytes);
        uint recordSize = BitConverter.ToUInt32(recordSizeBytes, 0);

        if (recordSize > Int32.MaxValue)]
            throw new NotSupportedException($"Maximum supported record size is {Int32.MaxValue}.");

        return (int)recordSize;
    }

    private static async Task<string> ReadKeyId(Stream source)
    {
        string keyId = null;

        int keyIdLength = source.ReadByte();

        if (keyIdLength == -1)
            throw new FormatException("Invalid coding header.");

        if (keyIdLength > 0)
        {
            byte[] keyIdBytes = await ReadCodingHeaderBytesAsync(source, keyIdLength);
            keyId = Encoding.UTF8.GetString(keyIdBytes);
        }

        return keyId;
    }

    private static async Task<CodingHeader> ReadCodingHeaderAsync(Stream source)
    {
        return new CodingHeader
        {
            Salt = await ReadCodingHeaderBytesAsync(source, SALT_LENGTH),
            RecordSize = await ReadRecordSizeAsync(source),
            KeyId = await ReadKeyId(source)
        };
    }
}

With the coding header retrieved the content can be decrypted.

Decrypting the content and retrieving the payload.

The pseudorandom key and content encryption key should be calculated in exactly the same way as during encryption. With those the records can be read and decrypted. The operation should be done record by record (as mentioned in previous post the nonce guards the order) until last record is reached, where reaching last record means not only the end of content but must be confirmed by that last record being delimited with 0x02 byte.

The tricky part is extracting the data from the record. In order to do that we need to detect the location of the delimiter and make sure it meets all the requirements. All the records must be of equal length (except the last one) but they doesn't have to contain the same amount of data as there can be padding consisting of any number of 0x00 bytes at the end. This is something which I haven't included into encryption implementation but must be correctly handled here. So delimiter should be the first byte from the end which value is not 0x00. As explained in previous post there are two valid delimiters: 0x01 (for all the records except the last one) and 0x02 (for the last record). Any other delimiter means that record is invalid, also a record which contains only padding is invalid. Below method ensure all those conditions are met.

public static class Aes128GcmEncoding
{
    private static int GetRecordDelimiterIndex(byte[] plainText, int recordDataSize)
    {
        int recordDelimiterIndex = -1;
        for (int plaintTextIndex = plainText.Length - 1; plaintTextIndex >= 0; plaintTextIndex--)
        {
            if (plainText[plaintTextIndex] == 0)
                continue;

            if ((plainText[plaintTextIndex] == RECORD_DELIMITER)
                || (plainText[plaintTextIndex] == LAST_RECORD_DELIMITER))
            {
                recordDelimiterIndex = plaintTextIndex;
            }

            break;
        }

        if ((recordDelimiterIndex == -1)
            || ((plainText[recordDelimiterIndex] == RECORD_DELIMITER)
                && ((plainText.Length -1) != recordDataSize)))
        {
            throw new FormatException("Invalid record delimiter.");
        }

        return recordDelimiterIndex;
    }
}

With this method content decryption can be implemented.

public static class Aes128GcmEncoding
{
    private static async Task DecryptContentAsync(Stream source, Stream destination, int recordSize, byte[] pseudorandomKey, byte[] contentEncryptionKey)
    {
        GcmBlockCipher aes128GcmCipher = new GcmBlockCipher(new AesFastEngine());

        ulong recordSequenceNumber = 0;

        byte[] cipherText = new byte[recordSize];
        byte[] plainText = null;
        int recordDataSize = recordSize - RECORD_OVERHEAD_SIZE;
        int recordDelimiterIndex = 0;

        do
        {
            int cipherTextLength = await source.ReadAsync(cipherText, 0, cipherText.Length);
            if (cipherTextLength == 0)
                throw new FormatException("Invalid records order or missing record(s).");

            aes128GcmCipher.Reset();
            AeadParameters aes128GcmParameters = new AeadParameters(new KeyParameter(contentEncryptionKey),
                128, GetNonce(pseudorandomKey, recordSequenceNumber));
            aes128GcmCipher.Init(false, aes128GcmParameters);

            byte[] plainText = new byte[aes128GcmCipher.GetOutputSize(cipherText.Length)];
            int lenght = aes128GcmCipher.ProcessBytes(cipherText, 0, cipherText.Length, plainText, 0);
            aes128GcmCipher.DoFinal(plainText, lenght);

            recordDelimiterIndex = GetRecordDelimiterIndex(plainText, recordDataSize);

            if ((plainText[recordDelimiterIndex] == LAST_RECORD_DELIMITER) && (source.ReadByte() != -1))
                throw new FormatException("Invalid records order or missing record(s).");

            await destination.WriteAsync(plainText, 0, recordDelimiterIndex);
        }
        while (plainText[recordDelimiterIndex] != LAST_RECORD_DELIMITER);
    }
}

HttpClient plumbing

With the decoding implementation ready the components required by HttpClient can be prepared. I've decided to reuse the same wrapping pattern as with Aes128GcmEncodedContent.

public sealed class Aes128GcmDecodedContent : HttpContent
{
    private readonly HttpContent _contentToBeDecrypted;
    private readonly Func<string, byte[]> _keyLocator;

    public Aes128GcmDecodedContent(HttpContent contentToBeDecrypted, Func<string, byte[]> keyLocator)
    {
        _contentToBeDecrypted = contentToBeDecrypted;
        _keyLocator = keyLocator;
    }

    protected override async Task SerializeToStreamAsync(Stream stream, TransportContext context)
    {
        if (!_contentToBeDecrypted.Headers.ContentEncoding.Contains("aes128gcm"))
            throw new NotSupportedException($"Encryption type not supported or stream isn't encrypted.");

        Stream streamToBeDecrypted = await _contentToBeDecrypted.ReadAsStreamAsync();

        await Aes128GcmEncoding.DecodeAsync(streamToBeDecrypted, stream, _keyLocator);
    }

    protected override bool TryComputeLength(out long length)
    {
        length = 0;

        return false;
    }
}

But this time it is not our code which is creating the content object - it comes from response. In order to wrap the content coming from response the HttpCLient pipeline needs to be extended with DelegatingHandler which will take care of that upon detecting desired Content-Encoding header value. The DelegatingHandler also gives an opportunity for setting Accept-Encoding header so the other side knows that encrypted content is supported.

public sealed class Aes128GcmEncodingHandler : DelegatingHandler
{
    private readonly Func<string, byte[]> _keyLocator;

    public Aes128GcmEncodingHandler(Func<string, byte[]> keyLocator)
    {
        _keyLocator = keyLocator;
    }

    protected override async Task SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        request.Headers.AcceptEncoding.Add(new StringWithQualityHeaderValue("aes128gcm"));

        HttpResponseMessage response = await base.SendAsync(request, cancellationToken);

        if (response.Content.Headers.ContentEncoding.Contains("aes128gcm"))
        {
            response.Content = new Aes128GcmDecodedContent(response.Content, _keyLocator);
        }

        return response;
    }
}

With those components in place we can try requesting some encrypted content from server.

Test run

To see decryption in action the HttpClient pipeline needs to be set to use components created above (assuming the server will respond with encrypted content).

IDictionary<string, byte[]> _keys = new Dictionary<string, byte[]>
{
    { String.Empty, Convert.FromBase64String("yqdlZ+tYemfogSmv7Ws5PQ==") },
    { "a1", Convert.FromBase64String("BO3ZVPxUlnLORbVGMpbT1Q==") }
};
Func<string, byte[]> keyLocator = (keyId) => _keys[keyId ?? String.Empty];

HttpMessageHandler encryptedContentEncodingPipeline = new HttpClientHandler();
encryptedContentEncodingPipeline = new Aes128GcmEncodingHandler(keyLocator)
{
    InnerHandler = encryptedContentEncodingPipeline
};

using (HttpClient encryptedContentEncodingClient = new HttpClient(encryptedContentEncodingPipeline))
{
    string decryptedContent = encryptedContentEncodingClient.GetStringAsync("<URL>").Result;
}

This gives full support for aes128gcm content encoding in HttpClient. All the code is available here for anybody who would like to play with it.

When you hear HTTP and encryption typically first thought goes to SSL. SSL allows for encoding entire communication between client and server, but what if there is a need to encrypt the content in a way that the other side can only store it and whoever will request it in the future will need a proper key to decrypt it? The "Encrypted Content-Encoding for HTTP" (in version 07 at the moment of writing this) aims at providing standard solution for such scenarios. In this and next post I'm going to show how it can be used with HttpClient.

The "aes128gcm" encoding

The "Encrypted Content-Encoding for HTTP" introduces new value for Content-Encoding header - aes128gcm. This encoding allows for transferring encrypted data together with information necessary to decrypt them when somebody knows the key. The encoded body consists of a coding header and encrypted content represented by number of fixed size encrypted records (last record can be smaller than others and there is a basic mechanism for preventing removal or reordering of records). For the encryption purposes the AES in Galois/Counter mode with 128 bit key is being used.

Adding encoding capability to HttpClient

What I want to achieve is support for aes128gcm encoding on top of any content type. From HttpClient perspective it seems like something that can be easily achieved with custom HttpContent which would server as wrapper over other ones.

public sealed class Aes128GcmEncodedContent : HttpContent
{
    private readonly HttpContent _contentToBeEncrypted;
    private readonly byte[] _key;
    private readonly string _keyId;
    private readonly int _recordSize;

    public Aes128GcmEncodedContent(HttpContent contentToBeEncrypted, byte[] key, string keyId, int recordSize)
    {
        _contentToBeEncrypted = contentToBeEncrypted;

        Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
        Headers.ContentEncoding.Add("aes128gcm");
    }
}

The key, key identifier and record size are parameters which control the encoding algorithm so they will be needed. I'm also setting Content-Encoding header to aes128gcm and Content-Type to application/octet-stream. The second one is suggested by specification but there is also an option of skipping Content-Type header - the goal is to prevent exposure of original Content-Type. In order to add the encoding I've decided to override the SerializeToStreamAsync method which should allow me to perform work over streams whenever possible.

public sealed class Aes128GcmEncodedContent : HttpContent
{
    ...

    protected override async Task SerializeToStreamAsync(Stream stream, TransportContext context)
    {
        Stream streamToBeEncrypted = await _contentToBeEncrypted.ReadAsStreamAsync();

        await Aes128GcmEncoding.EncodeAsync(streamToBeEncrypted, stream, _key, _keyId, _recordSize);
    }

    protected override bool TryComputeLength(out long length)
    {
        length = 0;

        return false;
    }
}

Implementing the encoding algorithm

At high level the encoding algorithm consists of just a few steps, so it is simpler to show the implementation and then describe each of the steps.

public static class Aes128GcmEncoding
{
    public static async Task EncodeAsync(Stream source, Stream destination, byte[] key, string keyId, int recordSize)
    {
        ...

        if ((key == null) || (key.Length != 16))
            throw new ArgumentException(
                $"The '{nameof(key)}' parameter must be 16 octets long.", nameof(key));

        if (recordSize < 18)
            throw new ArgumentException(
                $"The '{nameof(recordSize)}' parameter must be at least 18.", nameof(recordSize));

        byte[] salt = GenerateSalt();

        byte[] pseudorandomKey = HmacSha256(salt, key);
        byte[] contentEncryptionKey = GetContentEncryptionKey(pseudorandomKey);

        await WriteCodingHeaderAsync(destination, salt, keyId, recordSize);

        await EncryptContentAsync(source, destination, recordSize, pseudorandomKey, contentEncryptionKey);
    }
}

First some validations must be performed (I've skipped the null checks for brevity):

  • The key must be provided and must have exactly 128 bits. This comes from the encryption algorithm being used.
  • The record size must be at least 18 bytes. This comes from the fact that the encryption algorithm produces output longer by 16 bytes from the source and a delimiter byte is required (so if the record size will be 18 bytes it means we can put exactly 1 byte of data into it).

If the provided parameters are valid the salt can be generate.

public static class Aes128GcmEncoding
{
    ...

    private static readonly SecureRandom _secureRandom = new SecureRandom();

    ...

    private static byte[] GenerateSalt()
    {
        byte[] salt = new byte[16];
        _secureRandom.NextBytes(salt, 0, 16);

        return salt;
    }

    ...
}

The salt will be used to generate the content encryption key which will serve as the actual key for encryption. This is needed in order to prevent key exposure in cases when different content is being encrypted using the same keying material - without the random part this wouldn't be safe. In order to derive the content encryption key from salt and key the HKDF algorithm must be used. First step is calculating pseudorandom key which is a result of HMAC SHA-256 hash of salt with the key.

public static class Aes128GcmEncoding
{
    ...

    private static byte[] HmacSha256(byte[] key, byte[] value)
    {
        byte[] hash = null;

        using (HMACSHA256 hasher = new HMACSHA256(key))
        {
            hash = hasher.ComputeHash(value);
        }

        return hash;
    }

    ...
}

With pseudorandom key the content encryption key can be calculated as truncated to 16 bytes HMAC SHA-256 hash of pseudorandom key with content encryption key info parameter. The content encryption key info parameter is ASCII-encoded Content-Encoding: aes128gcm string terminated by 0x00 and 0x01 bytes.

public static class Aes128GcmEncoding
{
    ...

    private static readonly byte[] _contentEncryptionKeyInfoParameter;

    ...

    static Aes128GcmEncoding()
    {
        _contentEncryptionKeyInfoParameter = GetInfoParameter("Content-Encoding: aes128gcm");
    }

    ...

    private static byte[] GetInfoParameter(string infoParameterString)
    {
        byte[] infoParameter = new byte[infoParameterString.Length + 2];
        Encoding.ASCII.GetBytes(infoParameterString, 0, infoParameterString.Length, infoParameter, 0);

        infoParameter[infoParameter.Length - 1] = 1;

        return infoParameter;
    }

    private static byte[] GetContentEncryptionKey(byte[] pseudorandomKey)
    {
        byte[] contentEncryptionKey = HmacSha256(pseudorandomKey, _contentEncryptionKeyInfoParameter);
        Array.Resize(ref contentEncryptionKey, 16);

        return contentEncryptionKey;
    }

    ...
}

Those are all the prerequisites and now the body of the message can be created. First the header must be written and then the encrypted records.

Writing the coding header

The coding header has four fields: salt, record size, key identifier size and key identifier. The size of the header is variable as it depends on length of the key identifier, this is why the key identifier size must be provided. The size is being kept as a single byte which means that key identifier can take up to 255 bytes. The key identifier is expected to be a UTF-8 encoded string, so with a simple method we can transform it to array of bytes and verify the length.

public static class Aes128GcmEncoding
{
    ...

    private static byte[] GetKeyIdBytes(string keyId)
    {
        byte[] keyIdBytes = String.IsNullOrEmpty(keyId) ? new byte[0] : Encoding.UTF8.GetBytes(keyId);
        if (keyIdBytes.Length > Byte.MaxValue)
        {
            throw new ArgumentException($"The '{nameof(keyId)}' parameter is too long.", nameof(keyId));
        }

        return keyIdBytes;
    }

    ...
}

Second thing which needs to be transformed into byte array is the record size. There are 4 bytes reserved for record size in the header which means that it can store an unsigned 32-bit integer, but my implementation is limited to signed integer which makes it simpler (especially as some of the methods are available only over arrays which can't be bigger than 2GB).

public static class Aes128GcmEncoding
{
    ...

    private static byte[] GetRecordSizeBytes(int recordSize)
    {
        byte[] recordSizeBytes = BitConverter.GetBytes(recordSize);
        if (BitConverter.IsLittleEndian)
        {
            Array.Reverse(recordSizeBytes);
        }

        return recordSizeBytes;
    }

    ...
}

The header needs to be put together and written to the target stream.

public static class Aes128GcmEncoding
{
    ...

    private static async Task WriteCodingHeaderAsync(Stream destination, byte[] salt, string keyId, int recordSize)
    {
        byte[] keyIdBytes = GetKeyIdBytes(keyId);
        byte[] recordSizeBytes = GetRecordSizeBytes(recordSize);

        byte[] codingHeader = new byte[21 + keyIdBytes.Length];

        salt.CopyTo(codingHeader, 0);
        recordSizeBytes.CopyTo(codingHeader, 16);
        codingHeader[20] = (byte)keyIdBytes.Length;
        keyIdBytes.CopyTo(codingHeader, 21);

        await destination.WriteAsync(codingHeader, 0, codingHeader.Length);
    }

    ...
}

Encrypting the content and writing the records

The implementation of AES GCM is beyond the scope of a blog post and I'm not even going to attempt doing that, instead I've chosen to use Bouncy Castle. As mentioned previously the encrypted content must be represented by fixed size records (last can be shorter), which means that source data needs to be properly split. Also the last record must be properly detected because it should be terminated with 0x02 byte while all the previous ones should be terminated with 0x01 byte. Detecting last record is easy if the content doesn't split equally between records as the final read will return smaller number of bytes than requested. The situation when content does split equally is a little bit more tricky - it requires checking if stream can be further read before writing the record to the target. The safest way to do that seems to be reading a single byte in advance and then adding it to next record. I've started the implementation with a helper method which reads the required number of bytes, prepends it with that "peeked" byte and initially sets the delimiter based on number of returned bytes.

public static class Aes128GcmEncoding
{
    ...

    private static async Task GetPlainTextAsync(Stream source, int recordDataSize, byte? peekedByte)
    {
        int readDataSize;
        byte[] plainText = new byte[recordDataSize + 1];

        if (peekedByte.HasValue)
        {
            plainText[0] = peekedByte.Value;
            readDataSize = (await source.ReadAsync(plainText, 1, recordDataSize - 1)) + 1;
        }
        else
        {
            readDataSize = await source.ReadAsync(plainText, 0, recordDataSize);
        }

        if (readDataSize == recordDataSize)
        {
            plainText[plainText.Length - 1] = 1;
        }
        else
        {
            Array.Resize(ref plainText, readDataSize + 1);
            plainText[plainText.Length - 1] = 2;
        }

        return plainText;
    }

    ...
}

The number of bytes to read is being calculated from record size by subtracting the previously mentioned overhead (17 bytes). This allows for putting the core of encryption routine together.

public static class Aes128GcmEncoding
{
    ...

    private static async Task EncryptContentAsync(Stream source, Stream destination, int recordSize, byte[] pseudorandomKey, byte[] contentEncryptionKey)
    {
        GcmBlockCipher aes128GcmCipher = new GcmBlockCipher(new AesFastEngine());

        ulong recordSequenceNumber = 0;
        int recordDataSize = recordSize - 17;

        byte[] plainText = null;
        int? peekedByte = null;

        do
        {
            plainText = await GetPlainTextAsync(source, recordDataSize, (byte?)peekedByte);

            if (plainText[plainText.Length - 1] != 2)
            {
                peekedByte = source.ReadByte();
                if (peekedByte == -1)
                {
                    plainText[plainText.Length - 1] = 2;
                }
            }

            // TODO: Encrypt and write the record
        }
        while (plainText[plainText.Length - 1] != 2);
    }

    ...
}

The AES GCM requires one more parameter which needs to be calculate - nonce. The aes128gcm encoding uses nonce additionally for removal and reordering protection by performing a XOR with record sequence number as last step. The first argument for that XOR is the result of same the HKDF function as the one discussed in context of content encryption key, the difference is key info parameter (Content-Encoding: nonce) and length (12 bytes).

public static class Aes128GcmEncoding
{
    ...

    private static readonly byte[] _nonceInfoParameter;

    ...

    static Aes128GcmEncoding()
    {
        ...
        _nonceInfoParameter = GetInfoParameter("Content-Encoding: nonce");
    }

    ...

    private static byte[] GetNonce(byte[] pseudorandomKey, ulong recordSequenceNumber)
    {
        byte[] nonce = HmacSha256(pseudorandomKey, _nonceInfoParameter);
        Array.Resize(ref nonce, 12);

        byte[] recordSequenceNumberBytes = BitConverter.GetBytes(recordSequenceNumber);
        if (BitConverter.IsLittleEndian)
        {
            Array.Reverse(recordSequenceNumberBytes);
        }
        int leadingNullBytesCount = 12 - recordSequenceNumberBytes.Length;

        for (int i = 0; i < leadingNullBytesCount; i++)
        {
            nonce[i] = (byte)(nonce[i] ^ 0);
        }

        for (int i = 0; i < recordSequenceNumberBytes.Length; i++)
        {
            nonce[leadingNullBytesCount + i] =
                (byte)(nonce[leadingNullBytesCount + i] ^ recordSequenceNumberBytes[i]);
        }

        return nonce;
    }

    ...
}

With nonce calculated every record can be encrypted and written to the target.

public static class Aes128GcmEncoding
{
    ...

    private static async Task EncryptContentAsync(Stream source, Stream destination, int recordSize, byte[] pseudorandomKey, byte[] contentEncryptionKey)
    {
        GcmBlockCipher aes128GcmCipher = new GcmBlockCipher(new AesFastEngine());

        ulong recordSequenceNumber = 0;
        int recordDataSize = recordSize - 17;

        byte[] plainText = null;
        int? peekedByte = null;

        do
        {
            plainText = await GetPlainTextAsync(source, recordDataSize, (byte?)peekedByte);

            if (plainText[plainText.Length - 1] != 2)
            {
                peekedByte = source.ReadByte();
                if (peekedByte == -1)
                {
                    plainText[plainText.Length - 1] = LAST_RECORD_DELIMITER;
                }
            }

            aes128GcmCipher.Reset();
            AeadParameters aes128GcmParameters = new AeadParameters(new KeyParameter(contentEncryptionKey),
                128, GetNonce(pseudorandomKey, recordSequenceNumber));
            aes128GcmCipher.Init(true, aes128GcmParameters);

            byte[] cipherText = new byte[aes128GcmCipher.GetOutputSize(plainText.Length)];
            int lenght = aes128GcmCipher.ProcessBytes(plainText, 0, plainText.Length, cipherText, 0);
            aes128GcmCipher.DoFinal(cipherText, lenght);

            await destination.WriteAsync(cipherText, 0, cipherText.Length);
        }
        while (plainText[plainText.Length - 1] != 2);
    }

    ...
}

Take it for a spin

With all the pieces in place the Aes128GcmEncodedContent can be used to make an actual request.

using (HttpClient encryptedContentEncodingClient = new HttpClient())
{
    HttpContent contentToBeEncrypted = new StringContent("I am the walrus", Encoding.UTF8);

    byte[] key = Convert.FromBase64String("yqdlZ+tYemfogSmv7Ws5PQ==");
    HttpContent encryptedContent = new Aes128GcmEncodedContent(contentToBeEncrypted, key, null, 4096);

    await encryptedContentEncodingClient.PostAsync("<URL>", encryptedContent);
}

Both Aes128GcmEncodedContent and Aes128GcmEncoding can be grabbed directly from here.

In next post I'm going to focus on decoding part.

The web socket protocol is currently the most popular one for pushing data to browsers, however it's not the only one. The Server-Sent Events (SSE) is a very interesting alternative which can provide better performance for specific use cases.

What is SSE

The Server-Sent Events is a unidirectional (server to browser) protocol for streaming events. The protocol delivers text-based messages over a long-lived HTTP connection. It also has built in support for events identification, auto-reconnection (with tracking of last received event) and notifications through DOM events. Its biggest advantage is high performance as events can be pushed immediately with minimum overhead (there is an already open HTTP connection waiting, which thanks to text-based messages can utilize HTTP compression mechanisms). A considerable limitation is general lack of support for binary streaming (but JSON or XML will work nicely).

Why use SSE

In general web sockets can do everything that Server-Sent Events can and more as they provide bidirectional communication. There is also broader browser support (93%) for web sockets. So why would one consider the SSE (assuming bidirectional isn't a requirement, or the client to server communication is occasional and can be done in a REST style)? The fact that it runs over a long-lived HTTP connection is the game changer here. In case of web sockets we are talking about custom TCP based protocol which needs to be supported by the server and entire infrastructure (proxies, firewalls etc.), any legacy element along the way may cause an issue. There are no such issues for SSE, anything that speaks HTTP will speak SSE and the aspect of browser support (87%) can be addressed with polyfills. Taking into consideration this and notably lower latency, Server-Sent Events is a very compelling choice for scenarios like stock ticker or notifications.

Bringing SSE to ASP.NET Core

One of key concepts behind ASP.NET Core is modular HTTP request pipeline which can be extended through middlewares, so I'm going to create one for Server-Sent Events. But first some prerequisites are needed.

The middleware will require an abstraction for representing a client. As previously stated SSE runs over a long-lived HTTP connection, which means that channel for communication with client is HttpResponse instance. The abstraction will simply wrap around it.

public class ServerSentEventsClient
{
    private readonly HttpResponse _response;

    internal ServerSentEventsClient(HttpResponse response)
    {
        _response = response;
    }
}

Also there is a need for some kind of service which will serve as bridge between the middleware and the rest of application. Its primary goal will be managing the collection of connected clients. Below is a simple implementation based on ConcurrentDictionary.

public class ServerSentEventsService
{
    private readonly ConcurrentDictionary<Guid, ServerSentEventsClient> _clients = new ConcurrentDictionary<Guid, ServerSentEventsClient>();

    internal Guid AddClient(ServerSentEventsClient client)
    {
        Guid clientId = Guid.NewGuid();

        _clients.TryAdd(clientId, client);

        return clientId;
    }

    internal void RemoveClient(Guid clientId)
    {
        ServerSentEventsClient client;

        _clients.TryRemove(clientId, out client);
    }
}

With those elements in place the middleware can be created. It will have two responsibilities: establishing the connection and cleaning up when client closes the connection.

In order to establish the connection the middleware should inspect the Accept header of incoming request, if its value is text/event-stream it means that client is attempting to open SSE connection. In such case the Content-Type response header should be set to text/event-stream, headers should be send and connection needs to be kept open.

The clean up part requires detecting that client has closed the connection. This can be done by waiting on CancellationToken available through HttpContext.RequestAborted property. An important thing to note here is that closed connection can only be detected when sending new event. This limitation is often being solved by sending dedicated heartbeat event which client should simply ignore.

public class ServerSentEventsMiddleware
{
    private readonly RequestDelegate _next;
    private readonly ServerSentEventsService _serverSentEventsService;

    public ServerSentEventsMiddleware(RequestDelegate next, ServerSentEventsService serverSentEventsService)
    {
        _next = next;
        _serverSentEventsService = serverSentEventsService;
    }

    public Task Invoke(HttpContext context)
    {
        if (context.Request.Headers["Accept"] == "text/event-stream")
        {
            context.Response.ContentType = "text/event-stream";
            context.Response.Body.Flush();

            ServerSentEventsClient client = new ServerSentEventsClient(context.Response);
            Guid clientId = _serverSentEventsService.AddClient(client);

            context.RequestAborted.WaitHandle.WaitOne();

            _serverSentEventsService.RemoveClient(clientId);

            return Task.FromResult(true);
        }
        else
        {
            return _next(context);
        }
    }
}

With the connection management part in place the sending part can be added. The message format in SSE is a very simple one. The basic building blocks of every message are fields which general format looks like this: <FieldName>: <FieldValue>\n. There are three types of fields (well in fact four as there is an additional one for controlling client reconnect interval):

  • id - The identifier of the event.
  • event - The type of the event.
  • data - A single line of data (entire payload of message is represented by one or more adjacent data fields).

Only the data field is required and the entire message is being terminated by additional new line (\n).

public class ServerSentEvent
{
    public string Id { get; set; }

    public string Type { get; set; }

    public IList<string> Data { get; set; }
}

internal static class ServerSentEventsHelper
{
    internal static async Task WriteSseEventAsync(this HttpResponse response, ServerSentEvent serverSentEvent)
    {
        if (!String.IsNullOrWhiteSpace(serverSentEvent.Id))
            await response.WriteSseEventFieldAsync("id", serverSentEvent.Id);

        if (!String.IsNullOrWhiteSpace(serverSentEvent.Type))
            await response.WriteSseEventFieldAsync("event", serverSentEvent.Type);

        if (serverSentEvent.Data != null)
        {
            foreach(string data in serverSentEvent.Data)
                await response.WriteSseEventFieldAsync("data", data);
        }

        await response.WriteSseEventBoundaryAsync();
        response.Body.Flush();
    }

    private static Task WriteSseEventFieldAsync(this HttpResponse response, string field, string data)
    {
        return response.WriteAsync($"{field}: {data}\n");
    }

    private static Task WriteSseEventBoundaryAsync(this HttpResponse response)
    {
        return response.WriteAsync("\n");
    }
}

The above helper can be used in order to expose the send method on the client abstraction.

public class ServerSentEventsClient
{
    ...

    public Task SendEventAsync(ServerSentEvent serverSentEvent)
    {
        return _response.WriteSseEventAsync(serverSentEvent);
    }
}

Last step is exposing send method at the service level - it should perform send for all connected clients.

public class ServerSentEventsService
{
    ...

    public Task SendEventAsync(ServerSentEvent serverSentEvent)
    {
        List<Task> clientsTasks = new List<Task>();
        foreach (ServerSentEventsClient client in _clients.Values)
        {
            clientsTasks.Add(client.SendEventAsync(serverSentEvent));
        }

        return Task.WhenAll(clientsTasks);
    }
}

We can say that this gives us what project managers like to call minimum viable product. After extending pipeline with the middleware and adding service to services collection (as singleton) we can send events from any desired place in the application. In case of a need for exposing more than one endpoint a derived services can be created, added to services collection and passed to the respective middlewares during initialization.

I've made an extended version (support for reconnect interval, extensibility point for auto-reconnect and extensions for service and middleware registration) available on GitHub and as a NuGet package.

SSE at work

I've also created a demo application which utilizes the above components, it can be found here. The application exposes two SSE endpoints:

  • /see-heartbeat which can be "listened" by navigating to /sse-heartbeat-receiver.html. It sends an event every 5s and is implemented through an ugly background thread.
  • /sse-notifications which can be "listened" by navigating to /notifications/sse-notifications-receiver. Sending events to this endpoint can be done by navigating to /notifications/sse-notifications-sender.

It might be a good starting point for those who would like to play with what I've shown here.

This is one of those "I had to explain this couple times already so next time I want something I can redirect people to" kind of post.

What I want to write about is difference in behavior between using new() and DbSet.Create() for instantiating new entities. In order to do this I've created a very simple model and context.

public class Planet
{
    public virtual int Id { get; set; }

    public virtual string Name { get; set; }

    ...

    public virtual ICollection Natives { get; set; }
}

public class Character
{
    public virtual int Id { get; set; }

    public virtual string Name { get; set; }

    ...

    public virtual int HomeworldId { get; set; }

    public virtual Planet Homeworld { get; set; }
}

public interface IStarWarsContext
{
    DbSet Planets { get; set; }

    DbSet Characters { get; set; }

    int SaveChanges();
}

public class StarWarsContext : DbContext, IStarWarsContext
{
    public DbSet Planets { get; set; }

    public DbSet Characters { get; set; }
}

I've also created a very simple view which lists Charactes already present in database and allows for adding new ones.

@using (Html.BeginForm())
{
    <fieldset>
        <legend>New StarWars Character</legend>
        <div>
            @Html.LabelFor(m => m.Name)
        </div>
        <div>
            @Html.TextBoxFor(m => m.Name)
        </div>
        <div>
            @Html.LabelFor(m => m.Homeworld)
        </div>
        <div>
            @Html.DropDownListFor(m => m.Homeworld, Model.Planets)
        </div>
        ...
        <div>
            <input type="submit" value="Add" />
        </div>
    </fieldset>
}
<table>
    <thead>
        <tr>
            <th>Name</th>
            <th>Homeworld</th>
            ...
        </tr>
    </thead>
    <tbody>
        @foreach (Character character in Model.Characters)
        {
            <tr>
                <td>@character.Name</td>
                <td>@character.Homeworld.Name</td>
                ...
            </tr>
        }
    </tbody>
</table>

The view is powered by following ViewModel and controller.

public class StarWarsViewModel
{
    public string Name { get; set; }

    ...

    public int Homeworld { get; set; }

    public IEnumerable Planets { get; set; }

    public IReadOnlyList Characters { get; set; }
}

public class StarWarsController : Controller
{
    private IStarWarsContext _startWarsContext;

    public StarWarsController(IStarWarsContext startWarsContext)
    {
        _startWarsContext = startWarsContext;
    }

    [HttpGet]
    public ActionResult Index()
    {
        return View(GetViewModel());
    }

    [HttpPost]
    public ActionResult Index(StarWarsViewModel viewModel)
    {
        AddCharacter(viewModel);

        return View(GetViewModel());
    }

    private StarWarsViewModel GetViewModel()
    {
        return new StarWarsViewModel
        {
            Planets = _startWarsContext.Planets
                .Select(p => new { p.Id, p.Name })
                .ToList()
                .Select(p => new SelectListItem { Value = p.Id.ToString(), Text = p.Name }),
            Characters = _startWarsContext.Characters.ToList()
        };
    }

    private void AddCharacter(StarWarsViewModel viewModel)
    {
        throw new NotImplementedException();
    }
}

The AddCharacter method is the point of interest here. There are two ways to implement it and they will result in a different behavior.

Creating entity with new()

Following first Entity Framework tutorial which pops up on Google will result in code similar to the one below.

private void AddCharacter(StarWarsViewModel viewModel)
{
    Character character = new Character();
    character.Name = viewModel.Name;
    ...
    character.HomeworldId = viewModel.Homeworld;

    _startWarsContext.Characters.Add(character);
    _startWarsContext.SaveChanges();
}

Running this code and adding a new Character will result in NullReferenceException coming from the part of view which generates the table (to be more exact from @character.Homeworld.Name). The reason for the exception is the fact that Entity Framework needs to lazy load the Planet entity but the just added Character entity is not a dynamic proxy so lazy loading doesn't work for it. Only Entity Framework can create a dynamic proxy, but in this scenario there is no way for it to do it - the caller already owns the reference to the entity and it cannot be changed to a different class.

Creating entity with DbSet.Create()

In order to be able create new entities as proper dynamic proxies the DbSet class provides Create method. This method returns new dynamic proxy instance which isn't added or attached to the context. To use it only a single line of code needs to be changed.

private void AddCharacter(StarWarsViewModel viewModel)
{
    Character character = _startWarsContext.Characters.Create();
    character.Name = viewModel.Name;
    ...
    character.HomeworldId = viewModel.Homeworld;

    _startWarsContext.Characters.Add(character);
    _startWarsContext.SaveChanges();
}

After this simple change the code works as expected, related entities are being lazy loaded when needed.

The takeaway

The sample above is built in a way which highlights the difference between new() and DbSet.Create() (it even has an N+1 selects hiding in there for the sake of simplicity). In real life this rarely causes an issue as there is couple other things which can impact the behavior (related entity already present in context or usage of Include() method). But when this causes an issue it's always unexpected and I've seen smart people wrapping they heads around what is going on. It is important to understand the difference and use both mechanisms appropriately (sometimes lack of lazy loading maybe desired).

Older Posts