Prevent IIS (and potentially other reverse proxies) from unexpectedly compressing ASP.NET Core response

My Server-Sent Events Middleware seems to be a mine of interesting issues. The latest one was about events being delivered with delay (in general one behind) under specific conditions.

The nature of the issue

Initially there was no hint at what are the conditions required for issue to manifest itself. The demo application was working correctly while the one on which the person who reported the issue was working didn't. Luckily the reporter was extremely helpful in diagnosing the issue and devoted some time to find the difference between his and mine code. The difference between working and not working scenario was presence of Response Compression Middleware which I've added while working on previous issue.

My first thought was that Response Compression Middleware must be writing to the response stream differently then my code (when Response Compression Middleware is present it wraps the original response stream). I've gone through the source code of BodyWrapperStream and found nothing. I went deeper and analyzed DeflateStream also without finding anything specific.

At this point I've decided to change approach and use Fiddler to see what was happening on the wire. To my surprise the first thing I've noticed is that the response was still gziped. That really baffled me so I've double checked that the Response Compression Middleware was removed. It was, so it must have been something external to my application. The only external component I was able to identify was IIS Express, so I quickly changed the launch drop down option to run on Kestrel only. That was it, without IIS in front everything was working as expected which meant that IIS (serving as reverse proxy) was compressing the response on its own which resulted in delayed delivery of events.

Preventing IIS from compressing the response

The first obvious option was changing the IIS configuration. This would certainly work but I would have to put the details into documentation and leave it as a trap for others using the same deployment scenario. I wanted to avoid that so I've started researching for other solution. The general conclusion from materials that I've found was that IIS will compress the response if it doesn't detect the Content-Encoding header. That gave me an idea. One of valid values for Content-Encoding is identity which indicates no compression/modification, it might be enough to prevent IIS from adding compression. I've added the code for setting the header to the middleware.

public class ServerSentEventsMiddleware
{
    ...

    public async Task Invoke(HttpContext context)
    {
        if (context.Request.Headers[Constants.ACCEPT_HTTP_HEADER] == Constants.SSE_CONTENT_TYPE)
        {
            DisableResponseBuffering(context);

            context.Response.Headers.Append("Content-Encoding", "identity");

            ...
        }
        else
        {
            await _next(context);
        }
    }

    ...
}

Running the demo application without Response Compression Middleware and behind IIS has confirmed that the solution was working. Now I had to make sure that I haven't broken anything else.

Maintaining compatibility with Response Compression Middleware

As the middleware is now setting the Content-Encoding it could somehow interfere with Response Compression Middleware. I've re-enabled it and run the test again. Screenshot bellow shows the result.

Chrome Developer Tools Network Tab - Multiple Content-Encoding

The response contains two Content-Encoding headers. The reason for this is that Response Compression Middleware is also blindly using IHeaderDictionary.Append. Unfortunately, the fact that header is present twice confuses the browser. The response is coming compressed but the browser treats it as not compressed. I couldn't change how Response Compression Middleware works so I had to be smarter when setting the header. Simply checking if the header is already present didn't work because Response Compression Middleware sets it upon first attempt to write. I was saved by HttpResponse.OnStarting which allows for interacting with the response just before sending the headers. I've replaced my header setting code with following method.

private void HandleContentEncoding(HttpContext context)
{
    context.Response.OnStarting(() =>
    {
        if (!context.Response.Headers.ContainsKey("Content-Encoding"))
        {
            context.Response.Headers.Append("Content-Encoding", "identity");
        }

        return _completedTask;
    });
}

This fixed the problem with two headers and allowed me to close the issue. The approach is universal and can be used in other scenarios with same requirement.