Some time ago I've written about streaming JSON objects from server to client in ASP.NET Core. Later I've created a small library with this functionality which supports IAsyncEnumerable<T> and has separated implementations for System.Text.Json and Newtonsoft.Json. The number of downloads on NuGet isn't high but looks like somebody is using it.

Recently, I've received a question, how this JSON objects stream can be consumed with HttpClient. I've once read that instead of writing a response to an email it's better to write a blog post, so here I am.

Consuming NDJSON Stream With HttpClient

My demo project consumes the stream from browser client. In order to do it, it's using cool browsers capabilities like Streams API. Unfortunately this doesn't make things obvious enough for somebody not familiar with those APIs to immediately port the implementation.

The way in which one should be thinking about NDJSON response is a string. It's a string that is being append new lines and every line represents a single JSON object. So, in case of .NET, the good old tools we have for "string as stream" are good enough. For example StreamReader. We can use it to asynchronously read lines from response stream and then deserialize them into objects. Below code wraps the whole thing in nice extension method which returns IAsyncEnumerable<T>.

internal static class HttpContentNdjsonExtensions
{
    private static readonly JsonSerializerOptions _serializerOptions
        = new JsonSerializerOptions
        {
            PropertyNameCaseInsensitive = true,
            PropertyNamingPolicy = JsonNamingPolicy.CamelCase
        };

    public static async IAsyncEnumerable<TValue> ReadFromNdjsonAsync<TValue>(this HttpContent content)
    {
        if (content is null)
        {
            throw new ArgumentNullException(nameof(content));
        }

        string? mediaType = content.Headers.ContentType?.MediaType;

        if (mediaType is null || !mediaType.Equals("application/x-ndjson", StringComparison.OrdinalIgnoreCase))
        {
            throw new NotSupportedException();
        }

        Stream contentStream = await content.ReadAsStreamAsync().ConfigureAwait(false);

        using (contentStream)
        {
            using (StreamReader contentStreamReader = new StreamReader(contentStream))
            {
                while (!contentStreamReader.EndOfStream)
                {
                    yield return JsonSerializer.Deserialize<TValue>(await contentStreamReader.ReadLineAsync()
                        .ConfigureAwait(false), _serializerOptions);
                }
            }
        }
    }
}

With that extension method an NDJSON can be called and results asynchronously enumerated. The one important thing is to make sure, that we will be given the response stream once headers have been received (HttpCompletionOption.ResponseHeadersRead).

HttpClient httpClient = new HttpClient();

using (HttpResponseMessage response = await httpClient.GetAsync("...", HttpCompletionOption.ResponseHeadersRead))
{
    response.EnsureSuccessStatusCode();

    await foreach (WeatherForecast weatherForecast in response.Content!.ReadFromNdjsonAsync<WeatherForecast>())
    {
        ...
    }
}

That's It

Yes, that's it. This is simple, gets the job done, and when used in right scenario can notably improve perceived performance.