Gfrktyl

I am writing a piece of software that aims at streaming the modified lines of an external file located at given uri (and retrieved as a QueryString parameter).

I am trying to minimize the impact of big files so that I can maximize the use of asynchronous calls (i.e. async / await). I am using AsyncEnumerable (https://github.com/Dasync/AsyncEnumerable) to have at the same time the feature of enumerable and and async / await, in order to free up threads and memory (I am not going to keep the whole file in memory before sending the modified version back).

I created an extension method that can fetch the lines of a remote file.

public static class StringExtensions

{

    public static AsyncEnumerable<string> ReadLinesAsyncViaHttpClient(this string uri)

    {

        return new AsyncEnumerable<string>(async yield =>

        {

            using (var httpClient = new HttpClient())

            {

                using (var responseStream = await httpClient.GetStreamAsync(uri))

                {

                    using (var streamReader = new StreamReader(responseStream))

                    {

                        while(true)

                        {

                            var line = await streamReader.ReadLineAsync();



                            if (line != null)

                            {

                                await yield.ReturnAsync(line);

                            }

                            else

                            {

                                return;

                            }

                        } 

                    }

                }

            }

        });

    }

}

And used that extension method in a Controller action:

using System;

using System.Collections.Async;

using System.IO;

using System.Threading.Tasks;

using Microsoft.AspNetCore.Mvc;



namespace WebApplicationTest.Controllers

{

    [Route("api/[controller]")]

    [ApiController]

    public class LinesController : ControllerBase

    {

        private static readonly Random Random = new Random();



        // GET api/lines

        // POST api/lines

        [HttpGet]

        [HttpPost]

        public async Task<IActionResult> GetAsync([FromQuery] string fileUri)

        {         

            using (var streamWriter = new StreamWriter(HttpContext.Response.Body))

            {

                await fileUri.ReadLinesAsyncViaHttpClient().ForEachAsync(async line =>

                {

                    // Some dumb process on each (maybe big line)

                    line += Random.Next(0, 100 + 1);

                    await streamWriter.WriteLineAsync(line);

                });

            }



            return Ok();

        }

    }

}

I used the HttpContext.Response.Body stream instead of the PushStreamContent cause according to this answer: https://stackoverflow.com/a/48115434/4636721 (in .NET Core we don't need to use the PushStreamContent if no need of HttpMessage)

It seems to work I mean, I get a payload with the lines of the initial file passed in the query string and with some random numbers at the end of each line. It seems there is no increase in memory but I would like to have another pair of eyes to check what I have done.

[EDIT 2]
Turns out there is already an issue on the ASP.NET MVC Github about that: https://github.com/aspnet/Mvc/issues/6224

As the author of the AsyncEnumerable library, I confirm the correctness of the code.

P.S. we've been using this library for 3 years at the heart of a large distributed software at my current workplace.

A small improvement you can do to avoid the pretty deep nesting is, stacking all usings without extra brackets.

public static AsyncEnumerable<string> ReadLinesAsyncViaHttpClient(this string uri)

{

    return new AsyncEnumerable<string>(async yield =>

    { 

        using (var httpClient = new HttpClient())

        using (var responseStream = await httpClient.GetStreamAsync(uri))

        using (var streamReader = new StreamReader(responseStream))

        {

            while(true)

            {

                 var line = await streamReader.ReadLineAsync();

                 if (line != null)

                 {

                     await yield.ReturnAsync(line); 

                 }

                 else

                 {

                     return;

                 } 

             }

        }

    });

}

Apart from that I don't know if this the kind of thing that should be a extension method. This should rather be a class.

(I'm currently do not have my hands on a pc, so I cannot comment on the technical side of things >:( )