Streaming output from rest service

Topics: Web Api
Jun 18, 2011 at 7:31 PM

Hi,

I have a rest service which needs to return a large amount of data (10-40Mb) in response to a request.  The service itself returns HttpResponseMessage<IEnumerable<T>>, with T being a "dumb" data transfer object (less than 10 properties which are all simple value types).  It returns in XML or JSON formats.  In order not to use excessive memory, I want to enumerate the items directly into the output stream, rather than build the whole response in memory.

My first hurdle was that the XML/JSON serializers do not support serializing an IEnumerable directly.  As the return types are simple, I have written some output formatters which can take each item from the IEnumerable, format it, and write to the stream.

My question is, as I have access to the stream from the formatter, is this sufficient to actually stream the response using the web api?  I know in "bare" WCF, I also need to set transferMode=StreamedResponse etc to get it to work, but I'm not clear what is required here.

Thanks,

Miles

Coordinator
Jun 18, 2011 at 9:46 PM

Hi Miles

I just deleted my last response as after talking to my team it looks like the net result was still sync streaming.

It looks like we may have a bug related to streaming of responses.  I will keep looking into this and update when I have a solid answer.

Thanks for your patience

Glenn

Aug 4, 2011 at 5:29 PM
Edited Aug 6, 2011 at 11:16 PM

Does the WCF Web API use the WCF StreamedResponse model, or is it capable of returning a response by writing data in buffers to the response stream?

I guess I am trying to figure out what the equivalent to HttpContext.Response.OutputStream is in the WCF Web API.

As an example, I currently have a generic HTTP handler (.ashx) that processes a request that provides the unique identifier of the resource with a large amount of binary data like this:

private static void WriteResponse(HttpContext context, Guid id)
{
    using(var connection = ConnectionFactory.Create())
    {
        using (var command = new SqlCommand("[dbo].[GetResponse]", connection))
        {
            command.CommandType = CommandType.StoredProcedure;
 
            command.Parameters.Add("@Id", SqlDbType.UniqueIdentifier);
            command.Parameters.AddReturnValue();
 
            command.Parameters["@Id"].Value = id;
 
            command.Open();
 
            using(var reader = command.ExecuteReader(CommandBehavior.SequentialAccess))
            {
                if(reader.Read())
                {
                    WriteResponse(context, reader);
                }
            }
        }
    }
}
private static void WriteResponse(HttpContext context, SqlDataReader reader)
{
    if (context == null || reader == null)
    {
        return;
    }
 
    DateTime expiresOn      = DateTime.UtcNow;
    string contentType      = String.Empty;
    long contentLength      = 0;
    string fileName         = String.Empty;
    string fileExtension    = String.Empty;
 
    expiresOn               = reader.GetDateTime(0);
    fileName                = reader.GetString(1);
    fileExtension           = reader.GetString(2);
    contentType             = reader.GetString(3);
    contentLength           = reader.GetInt64(4);
 
    context.Response.AddHeader("Content-Disposition"String.Format(null"attachment; filename={0}", fileName));
 
    WriteResponse(context, reader, contentType, contentLength);
}
private static void WriteResponse(HttpContext context, SqlDataReader reader, string contentType, long contentLength)
{
    if (context == null || reader == null)
    {
        return;
    }
 
    int ordinal     = 5;
    int bufferSize  = 4096 * 1024; // 4MB
    byte[] buffer   = new byte[bufferSize];
    long value;
    long dataIndex;
 
    context.Response.Buffer         = false;
    context.Response.ContentType    = contentType;
    context.Response.AppendHeader("content-length", contentLength.ToString());
 
    using (var writer = new BinaryWriter(context.Response.OutputStream))
    {
        dataIndex   = 0;
        value       = reader.GetBytes(ordinal, dataIndex, buffer, 0, bufferSize);
 
        while(value == bufferSize)
        {
            writer.Write(buffer);
            writer.Flush();
 
            dataIndex   += bufferSize;
            value       = reader.GetBytes(ordinal, dataIndex, buffer, 0, bufferSize);
        }
 
        writer.Write(buffer, 0, (int)value);
        writer.Flush();
    }
}
How would one go about writing to the output stream of a response like above using the WCF Web API?
Nov 3, 2011 at 10:13 AM

I'm also interested in the answer.

Nov 3, 2011 at 1:56 PM

Take a look at the ActionOfStreamContent class http://wcf.codeplex.com/SourceControl/changeset/view/861c1ae7cf1e#WCFWebApi%2fHttp%2fSrc%2fMicrosoft.ApplicationServer.Http%2fmicrosoft%2fApplicationServer%2fHttp%2fActionOfStreamContent.csin 

I believe that will do what you are looking for.  Ping me if you need an example of how you use it.

 

Nov 3, 2011 at 7:09 PM

Thanks! I ended up using System.Net.Http.StreamContent setting the Content in the HttpResponseMessage so I'm not really streaming in chunks. It works for now but I will probably have to change this to something better so thank you for the input.

[WebGet(UriTemplate = "attachment/download?url={url}")]
    public HttpResponseMessage DownloadAttachment(string url)
    {
      string contentType;
      var stream = FileService.Current.Download(url, out contentType);
      var content = new StreamContent(stream);
      var response = new HttpResponseMessage()
                  {
                    Content = content
                  };
      return response;
    }

 

Nov 4, 2011 at 1:51 PM
Edited Nov 4, 2011 at 1:54 PM
DarrelMiller wrote:

Take a look at the ActionOfStreamContent class http://wcf.codeplex.com/SourceControl/changeset/view/861c1ae7cf1e#WCFWebApi%2fHttp%2fSrc%2fMicrosoft.ApplicationServer.Http%2fmicrosoft%2fApplicationServer%2fHttp%2fActionOfStreamContent.csin 

I believe that will do what you are looking for.  Ping me if you need an example of how you use it.

 

Hi DarrelMiller

The class seems to be declared as Internal.... 

I saw in another thread that you are thinking on opening it up for Public use... will that happen?

 

Søren

Nov 4, 2011 at 2:30 PM

I certainly hope so.  Until then, I would say copy/paste is your friend :-)