Posting and Processing MultipartContent

Topics: Web Api
May 27, 2011 at 12:58 PM
Edited May 27, 2011 at 1:27 PM

Hi guys,

I'm trying to build a REST API where one of the resources that can be posted is made up of a serializable object as well as a related file that needs to be stored against that object on the server-side. There doesn't seem to be any examples of how to post (i.e. upload) files to a WCF Web API service, so I've been trying different things. One of the things that seems moderately sensible is to package up the object and the associated file in a MultipartContent (or even MultipartFormDataContent) instance. Here's a snippet from a simple test for this:

[TestMethod]

public void JobsResource_PostFileJobTest()
{
  var job = new Job()
  {
    Id = Guid.NewGuid(),
    CreatedBy = "Joe Bloggs",
    CreatedOn = new DateTime(2011, 05, 27, 12, 40, 0, DateTimeKind.Utc),
    Status = "New"
  };
 
  var payload = new byte[10] { 0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28, 0x29 };
 
  // Create a multi-part post to upload the basic Job metadata and the associated ZIP file
  var content = new MultipartFormDataContent();
  var jobMetadata = new StringContent(job.ToString());
  jobMetadata.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/json");
  content.Add(jobMetadata);
  content.Add(new ByteArrayContent(payload));

  var response = HttpClient.Post("http://localhost:9000/jobs/", content);
    ...
}

The problem comes when the API receives the HTTP POST request and needs to handle the data. I've tried several methods, from ReadAs<MultiPartContent>() to copying the stream from the request Content to a newly instantiated MultiPartContent object. But none of them seem to be able to "rebuild" the source content.

[WebInvoke(UriTemplate = "", Method = "POST")]
public HttpResponseMessage Post(HttpRequestMessage request)
{
  // Use ReadAs<> (doesn't work)
 var content = request.Content.ReadAs<MultiPartContent>(); // Can't use an HttpContent object in ReadAs methods

 // Use Stream Copy (doesn't work)
 var content = new MultipartContent();
 request.Content.ContentReadStream.CopyTo(content.ContentReadStream); // The resulting content object does not have 2 parts as expected
  ...
}

So, I guess the question is, has anyone managed to do this? I'm sure it will be something trivial, but so far I've not been able to figure it out. Do I have to actually write a MultiPartContent custom processor that re-builds a request into a MultiPartContent instance? This would be a bit like re-inventing the wheel, and before I get on with it, I was wondering if anyone has any better ideas.

Thank you! And thanks to the WCF team for building such fantastic tools for us developers!

AL

May 27, 2011 at 3:12 PM

I don't believe reading MultiPartContent has been implemented.  I'm going to try and do it with ObjectContent<List<Object>> and a MultipartContentFormatter.

Don't hold your breath though :-)

May 27, 2011 at 4:23 PM

Thanks Darrel, if you manage it before I turn blue, that would be a good thing :)

In the mean time I'll work on a different part of the project and revisit this in a few days to see what to do next.

Thank you!

AL

Jun 23, 2011 at 2:21 PM
Edited Jun 23, 2011 at 2:21 PM

@drfonz, did you manage to resolve your issue? I'm working on a project that might eventually need this, and I'm trying to be ready for the bump in the road.

@Darrel did you try your approach? 

Jun 23, 2011 at 2:35 PM
Edited Jun 23, 2011 at 2:35 PM

I left it too late in the project, so I had to come up with a hack to work around this. The parser I wrote works on a very specific multi-part message, I'm adding the code below so that you can have a look and perhaps modify it to your own specs. I'd be much happier if the WCF team could deliver proper multi-part parsing capabilities in a future drop, but until then, this does the job for me...

 

namespace MyNamespace.Processors
{
  using System;
  using System.IO;
  using System.Text;

  public class MultipartContentProcessor
  {
    public MultipartContentProcessor(Stream stream)
    {
      this.Parse(stream, Encoding.UTF8);
    }

    public MultipartContentProcessor(Stream stream, Encoding encoding)
    {
      this.Parse(stream, encoding);
    }

    public bool IsValid
    {
      get;
      private set;
    }

    public string FormData
    {
      get;
      private set;
    }

    public byte[] FileContents
    {
      get;
      private set;
    }

    private static int IndexOf(byte[] searchWithin, byte[] searchFor, int startIndex)
    {
      int index = 0;
      int startPos = Array.IndexOf(searchWithin, searchFor[0], startIndex);

      if (startPos != -1)
      {
        while ((startPos + index) < searchWithin.Length)
        {
          if (searchWithin[startPos + index] == searchFor[index])
          {
            index++;
            if (index == searchFor.Length)
            {
              return startPos;
            }
          }
          else
          {
            startPos = Array.IndexOf<byte>(searchWithin, searchFor[0], startPos + index);
            if (startPos == -1)
            {
              return -1;
            }
            index = 0;
          }
        }
      }

      return -1;
    }

    private static byte[] ToByteArray(Stream stream)
    {
      byte[] buffer = new byte[32768];
      using (var ms = new MemoryStream())
      {
        while (true)
        {
          var read = stream.Read(buffer, 0, buffer.Length);
          if (read <= 0)
          {
            return ms.ToArray();
          }
          ms.Write(buffer, 0, read);
        }
      }
    }

    private void Parse(Stream stream, Encoding encoding)
    {
      this.IsValid = false;

      // Read the stream into a byte array
      var data = ToByteArray(stream);

      // Copy to a string for header parsing
      var content = encoding.GetString(data);

      // The first line should contain the delimiter
      var delimiterEndIndex = content.IndexOf("\r\n");

      if (delimiterEndIndex > -1)
      {
        var delimiter = content.Substring(0, content.IndexOf("\r\n"));

        // Look for Form Data
        var startForm = content.IndexOf("form-data\r\n\r\n") >= 0 ? content.IndexOf("form-data\r\n\r\n") + "form-data\r\n\r\n".Length : -1;
        var endForm = content.Substring(startForm).IndexOf("\r\n" + delimiter);
        var formData = content.Substring(startForm, endForm);

        if (!string.IsNullOrWhiteSpace(formData))
        {
          FormData = Uri.UnescapeDataString(formData);
        }

        // look for the zip file block
        var startIndex = content.IndexOf("Content-Disposition: inline\r\n\r\n") >= 0 ? content.IndexOf("Content-Disposition: inline\r\n\r\n") + "Content-Disposition: inline\r\n\r\n".Length : -1;
        var delimiterBytes = encoding.GetBytes("\r\n" + delimiter);
        var endIndex = IndexOf(data, delimiterBytes, startIndex);

        var contentLength = endIndex - startIndex;

        // Extract the file contents from the byte array
        var fileData = new byte[contentLength];

        Buffer.BlockCopy(data, startIndex, fileData, 0, contentLength);

        this.FileContents = fileData;
        this.IsValid = FileContents.Length > 0 && FormData.Length > 0;

      }
    }
  }
}

 

As for how this is invoked, I have a POST method like this:

 

    [WebInvoke(UriTemplate = "", Method = "POST")]
    public HttpResponseMessage Post(HttpRequestMessage request)
    {
      // Break the request into its constituent parts (should be a Multipart message for this to be a valid POST)
      var content = new Processors.MultipartContentProcessor(request.Content.ContentReadStream);

      if (content.IsValid)
      {
        ...
      }
    }

 

Finally, the client of the service will post data built in this fashion:

 

      var postData = new MultipartFormDataContent();
      
      // Encode the form data and add it to the POST payload
      var formValues = new Dictionary
        {
          { "CreatedBy", UserProfile.Current.DisplayName }, 
          { "CreatedOn", string.Format("{0}", DateTime.UtcNow) }, 
           ...
        };
      postData.Add(new FormUrlEncodedContent(formValues));

      // Add the file to the payload
      var zipBlock = new ByteArrayContent(zipFile.FileContents);
      zipBlock.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
      zipBlock.Headers.AddWithoutValidation("Content-Disposition", "inline");
      postData.Add(zipBlock);


      // Post the job information to the Service
      var response = client.Post(ServiceUrl, postData);

As you can see, a total hack that's very specific, but as I said time was at a premium and I had toput something together quickly :)
Hope it helps!
AL

 

Jun 23, 2011 at 2:51 PM

Thanks drfonz! I'm checking your workaround now, and will use it as a reference for sure.

Jun 23, 2011 at 3:21 PM

@racielrod  Sorry, haven't had a chance yet.  I'll put it as next on my web api todo list :-)

Apr 19, 2012 at 2:07 AM

Comments and questions:

The work around is nice, but very specific. One example: the boundary string should be obtained from the Content-Type header parameter and not from the actual data.

Is there any progress on implementing the generic multipart parser ?

Alternatively, are there any tips for easily constructing an HttpContent object from a byte array or a stream ? This function is already made for the (single content) message and if can be used for parsing a part, will eliminate the need to redo it.

Apr 19, 2012 at 6:56 AM

We provide first-class support for MIME multipart in a variety of scenarios (not just File upload) but also MIME multipart related, etc. and you can even do batching if you want.

There is a nice blog on how to do this here [1]

Hope this helps,

Henrik

[1] http://www.strathweb.com/2012/04/html5-drag-and-drop-asynchronous-multi-file-upload-with-asp-net-webapi/