Preview 3 - Cannot write more bytes to the buffer than the configured maximum buffer size: 65536

Topics: Web Api
Jan 20, 2011 at 7:08 AM

Just updated a service to Preview 3, that was working fine in the previous release

I'm getting an exception with a service that generates about 200k of data.  I'm unsure if the exception is due to change in the WCF code drop or a local change.

The code appears to blow up in

System.ServiceModel.Channels.HttpMessageEncoderFactory.HttpMessageEncoder

(164) blows up--->                byte[] messageBytes = response.Content.ReadAsByteArray();
Exception: Cannot write more bytes to the buffer than the configured maximum buffer size: 65536.

Can I increase the buffer size somewhere?

Thanks,

Jason

 

Coordinator
Jan 20, 2011 at 8:33 AM

Hi

We're looking into this. It looks like there may be some defaults in the new http client that we need to up when we create the response.

Thanks for your patience.

Jan 20, 2011 at 11:59 PM

This got me around the issue in the interrum.


# HG changeset patch # User Jason Finch # Date 1295567530 -36000 # Node ID 0dc3deb4fb7fd4ebfd2a7380602d0e212f1d649e # Parent ab97dc50680916bbc8a1e20fd88041203a9646d2 workaround for http://wcf.codeplex.com/Thread/View.aspx?ThreadId=242523 diff -r ab97dc506809 -r 0dc3deb4fb7f Http/Src/Microsoft.ServiceModel.Http/System/ServiceModel/Channels/HttpMessageEncoderFactory.cs --- a/Http/Src/Microsoft.ServiceModel.Http/System/ServiceModel/Channels/HttpMessageEncoderFactory.cs Sat Jan 15 21:29:48 2011 -0800 +++ b/Http/Src/Microsoft.ServiceModel.Http/System/ServiceModel/Channels/HttpMessageEncoderFactory.cs Fri Jan 21 09:52:10 2011 +1000 @@ -161,7 +161,7 @@ return new ArraySegment(); } - byte[] messageBytes = response.Content.ReadAsByteArray(); + byte[] messageBytes = ReadStream(response.Content.ContentReadStream); int messageLength = messageBytes.Length; int totalLength = messageLength + messageOffset; if (totalLength > maxMessageSize) @@ -174,6 +174,21 @@ return new ArraySegment(totalBytes, messageOffset, messageLength); } + private byte[] ReadStream(Stream input) + { + var buffer = new byte[16 * 1024]; + using (var ms = new MemoryStream()) + { + int read; + while ((read = input.Read(buffer, 0, buffer.Length)) > 0) + { + ms.Write(buffer, 0, read); + } + return ms.ToArray(); + } + } + + public override void WriteMessage(Message message, Stream stream) { if (message == null) @@ -187,7 +202,7 @@ } message.Properties.Encoder = this; - + HttpResponseMessage response = GetHttpResponseMessageOrThrow(message); if (response.Content != null)

Coordinator
Jan 23, 2011 at 8:23 PM

Thanks Jafin!

Apr 19, 2011 at 3:11 PM

Glen,

Do you know if this was addressed in the Preview 4?

Thanks,
Gustavo

Apr 19, 2011 at 4:31 PM

Hi guys, if you are working with Preview 3 and don't want to change the WCF source code, you can try with this approach (nor fancy or elegant but it works)

    public class CustomJsonProcessor : JsonProcessor
    {
        public CustomJsonProcessor(HttpOperationDescription operation, MediaTypeProcessorMode mode) : base(operation, mode) { }
        protected override System.ServiceModel.Dispatcher.ProcessorResult OnExecute(object[] input)
        {
            var result = base.OnExecute(input);
            int i = this.Parameter != null ? 2 : 3;
            var response = (HttpResponseMessage)input[++i];
            response.Content.LoadIntoBuffer(int.MaxValue); //set your max here.
            return result;
        }
    }
Regards,
Gustavo
Coordinator
Apr 19, 2011 at 4:41 PM
Should be fixed yes.

Sent from my Windows Phone

From: machadogj
Sent: Tuesday, April 19, 2011 7:11 AM
To: Glenn Block
Subject: Re: Preview 3 - Cannot write more bytes to the buffer than the configured maximum buffer size: 65536 [wcf:242523]

From: machadogj

Glen,

Do you know if this was addressed in the Preview 4?

Thanks,
Gustavo

Apr 19, 2011 at 6:38 PM

I have been successfully returning content that is > 64K. so it seems to be working for me.

Apr 19, 2011 at 7:42 PM
Edited Apr 19, 2011 at 9:00 PM

I'm not sure what's going on, but I'm still getting that error when returning a large list of items.  Researching now ...

If I add this: to the ContactsResource in the QueryableSample:

        public ContactsResource()
        {
            for (int i = totalContacts; i < 30000; i++)
            {
                contacts.Add(new Contact { Name = "New Contact", Id = totalContacts++ });
            }
        }

When calling Get from the sample I get the error:

An exception of type 'System.Net.Http.HttpException' occurred in Microsoft.Net.Http.DLL but was not handled in user code
Additional information: Cannot write more bytes to the buffer than the configured maximum buffer size: 65536.

Can anyone else reproduce this?

Thanks,
David

 

Apr 21, 2011 at 6:58 PM

bump ...

Apr 21, 2011 at 8:29 PM
Edited Apr 21, 2011 at 8:31 PM

I hate to tell you this, but it works for me!  It returns 1MB of data.  This was using Fiddler.  Are you getting the error using the HttpClient?  If so, you may need to use the Send overload instead of the Get method so you can set the HttpCompletionOption to only read the headers.

Coordinator
Apr 21, 2011 at 8:47 PM

Darrel, can you post a sample?

Apr 21, 2011 at 8:56 PM

Warning: Untested code ahead

This should allow HttpClient to retrieve content of any size:

            var httpClient = new HttpClient();
            HttpResponseMessage response = null;
            var request = new HttpRequestMessage() {
                                                      Method = HttpMethod.Get,
                                                      RequestUri = new Uri("http://localhost:8300/Contacts")
                                                   };
            request.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));

            
            response = httpClient.Send(request, HttpCompletionOption.ResponseHeadersRead);

            var stream = new MemoryStream();
            response.Content.CopyTo(stream);

If you don't send the HttpCompletionOption.ResponseHeadersRead then the HttpClient will automatically attempt to buffer the response in a 64K buffer.

Apr 21, 2011 at 9:37 PM

Thanks, Darrel!  The HttpCompletionOption.ResponseHeadersRead was the secret clue.  Works fine now.