This project is read-only.
2

Closed

Resource leak in HttpClient?

description

I wrote a quick program that uses the HttpClient and was trying to use it to make around 5 million requests. About 420,000 requests into the process my program died with an out of memory exception. The program doesn't do anything else that allocates memory, but I can include the full thing if requested. Is it OK to re-use the same client in a loop, or should I Dispose() it each iteration?

Also of note, I'm pretty sure that for HEAD requests it allocates a content buffer matching the Content-Length header, even though no body is supposed to be sent with HEADs. Could that be part of the problem? Without upping the MaxResponseContentBufferSize it complained.

Here's the relevant code:

var client = new HttpClient(baseUri);
client.MaxResponseContentBufferSize = 4 * 4096 * 1024; // 16MB
foreach(var imageId in images)
{
using (var request = new HttpRequestMessage(HttpMethod.Head, imageId.ToString("N")))
using (var response = client.Send(request))
{
    response.EnsureSuccessStatusCode();
}
}

And the error(s): (The "Error migrating {GUID}" part is my error message.)
Error migrating 4a3038df-d758-4fd3-9595-4ebd7c0c2a21. System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.IO.Stream.InternalCopyTo(Stream destination, Int32 bufferSize)
at System.IO.Stream.CopyTo(Stream destination, Int32 bufferSize)
at System.Net.Http.StreamContent.SerializeToStream(Stream stream, TransportContext context)
at System.Net.Http.HttpContent.LoadIntoBuffer(Int32 maxBufferSize)
at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
at ImageMigrate.Program.Migrate(Guid imageId, HttpClient client) in C:\Playground\ImageMigrate\Program.cs:line 106
at ImageMigrate.Program.Main(String[] args) in C:\Playground\ImageMigrate\Program.cs:line 40
Error migrating 281897b4-bf18-42ad-a7e8-f22a127d00a4. System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.IO.MemoryStream..ctor(Int32 capacity)
at System.Net.Http.HttpContent.LimitMemoryStream..ctor(Int32 maxSize, Int32 capacity)
at System.Net.Http.HttpContent.CreateMemoryStream(Int32 maxBufferSize, Exception& error)
at System.Net.Http.HttpContent.LoadIntoBuffer(Int32 maxBufferSize)
at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
at ImageMigrate.Program.Migrate(Guid imageId, HttpClient client) in C:\Playground\ImageMigrate\Program.cs:line 106
at ImageMigrate.Program.Main(String[] args) in C:\Playground\ImageMigrate\Program.cs:line 40
Error migrating ffa90c50-4c21-4467-aa4b-5b922a046ea8. System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Collections.Generic.Dictionary2.Resize()
at System.Collections.Generic.Dictionary
2.Insert(TKey key, TValue value, Boolean add)
at System.Net.Http.Headers.HttpHeaders.AddHeaderToStore(String name, HeaderStoreItemInfo info)
at System.Net.Http.Headers.HttpHeaders.CreateAndAddHeaderToStore(String name)
at System.Net.Http.Headers.HttpHeaders.GetOrCreateHeaderInfo(String name, Boolean parseRawValues)
at System.Net.Http.Headers.HttpHeaders.AddWithoutValidation(String name, String value)
at System.Net.Http.HttpClientChannel.AddHeaderValues(WebHeaderCollection source, Int32 index, String header, HttpHeaders destination)
at System.Net.Http.HttpClientChannel.CreateResponseMessage(HttpWebResponse webResponse, HttpRequestMessage request)
at System.Net.Http.HttpClientChannel.Send(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
at ImageMigrate.Program.Migrate(Guid imageId, HttpClient client) in C:\Playground\ImageMigrate\Program.cs:line 106
at ImageMigrate.Program.Main(String[] args) in C:\Playground\ImageMigrate\Program.cs:line 40
Closed Nov 14, 2011 at 11:22 PM by danroth27
No repro code was provided, but we have fixed similar issues in this space. Hopefully this issue is now addressed.

comments

kogir wrote May 11, 2011 at 6:38 PM

Update: Disposing the HttpClient inside the loop fixes the problem. Since there's no documentation I'm not sure if I was just using it wrong, or if there really is an issue here.

ericlaw1979 wrote Nov 5, 2013 at 8:51 PM

The latest .NET 4.5 of the HTTPClient code appears to still blindly allocate a buffer of the size specified in the Response Content-Length header. See the CreateMemoryStream function inside System.Net .Http.HttpContent.