Async services in Web API?

Topics: Web Api
Jul 19, 2011 at 12:34 PM

Greetings,

Will the new HTTP API support async requests similar to what ASP.NET does? I'm specifically thinking about offloading the thread to an IO Completion Port pool while doing IO, freeing up the request thread, then reactivating a new thread to finish and return the request once IO is done.

This is something I see as being quite important for scalability in REST services.

I know WCF has async support, but that requires you do implement separate Begin/End methods on your service contract and the client to know about them.  I want the REST contract to stay simple for the client, and since this is possible in ASP.NET/MVC I'd love to see something similar for the HTTP API :)

Jul 19, 2011 at 12:47 PM

An implementation using Task<T> or something similar would be awesome ;)  But if I need to do some work myself for the first version that's fine, but I hope you got some plans to support the async keyword/pattern thats coming in C# 5.

 

Jul 19, 2011 at 2:48 PM

SiggiG.

When you say "important for scalability", I'm assuming you are referring to the ability to support tens of thousands of concurrently connected clients, right?  The reason I ask is that for some HTTP services this is an important capability, and therefore it may be relevant to this Web API stack, however, it is not really consistent with a RESTful architecture.  

Long polling types of requests cause information to be transferred between client and server in a way that is not visible to intermediaries and therefore loses many of the benefits of a REST architecture.

I think you will find that RESTful architectures can scale quite effectively without supporting this type of asynchronous request mechanism.  I'm not saying it is not useful in general, just not so much for REST.

Jul 19, 2011 at 2:59 PM

Darrel,

I don't think that "long polling" is the only scenario where async is relevant.

Consider for example a system where the handling of a request requires "calling" other services that may have high latency (in the order of seconds). You will end up having lots of request handling threads just waiting for IO completion. The end result may be the exhaustion of threads on the thread-pool (I've observed this on production systems).

Pedro

Jul 19, 2011 at 3:04 PM

SiggiG.

1) To the best of my knowledge, Web API still supports WCF's AsyncPattern. 

2) On WCF, the AsyncPattern is a service implementation detail. It is *not* exposed on the contract/WSDL. The client does *not* have to know it. See: http://pfelix.wordpress.com/2008/06/27/wcf-and-the-asyncpattern-property-part-1/

Pedro

Jul 19, 2011 at 3:12 PM


I say it's important for scalability in the sense of getting more out of each webserver and serving up more requests/second.  I've previously had issue with services where the server stops responding at 40% CPU load simply because the request pools were filling up.  Tweaking pool sizes and using async allowed us to get a much better usage from each server.

I agree that REST isn't well suited for long polling requests but that is not what I was referring to.  Even if some of your requests take 1-2 seconds, if you have enough users pounding your services things will fill up quickly.
People that create high scalability websites also use async programming to offload IO work to the IOCP, simply to scale better.  There are always cases of some types of requests taking longer than others, so you don't want the "long running" ones (even if its a few seconds) blocking the others.
I'm confused why you don't think this is useful for REST? REST is basically "web services done like the rest of the web" so since it's useful for webpages, why does it not apply to REST services?

In my case I'm working on services for a userbase of around 400K users (currently around 50-60K concurrent) and rapidly growing, so performance and scalability is very much on my mind.

Jul 19, 2011 at 3:19 PM

Pedro, thanks for the clarification on the WCF async contract :)

If the Web API does support this, I wonder how you can use WebGetAttribute/WebInvokeAttribute together with an OperationContract? ie. how the request routing knows that it shold call SomeResource.BeginGet() instead of Get() and how does it discovert the EndGet method.

 

Jul 19, 2011 at 3:45 PM

SiggG,

1) Add an OperationContract and a WebGet attribute to the begin method.

2) The end method is found by name convention

3) Some code

    [ServiceContract]
    class AsyncService
    {
        private string SimulateLengthyIO(string s)
        {
            Thread.Sleep(3*1000);
            return s.ToUpper();
        }

        [OperationContract(AsyncPattern = true)]
        [WebGet(UriTemplate="async/{s}")]
        IAsyncResult BeginGet(string s, AsyncCallback cb, object state)
        {
            Func<string, string> f = SimulateLengthyIO;
            return f.BeginInvoke(s, cb, state);
        }
        HttpResponseMessage EndGet(IAsyncResult ar)
        {
            var dar = ar as AsyncResult;
            var f = dar.AsyncDelegate as Func<string, string>;
            return new HttpResponseMessage()
                       {
                           Content = new StringContent(f.EndInvoke(ar))
                       };
        }
    }

 

Jul 19, 2011 at 4:04 PM

Very interesting sample Pedro, thank you :)

I presume that since the WCF method is using AsyncPattern=true and you are returning IAsyncResult that WCF automatically puts the thread into the IOCP threadpool then runs EndGet on a thread from the request pool?

Jul 19, 2011 at 9:18 PM

Pedro, SiggiG,

I stand, well sit actually, corrected :-)  I've only really heard of people using this async approach for long polling before.  I've never been in a situation where where a server is handling thousands of simultaneous requests but has excess CPU capability.  

Am I correct in assuming  this is much more common when implementing intermediaries?

Darrel

Jul 19, 2011 at 9:36 PM

Unless you're doing something really CPU intensive in your services or running on servers with slow CPUs I think this situation can present itself quite frequently while doing any kind of IO in services or webs that handle a lot of requests.
In my case I'm using SQL Server, other services (including XML-RPC which is the main bottleneck) and even a distributed cache server.

If you are interested, Jeffrey Richter recently did an excellent talk on Channel 9 that touched on this subject:
http://channel9.msdn.com/Shows/AppFabric-tv/AppFabrictv-Threading-with-Jeff-Richter

Jul 19, 2011 at 11:50 PM

Darrel,

Yes, I've observed this exact behavior with intermediaries: server doesn't accept more request while having enough available CPU, due to thread-pool exhaustion.

Pedro

Coordinator
Jul 19, 2011 at 11:54 PM

SiggiG thanks for the feedback. We are looking at supporting Task/Task<T> in the programming model for web api. We have not locked on when we will get it in yet. We may be able to get some sample code out soon that shows how it can be enabled as an add-on.

Thanks

Glenn

Jul 20, 2011 at 12:24 AM

That's great to hear Glenn :) In the meanwhile I can do something similar to what Pedro posted, and already got some ideas on how to use Tasks for that.

Jul 20, 2011 at 12:40 AM

SiggiG,

I don't understand your statement: "since the WCF method is using AsyncPattern=true and you are returning IAsyncResult that WCF automatically puts the thread into the IOCP threadpool then runs EndGet on a thread from the request pool"

To the best of my knowledge,

  1) the request thread is released to the pool after the begin method returns.

  2) the end method is only executed (by a potentially different thread from the pool) after the call to the WCF provided callback (passed on the begin method)

  3) this allows something very interesting: having N operations "waiting" for I/O without any threads blocked on I/O (i.e. all the threads are available in the pool)

Pedro 

 

Jul 20, 2011 at 12:45 AM

SiggiG

Sometime ago, I and a colleague did a little experiment to support Task<T> on WCF, using the AsyncPattern and a custom operation invoker.

The code is available here: https://gist.github.com/731300 

It is very far from being production quality code, but may be the beginning of something.

Pedro