HttpWebRequest reuse of TCP Connections

An interesting issue with using System.Net.HttpWebRequest to fetch data from REST based LOB (Line of Business) system.

As part of our testing we ran into this interesting issue where the data belonging to a different user was returned by the REST service. As an example when making a call to the REST service passing the credentials of User X, the service was returning the data corresponding to User Y. It was unclear as to why a different user’s data is being returned and this has severe security implications.

On digging further found that the HttpWebRequest re-uses the underlying TCP connection and this has important implications.

One possibility we looked at was to disable the KeepAlive property on the HttpWebRequest. There are no real advantages of disabling KeepAlive because disabling KeepAlive means that you perform an expensive operation of opening up and closing sockets after each request/response. The socket teardown (after the FIN/RST) takes a significant amount of time (upto 4 minutes), so that socket becomes unusable for the clean-up period. When this happens, there could be too much demand for new socket and you could run into winsock error 10055/WSAENOBUFS.

HttpWebRequest provides ConnectionGroups for grouping the request to specific Uri. So the solution we took was to isolate different user requests into different connection groups. In addition we need to ensure that the ConnectionGroup is unique for unique users and not end up creating new ConnectionGroup for every new request.

One of the important design consideration is to ensure that the correct ConnectionGroup name is set on each request. The idea is to set a unique connection group name for unique users, not for every unique HTTP request.

If we set a new ConnectionGroupName for each unique request (such as a unique GUID for each new request), we are essentially creating a new connection group for every new request and that is going to open up a new connection each time. That is not what you want. When you open a new connection each time (by setting a unique connection group for each request), each of that connection will remain alive for the ServicePointManager.MaxServicePointIdleTime (100 seconds default) and will eventually run out of available connections/sockets. Connections get cleaned up by the ServicePointManager after a unique ServicePoint has been completely idle for the MaxServicePointIdleTime (100 seconds) where it will go and prune all existing connections and release them for subsequent use. If the ServicePointManager is kept busy all the time, then it won’t get a chance to close existing connections and hence will run out of socket space.

The ConnectionGroup will be automatically closed by the ServicePointManager after a ServicePoint has been completely idle for the MaxServicePointIdleTime. If we keep creating new requests/ ConnectionGroup for a specific ServicePoint (a unique address), then the ServicePoint will never get a chance to clean up automatically.

Instead we can however manually close a ConnectionGroup by calling: ServicePointManager.FindServicePoint and then once we get the ServicePoint of interest, call: ServicePoint.CloseConnectionGroup method.

The ServicePoint is managed per unique server/host name, there is no upper limit. Internally, the ServicePointManager just manages a hashtable, so there is no real upper limit. By default each ServicePoint has a single ConnectionGroup which has a default max connection limit of 2 (for desktop applications) and 12 * number of CPU for asp.net scenario. The complexity increases when you create unique connectiongroup per servicepoint, until you create so many sockets that you run out of sockets. At that point, you will start seeing errors.

In summary we fixed this issue by setting a unique connectiongroup name for each Http call. The connection group is generated based on server / host name and the user credentials. I will cover in my next post on how to manage the connection group name so that we don’t run out of sockets.