Useful Platitudes

Notes on programming by Daniel Mendel

Benchmarkers, Beware the Ephemeral Port Limit

If you’re benchmarking a web server using tools like ab, weighttp or siege you may encounter an issue that could skew your results – hanging the connection pool. To illustrate this point, let’s look at a couple of benchmarks using siege on OS X:

The first makes a total of 16,300 requests:

1
2
3
4
5
6
7
8
9
10
11
12
Transactions:               16300 hits
Availability:           100.00 %
Elapsed time:            12.03 secs
Data transferred:         0.09 MB
Response time:                0.09 secs
Transaction rate:      1350.79 trans/sec
Throughput:               0.01 MB/sec
Concurrency:            122.91
Successful transactions:       16300
Failed transactions:             0
Longest transaction:          0.15
Shortest transaction:         0.00

The second makes a total of 16,384 requests:

1
2
3
4
5
6
7
8
9
10
11
12
Transactions:               16384 hits
Availability:           100.00 %
Elapsed time:            21.73 secs
Data transferred:         0.09 MB
Response time:                0.16 secs
Transaction rate:       753.57 trans/sec
Throughput:               0.00 MB/sec
Concurrency:            121.28
Successful transactions:       16384
Failed transactions:             0
Longest transaction:          4.65
Shortest transaction:         0.00

The second reports the server to be about 50% slower than the first!

If the processes are monitored during the second test, both siege and the server process spin up to full capacity for an extended duration as one would expect, but at some point they become totally idle for about 15 seconds before kicking back into action to finish the test.

What gives?

Ephemeral Port Range

To understand what is happening, we have to look at how TCP connections are handled by the operating system. Whenever a connection is made between a client and server, the system binds that connection to an ephemeral port – a set of ports specified at the high end of the valid port range. This is how to reveal what the ephemeral port range is on your system:

1
2
3
> sysctl net.inet.ip.portrange.first net.inet.ip.portrange.last
net.inet.ip.portrange.first: 49152
net.inet.ip.portrange.last: 65535

The total number of ephemeral ports available on OS X is 16,383 ( on Linux it is usually 28,232 – it is possible to increase the ephemeral port range on OSX ). You might think that this should be more than enough to run our benchmarks since we have only 125 simultanious connections occuring at any given time. However, when one of these ports is closed it does not become immediately available for a new request.

TCP Connection States

During the lifetime of a request, each port goes through a series of states, from SYN_SENT when establishing a connection to ESTABLISHED when communication is actively happening, through a series of closing states eventually culminating in TIME_WAIT after the port has been closed.

During TIME_WAIT the port is held in limbo to ensure any remaining packets are not erroniously provided to a fresh connection. ( Check the current state of ports in use by running netstat -p tcp, get a full overview of the states in the man netstat text )

The duration of the TIME_WAIT state is the Maximum Segment Lifetime and is defined in net.inet.tcp.msl. We can check what it is:

1
2
> sysctl net.inet.tcp.msl
net.inet.tcp.msl: 15000

15 seconds. Bingo! There’s the slowdown we’ve had skewing our results.

Note that this limitation does not affect real-world requests to a live server because each TCP connection is defined by the tuple of source IP, source port, destination IP and destination port – so the ephemeral port limit only applies to a single client / server pair.

It’s possible to reconfigure your kernel to allow a lot more requests, see Richard Jones’s A Million User Comet Application with Mochiweb Part I.