Jair Trejo

Performance testing HTTP services with httperf

At Vinco Orbis we recently build a web service that in turn queries some other services through HTTP and returns a list of those for which the query was successful. It looked something like this:

import requests

successful = []
for service in get_services():
    r = requests.get(service.url)
    if r.status_code == 200:
        successful.append(service.name)

print successful

As you might suppose, making a bunch of HTTP requests in a loop wasn’t exactly speedy, so I set out to improve the performance by making this process asynchronous.

Of course, you can’t improve what you can’t measure, so first I had to find out exactly how slow this was. For that I used a little tool by HP called httperf, which can make a large number of HTTP requests and report on their performance.

For instance, if you run it like this:

$ httperf --hog --server=server.com --uri=/some-url --timeout=10
  --num-conns=500 --rate=50

It will perform up to 50 requests per second to http://server.com/some-url, up to a total of 500, waiting up to 10 seconds for each request to complete. It has support for setting request headers, HTTPS, testing whole sequences of requests in a session and even cookie management, so you can test whole HTTP flows.

The service I needed to test requires some POST parameters. Googling for how to specify them led me to this gist which uses the session execution feature of httperf. First I created the session specification file httperf_content:

/ method=POST contents="param1=abc&param2=def"

And I executed it 150 times, with no time between session steps:

$ httperf --hog --server 127.0.0.1 --port 5000
  --add-header="Content-Type: application/x-www-form-urlencoded\n"
  --wsesslog=150,0,httperf_content

Which reported the following results:

Total: connections 150 requests 150 replies 150 test-duration 470.479 s

Connection rate: 0.3 conn/s (3136.5 ms/conn, <=2 concurrent connections)
Connection time [ms]: min 524.7 avg 3136.5 max 15894.0 median 2735.5 stddev 2074.0
Connection time [ms]: connect 0.2
Connection length [replies/conn]: 1.000

Request rate: 0.3 req/s (3136.5 ms/req)
Request size [B]: 163.0

Reply rate [replies/s]: min 0.0 avg 0.3 max 1.0 stddev 0.2 (94 samples)
Reply time [ms]: response 3136.1 transfer 0.3
Reply size [B]: header 145.0 content 164.0 footer 0.0 (total 309.0)
Reply status: 1xx=0 2xx=149 3xx=0 4xx=0 5xx=1

CPU time [s]: user 67.66 system 399.51 (user 14.4% system 84.9% total 99.3%)
Net I/O: 0.1 KB/s (0.0*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

Session rate [sess/s]: min 0.00 avg 0.32 max 1.00 stddev 0.21 (150/150)
Session: avg 1.00 connections/session
Session lifetime [s]: 3.1
Session failtime [s]: 0.0
Session length histogram: 0 150

On line 8 we can see that the server can answer an average of 0.3 requests per second, each one taking 3136.5 ms.

On line 14 we can see a breakdown of the status codes returned by the service; apparently there was 149 200 OK responses and one 500 error.

I made my code asynchronous with requests_futures; a simple wrapper around Kenneth Reitz’s requests library that combines it with Python 3.2+ futures implementation. The futures library has been backported to Python 2.7, so even if you are still using Python 2 you can check it out.

import concurrent.futures
from concurrent.futures import ThreadPoolExecutor

from requests_futures.sessions import FuturesSession

successful = []

session = FuturesSession(executor=ThreadPoolExecutor(max_workers=4))
futures = {}
for service in get_services():
    future = session.get(service.url)
    futures[future] = service.name

for future in concurrent.futures.as_completed(futures):
    r = future.result()
    if r.status_code == 200:
        successful.append(futures[future])

print successful

And, according to httperf, it performs much better:

Request rate: 0.8 req/s (1248.2 ms/req)

I tried several values for max_workers, but after 4 increments in performance were negligible.

I also tried using stream=True in my requests, so that my service only read the headers and didn’t bother with the body of the request, but for this services the request body is minimal and it didn’t make much difference.

So, if you are building a web service and you are worried about performance, you can test your optimization ideas and then use this tool to see what works.