BigBench

BigBench is a http penetration tool. It allows you to test the performance of any web server with very high loads. It:

  • Is a fancy ruby solution for local and remote load benchmarking
  • Creates about 16% more load then Apache’s JMeter
  • Has an own very easy to use DSL
  • Makes remote testing a breeze with bots
  • Offers an awesome post processor environment to analyze your benchmarks that includes tracking iteration, polynomial regressions, normal distributions, …
  • Offers the ability to hook in and execute any code after the benchmarks are finished
  • Comes with included post processors that create statistics and graphs
  • Is very easy to extend!

Installation

gem install bigbench

Build Status

Further Reading

Requirements

  • Ruby 1.9+
  • Redis – only if you’re testing with multiple hosts

Test Receipts

How do the test receipts look like? As easy as possible. For example like this in example.rb:

BigBench.configure do |config|
  config.duration   = 2.minutes
  config.output     = "example.ljson"
  config.users      = 5
  config.basic_auth = ['username', 'password']
end</code>

<code>benchmark "default website pages" => "http://localhost:3000" do
    get "/"
    get "/blog"
    get "/imprint"
    get "/admin", :basic_auth => ['username', 'password']
end</code>

<code>benchmark "login and logout" => "http://localhost:3000" do
    post "/login",  :params => { :name => "[email protected]", :password => "secret" }
    post "/logout", :params => { :name => "[email protected]" }
end</code>

<code># Each request randomly chooses one of the provided params pairs
benchmark "dynamic parameters" => "http://localhost:3000" do
    post "/login",  :params => [
      { :name => "[email protected]",   :password => "secret"       },
      { :name => "[email protected]",  :password => "supersecret"  },
      { :name => "[email protected]",   :password => "unknown"      }
    ]
end</code>

<code>post_process :statistics

Generator

You can have your test receipts generated!

bigbench generate sample

Single Host vs. Multiple Hosts Testing

You can either test with a single machine right from your local host, or with multiple machines using bots. No matter what, the test receipt will stay the same.

Single Host

BigBench allows you to run your tests against every host from your local machine. The command for this looks like this:

bigbench local example.rb

Multiple Hosts with Bots

BigBench uses a bot design pattern which means you can run your tests from multiple hosts. Everything you need for this is a redis that is reachable from all testing hosts. Every host simply starts a bot that is checking for a new test receipt every minute like this:

bigbench bot redis_url:port redis_password

Then to run the tests from all hosts simply use the same receipt as you would use for a local run and call it like this:

bigbench bots example.rb redis_url:port redis_password

This will upload the test receipt to all bots and make them run it. Every bot reports its results back to the redis and the local machine then combines, and writes them to the output file. So you test with the same receipts and get the same results, no matter if your testing from the local host or with multiple bots.

BigBench Request Structure

Output

How does the recorded output look like? It’s in the *.ljson format which is nothing else but a textfile with a complete JSON object on every line. It looks like this:

{"elapsed":0.002233,"start":1333981203.542233,"stop":1333981203.54279,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.00331,"start":1333981203.5434968,"stop":1333981203.5438669,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.004248,"start":1333981203.544449,"stop":1333981203.544805,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.00521,"start":1333981203.545397,"stop":1333981203.5457668,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.00615,"start":1333981203.546355,"stop":1333981203.546707,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.007127,"start":1333981203.547328,"stop":1333981203.5476842,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.008024,"start":1333981203.548226,"stop":1333981203.5485811,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.008904,"start":1333981203.549105,"stop":1333981203.549461,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.009803,"start":1333981203.550003,"stop":1333981203.55036,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.010678,"start":1333981203.550882,"stop":1333981203.551235,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.011549,"start":1333981203.5517519,"stop":1333981203.552106,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.012417,"start":1333981203.5526242,"stop":1333981203.552974,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.013294,"start":1333981203.553495,"stop":1333981203.553851,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.014166,"start":1333981203.5543702,"stop":1333981203.554723,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.015043,"start":1333981203.555247,"stop":1333981203.5556,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
{"elapsed":0.01592,"start":1333981203.556119,"stop":1333981203.5564768,"duration":0,"benchmark":"index page","url":"http://localhost:3000/","path":"/","method":"get","status":"200"}
...

The advantage with this file format is, that it can be parsed and computed very efficiently because the JSON parser doesn’t have to parse a whole JSON array with with loads of objects but simply one object – line by line.

Post Processors

After the benchmark has finished you can create hooks and write plugins that do something with the collected data. To setup a hook simply use the post_process method to add a block or run a predefined plugin:

# Run BigBench::PostProcessor::Statistics
post_process :statistics</code>

<code># Run a block that could do anything
post_process do</code>

<code>    total_trackings, total_errors = 0, 0                
    each_tracking do |tracking|
        total_trackings += 1
        total_errors    += 1 unless tracking[:status] == 200
    end</code>

<code>    Twitter.post "Just run BigBench with #{total_trackings} trackings and #{total_errors} errors."</code>

<code>end

It’s also very easy to write an own post processor. The basic structure is like this:

module BigBench
    module PostProcessor
        module SamplePostProcessor</code>
            
<code>            def self.run!(options)
                # Do whatever you want here
            end</code>
            
<code>        end
    end
end

You can hook it in with:

post_process :sample_post_processor
# or
post_process BigBench::PostProcessor::SamplePostProcessor

Post Processor Environment API

BigBench automatically supports a great load of functionality for every post processor it would need anyways. This functionality is offered through the

Included Post Processors

By default BigBench ships with a few very useful post processors that might already fit your needs perfectly. The full list of included post processors is shown in the wiki:

Running Post Processors separately

You can also re-run the currently defined post processors or run a separate post processor you never even defined in the first place without collecting the test data again like this:

# Re-run the postprocessors defined in example.rb
bigbench process example.rb</code>

<code># Run a separate post processor independently - the already defined post processors are ignored
bigbench process example.rb statistics

Contribute, create great post processors and send me a pull request!

Load Comparison

BigBench is awfully good at creating high loads on web servers. A quick benchmark comparison to Apache’s JMeter shows that BigBench is able to create 16% more load than JMeter.

Test Configuration: Apache’s JMeter vs. BigBench

Parameter Value
Test Duration 2 Minutes
Concurrency(Threads) 20
Rack Server Thin
Rack Host localhost
Rack Request GET: 200, Body: “Test”
Ruby Version ruby 1.9.3p125 [x86_64-darwin11.3.0]
JMeter Version 2.6 r1237317
BigBench Version 0.2

Test Results

Value JMeter BigBench
Total Requests 48.014 55.484
Requests/sec 377 462
Percentages 100%% 116%

Version History

0.6

  • Added dynamic parameters that can be supplied as an array. Every request randomly chooses a set of parameters.

0.5

  • Changed configure syntax to a common ruby pattern block style
  • Refactored and simplified command line usage with thor
  • Added a generator for test files

0.4

  • Added command line tool to run the post processors again
  • Added command line tool to run any post processor on already collected data
  • Pimped the post processor environment. Available functions now include:
  • trackings array is now available with all hashes of the trackings at once
  • Clustering by any timebase, e.g. 1.second, 20.seconds, or 2.minutes which automatically calculates these values per time slice:
    • average duration
    • requests
    • methods(:get, :post, :put, ...)
    • statuses(200, 404, 403, ...)
    • paths("/", "/logout", "/login", ...)
    • benchmarks("index page", "user behavior", "bot crawling", ...)
  • Polynomial Regression of any Degree for all attributes including the derivatives and a formula printer
  • Statistics with the following values for all attributes:
    • max
    • min
    • mean
    • standard_deviation
    • squared_deviation or variance
  • Gaussian Normal Distribution for all attributes including a formula printer
  • Appearing method to quickly list all appearing statuses, methods, paths in the test results

0.3

  • Added post processors hook that run after the benchmark with a simple plugin structure
  • Added a first basic post processor that computes the benchmark statistics and prints them in the terminal
  • Added ability to execute a block of code after running the benchmark. The code can do anything usefully like send emails, post twitter notifications or startup new servers

0.2

  • Net::HTTP was too slow. Only reached 35% of Apache’s JMeter load. Changed requesting structure to eventmachine using em-http-request
  • Compared to JMeter it can create 16% more load than JMeter now
  • Changed config option from threads to users due to a better understanding
  • Added basic auth support
  • Added params hashes for request content

0.1

  • Initial Version using Net::HTTP
  • Local and bot testing
  • LJSON output
  • Global configuration