"Unless I see,[...] I will not believe." John 20:24-29.
If you are a developer, functional tests are as valuable as unit tests, especially with distributed architectures. Your computer is cute, fast and tough, but it's not the real world. Unit tests are mandatory : first it works, then it works fast. You can bench parts of your code, and watch its guts, but you need to watch the whole application in real conditions.
If you are a user, or worse, a sysadmin, you need to watch the application running from the outside. You don't need to audit every applications you use, but you should hurt them before.
Most benchmark tools are designed for http. Some of them are ugly like ab or jmeter some others are beautiful like wrk or tsung. Not all services are REST. Basho_bench is a benchmark tool, from Basho. Sometimes, software names are explicit enough. Basho is one of the spearhead of the Erlang world (with Nine Nines ). Basho built Riak, and when an Erlang tool is missing or squeaky, they build a new one. Rebar is the most famous tool from Basho, but Basho_bench is without a doubt as nice.
Basho_bench provides plumbings and maths. Scenarios are simple to write (the documentation is nice and useful), and the graphs are just as beautiful as they are meaningful. Think of Tsung. Now think the opposite and you will get Basho_bench.
Basho_bench is not Tsung, its target is not to hurt http website with a lumber with complex stories. The target of Basho_bench is A/B testing : pissing contest as well as performance regression tests.
Default drivers are provided, but it's easy to write new one : Erlang is concise and simple. Just read the documentation, experiment, and come back to see how to make your own driver.
For my own needs, I built a Carbon driver. The protocol is dummy enough to be a perfect playground. See the conciseness of the following example :
{mode, {rate, 1}}. % one hit per second {duration, 5}. % minutes {concurrent, 100}. {driver, basho_bench_driver_carbon}. {operations, [{set, 1}]}. % every action are 'set' {key_generator, {pareto_int, 100}}. % pareto, the 80/20 guy {carbon_batch_size, 150}. % throw 150 keys/values {carbon_keep_connect, true}. The driver itself is small : https://github.com/bearstech/basho_bench/blob/master/src/basho_bench_driver_carbon.erl
You can use any protocol known by erlang ( http , mysql , postgresql , memcache, redis , thrift , protobuff ...) or build your own client, Erlang loves protocols.
The api use an abstract action concept (reading, writing, searching ...) and you choose ratio of actions in your configuration file. Don't bother with error handling, let your scenario crash, it's Erlang. Don't be afraid, the supervisor will recycle your broken worker, it's green. Every crash will be counted to draw a red line in the graph. Your workers are stateful so you can imagine different steps in your scenario, like starting to fetch a fresh token before going further or any other sequential actions.
If you are using statsd like tools, it could be nice too correlate performance issues and metrics. R learning curve is almost flat, it's an opportunity to play with it. You can be conservative and play with your Excel/pyplot/gnuplot it's your choice, and it's just CSV files.
Basho_bench is a powerful and simple tool : easy to learn, easy to extend. Try it, just for the beauty of the graph.
