For most of our services we have custom JMeter based performance tests, they try to mimic production usage of given service. JMeter has built-in support for generating random values, you can also add a CSV file with values copied straight from production, add custom delays, etc. It is nice but this is in fact an approximation. There is nothing like production ;-). Testing in production still sounds a bit scary, but one of the techniques named among Testing in Production (TiP) patterns is traffic replication.

Also, somebody has way too much time on their hands. pic.twitter.com/Gae1nhkG8Y

— Jeff Darcy (@Obdurodon) August 17, 2015

This can be approached from several different angles, if you have something like Varnish or haproxy in front of your service most likely you can configure this quite easily. Another way is to log requests in your application and then replay request using a simple curl (or something similar) but this requires additional work.

After looking at different possibilities I came across Gor. Gor gives you something a bit different than two usecases mentioned above. In general – it can give you a flavour of continuous performance testing. So, long story short — what Gor can do for you?

In first place you need to install Gor on one of your production machines.

wget https://github.com/buger/gor/releases/download/v0.10.1/gor_0.10.1_x64.tar.gz
tar -xvf gor_0.10.1_x64.tar.gz

Gor is written in Go, so after executing this you will end up with single, self-contained executable file ;-). As a warm up let’s record some requests into a file.

sudo ./gor --input-raw :8080 --output-file requests.gor --http-allow-method POST

As you see in the example above, you can limit recorded requests only to POSTs (complete list of selectors can be found in Gor’s documentation). You can also rewrite recorded requests in all sort of ways using your own custom scripts. Nice. It is time for first downside of Gor — there is no stop button, in order to stop the recording you need to shutdown the Gor process (correct me if I am wrong).

In order replicate traffic, you need to cast the following spell:

./gor --input-file requests.gor --output-http "http://test01-example.com:8080"

Gor has some nice additions, which allows you to use this traffic for performance tests, you can specify the multiplier which will increase (or decrease) number of requests.

./gor --output-http-stats --input-file "requests.gor|200%" --output-http "http://test01-example.com:8080"

If you want to setup the constant traffic replication from your production machine into staging, you can run Gor in the following way:

sudo gor --input-raw :8080 --output-http "http://test01-example.com:8080"

It is worth to mention that this is the original usecase of Gor and the reason why it was created (this explains why it does not have STOP button).

You can now sit and look at dashboards of your test instance and compare response times / error rates etc with production. In practise this may be of course a bit more complicated, but this is a perfect case for read-only services which is not that rare this days.

As for downsides:

• Both traffic recording and replication adds load to your production machine, you need to remember about this. In general I would prefer setting traffic replication using external proxy, but in my current use case this fits perfectly
• Gor requires root access, which might be problematic in some hosting environments
• Gor can loose significant portion of requests — which is definitely not nice but it is getting better with every release.
• Lack of switches which allows to disable/enable Gor for a while is a bit annoying
• Replaying requests from file does not leave original time spacings between requests 🙁

Take a look at Gor, it is not perfect but it embodies a very useful idea. I wonder what do you think about it and how do you deal with traffic replication.