test_detective

Performance testing RabbitMQ with JMeter for reliable async pipelines

Engineering teams in Australia ship more event-driven services than ever, and several of the largest tech employers in Sydney and Melbourne — Atlassian, Canva, REA Group, the new owners of Afterpay — run brokers that process millions of messages a day. When those queues slow down, customer-facing checkout, search ranking, and notification flows all feel the hit at once. Performance testing RabbitMQ before that ever happens is not a nice-to-have.

JMeter is familiar to anyone who has load-tested a REST endpoint, but using it against a message broker is a different shape of work. Producers and consumers run on different timelines, the broker has its own buffers and disk pages, and the failure modes are softer than a 5xx. This guide walks through a practical setup: a RabbitMQ instance you can stand up in an afternoon, a JMeter plan that publishes and consumes AMQP messages, and metrics that tell you whether the system can hold a real workload from places as far apart as Sydney and Perth.

By the end, you should be able to set SLOs around throughput and tail latency, spot the most common broker misconfigurations, and feed the results into a CI pipeline so regressions get caught before they reach production.

Why message queues matter in modern backends

A message queue gives you three things at once: decoupling between producers and consumers, backpressure when downstream is slow, and a buffer against traffic spikes. RabbitMQ implements AMQP 0-9-1, which means clients speak a binary protocol over a persistent TCP connection. Messages can be routed by exchange type — direct, topic, fanout, or headers — into one or more queues.

In Australian software shops, this pattern is everywhere. A lender might process loan applications through a queue so its API stays responsive when a credit-check provider is having a rough arvo. An e-commerce storefront routes order, payment, and dispatch events through a broker so the site keeps taking orders through a Boxing Day sale. The key insight is that the broker becomes a single component whose failure degrades many features at once. Performance testing is what stops that component from causing an outage.

Setting up RabbitMQ for load testing

For a first run, a single-node broker on a beefy VM is plenty. A 4 vCPU, 8 GB RAM instance with an NVMe disk is a sensible baseline. RabbitMQ is an Erlang application, so memory and scheduler behaviour matter more than raw clock speed; the scheduler wants to keep work on the same core as long as possible.

The configuration that drives throughput lives in rabbitmq.conf. Set the memory high watermark well below the OS limit — 0.6 of available RAM is a reasonable starting point. Disk free limit should never dip near the 50 MB default, because the flow-blocking mode that follows will distort any test. Channel max, connection max, and queue length should be lifted from the conservative defaults before you drive real traffic.

If you are on AWS, the Sydney region (ap-southeast-2) is the natural pick for an Australian test environment; round-trip from Melbourne or Brisbane stays under 30 ms. Spinning up the broker in Singapore while clients sit in Sydney adds 80–120 ms of network latency per publish, which masks the real client behaviour. A single-node container with the management plugin on 15672 is simple to run with docker-compose, and the UI lets you watch queue depth during a JMeter run.

Building a JMeter test plan for AMQP

JMeter does not speak AMQP out of the box, but the jmeter-rabbitmq plugin adds a sampler, a config element, and a listener that cover the protocol cleanly. Drop the JAR into lib/ext and run JMeter in non-GUI mode for any serious test.

The plan starts with a Thread Group whose threads represent producers. Each thread holds a single AMQP connection and reuses a small pool of channels; opening a channel per message is one of the easiest ways to tank throughput. Use a CSV Data Set for varied payloads, or stick with a constant string for the first iteration. Body size is the single biggest knob — moving from 1 KB to 16 KB typically cuts throughput by a factor of three on the same hardware.

A second Thread Group runs consumers, with threads that subscribe and acknowledge or reject messages. To match producers to consumers across the test, set a correlation ID on every outgoing message and assert on it inside the consumer side. JMeter's Regular Expression Extractor or a JSON Assertion can pull the ID out so you can build a hit/miss counter that tells you whether the broker dropped anything along the way.

Producer-consumer scenarios and throughput goals

There is no single right answer for what throughput to target. Start from the steady state your system already handles: take a weekday morning reading from production metrics, multiply by a growth factor, and that becomes your baseline SLO. If the queue moves 1,200 messages per second on a typical Tuesday, the test should comfortably hold 3,000/sec before alert fatigue sets in.

Burst tests tell a different story. Many Australian retailers see a 20× spike on Black Friday and Boxing Day, while a payments processor might see a 5× spike at EOFY when businesses settle accounts. A burst scenario ramps producers to 5× steady state for a short window then drops back, and you watch how quickly the queue drains to empty. Healthy brokers recover in seconds; unhealthy ones stay backed up for hours because the consumer pool is sized for the steady state.

Latency targets belong in percentiles: p95 under 200 ms and p99 under 1 s for publish acknowledgement is a generous SLO for most workloads. Throughput without a latency budget is meaningless, because the broker can always go faster by buffering more.

Reading the metrics that actually matter

The RabbitMQ management UI is a good first stop. Watch the message rates graph — "publish in" and "deliver get" should track each other in steady state. Queue depth and the count of unacknowledged messages are the two numbers that tell you whether consumers are keeping up. A growing queue depth under a stable publish rate is the most common signal of a problem.

JMeter's Aggregate Report gives mean, median, 90th, 95th, and 99th percentiles for the consumer side. For producers, the key number is end-to-end publish latency — the time between opening the publish call and getting a publisher confirm. Use the Backend Listener to stream samples to InfluxDB or Prometheus and graph them in Grafana; the JMeter HTML report works for one-off runs but is awkward to compare across days.

On the broker itself, watch Erlang VM scheduler utilisation, file descriptors in use, and the memory and disk alarms. A saturated broker hits the memory high watermark before the CPU peaks, and the flow-block mode that follows looks like throughput collapsing near zero.

Common bottlenecks in broker and client config

The four most common misconfigurations, in order of frequency: prefetch too low, channel count too low, durability set wrong, and TLS terminating in the wrong place.

Prefetch controls how many unacknowledged messages a consumer can have in flight. A prefetch of one serialises work and crushes throughput; a prefetch of 1,000 is a queue-depth problem waiting to happen if consumers process slowly. The right value depends on message processing time — measure it for a single message, then set prefetch to cover roughly 50 ms of work.

Channels are cheap but not free, and producers that open a new channel per publish quickly exhaust the broker's channel table. Reuse one channel per thread. Connections are heavier still; one per producer thread is fine for hundreds of threads and you only start pooling at thousands.

For durability, classic non-durable queues are faster but lose messages on restart; durable queues with persistent messages cost I/O but survive. For payments, durability is non-negotiable; the small throughput cost is the price of not reconciling manually after a crash.

Wiring results into CI pipelines

A performance test that lives in a wiki dies on the first refactor. The test should run on every meaningful change — major dependency updates, broker upgrades, consumer rewrites — and the results should land somewhere the team already reads dashboards.

The cheapest path is to wrap the JMeter plan in Taurus, which can run JMeter, Gatling, or Locust and emit a pass/fail decision based on YAML thresholds. Thresholds should stay conservative: 10 percent regression on p99 latency or 20 percent drop in achieved throughput is a sensible tripwire for the main branch. Block merges on the threshold hit and watch trends over months to catch silent regressions before they surface during a high-traffic event like EOFY or a Wednesday arvo product launch.

Run the broker in the same VPC as the test runners; AWS Sydney or Melbourne — both in ap-southeast-2 and ap-southeast-4 — give you the geographic headroom. Do not point the test at production brokers, even read-only. A test broker with snapshotting or a known seed dataset keeps results comparable across runs, and storing the JTL files and the Taurus report in an S3 bucket with a sensible retention policy means old runs are still findable six months later.

Recommendations for smoother test runs

Performance testing a message queue is not glamorous, but it pays off the first time a consumer starts lagging and you see queue depth climbing in the management UI rather than discovering it through escalation tickets. Set SLOs against percentiles, keep brokers and test runners in the same region, and treat the test plan like any other piece of code — versioned, runnable from CI, and reviewed when it changes. Australian teams running real workloads on RabbitMQ have the geography and traffic patterns to learn faster than most markets; the rest is just writing the test and waiting for the graph to tell you the truth.