Six Anti-Patterns We Found in Our Distributed Messaging Stack

Over the past year, we worked through a long list of reliability issues across our messaging infrastructure. Looking back, what stands out is how few of these issues were actually specific to RabbitMQ, even though that's the broker we use.

The trigger was a series of incidents in the second half of 2025 (about 20 in total), accumulating to over 170 hours of impact across our regional clusters. Only a handful were genuine broker bugs; mostly, what we uncovered when we dug in was of our own making: poor configuration choices, overly optimistic assumptions about delivery semantics and a slow buildup of decisions nobody wanted to revisit because, well, the system kept working.

What followed was a get-well plan, which we have now mostly completed. We migrated to quorum queues, deployed an AMQP proxy in front of every cluster, rolled out idempotency fixes for the hardest-hit services and established SLOs that actually measure what matters. Since July, the incident rate has dropped sharply: a sign that the program is working as intended.

What surprised us as we worked through the plan was how little of what we found was specific to RabbitMQ. The same six patterns would have surfaced with Kafka, SQS, Pulsar, NATS, or really any system that moves messages between services. The terminology changes, but the underlying issues remain the same.

Here are the patterns we kept running into.

1. Treating the broker as a data store

Most broker documentation, more or less explicitly, recommends keeping messages small. How small varies depending on who you ask, but "well under 1 MB" is roughly the consensus. Cross that line and you have stopped using a queue in any meaningful sense; you have started using AMQP as a particularly expensive file-transfer protocol. Memory blows up, replication can no longer keep pace and a single chatty producer can drag the entire cluster down.

The cleanest example we have on file is a service that, somewhat heroically, managed to ship a 128 MB batch payload through RabbitMQ. The broker accepted it without complaint and then everything downstream of it had a bad few hours.

Kafka, perhaps counterintuitively, makes this temptation worse rather than better. The long retention windows make the log look like a perfectly viable durable store, so people start using topics as event-sourced databases or, in some cases, as data lakes. It is the same mistake, just dressed in nicer clothes.

The fix is to treat the broker as a notification mechanism rather than a transport for payloads. If something is large, it belongs in object storage with a reference passed through the queue. RabbitMQ 4.x technically accepts messages up to 16 MB by default, but the right way to think about that limit is as a safety net rather than a target. Our own ambition is to bring everything back below 1 MB, and eventually enforce it.

2. Assuming exactly-once delivery

This was the source of an embarrassing number of our 2025 incidents.

Every mainstream messaging system gives you at-least-once delivery by default. RabbitMQ documents this explicitly; SQS is at-least-once by design; Pulsar and NATS behave the same way. Even Kafka, which markets exactly-once via idempotent producers and transactions, only guarantees that property inside Kafka. The moment a consumer's side effects hit any external sink, you are back to at-least-once. Whether the cause is a connection drop, a consumer rebalance, a broker restart, or a pod reschedule, the outcome is the same: the message gets delivered twice.

Several of our consumers had quietly assumed exactly-once. They were fine 99.9% of the time, until something forced a RabbitMQ restart. Then the same message would get processed twice, downstream state would diverge and we would find ourselves dealing with an incident. Worse, knowing this was the case meant we could not safely roll-restart RabbitMQ at any point. Every restart became a coin flip. The downstream consequence was that security patches piled up unapplied, upgrades stopped happening altogether and routine maintenance became something we approached with a reluctance bordering on dread.

The fix is to make consumer processing idempotent. In our experience, the easiest place to find that idempotency is in the storage layer your consumer is already writing to. Most of our affected services were writing to Vertica, so we leaned on the database itself. In practice we rewrote inserts as merges and chose primary keys with enough care that writing the same row twice would always converge on the same final state. That allowed us to push deduplication down to where the data already lived.

That handled the common case. Long-running transactions turned out to be trickier. When a message was redelivered while its original transaction was still running, the same expensive query would run a second time, leaving two of them racing each other in the database, doing the same work. For these, we built an advisory-lock pattern keyed on the message ID, using the database's own session and query-introspection tables to detect duplicates already in flight and cancel them before re-running. This is, in effect, a lightweight mutex built from infrastructure the database already gave us.

For the rest (services with non-idempotent side effects, fan-out writes or external API calls the database can't dedup for you), we are still figuring it out. The current plan is an explicit inbox pattern with Redis as the deduplication store and Redlock for cross-consumer coordination, though we have not shipped it yet.

3. Holding onto deprecated features long past their warning date

We were running RabbitMQ 3.12 in 2025 (well into its end-of-life window) on top of classic mirrored queues, a feature that had been deprecated for years and was removed entirely in RabbitMQ 4.x. The Kafka and SQS equivalents are easy enough to imagine: clusters running ancient versions in production, deprecated client SDKs that nobody wants to touch or account-level features the vendor has been politely asking you to migrate off of for years now.

The reasoning behind all this was the usual one. Things worked. The migration was non-trivial. Whenever it came up in planning, there was always something more urgent waiting in the next sprint. By the time we forced ourselves to actually look at what was sitting underneath the system, it had become clear that every reliability project we wanted to do (broker upgrades, SLOs, monitoring improvements) was blocked on the queue migration. The deprecated thing was not, as we had quietly assumed, tech debt sitting in a corner. It was the keystone holding back the rest of the roadmap.

If your messaging stack was last updated in the previous decade, that is not stability. That is the kind of debt that wakes you up at 3 AM with a memory leak you have no way to patch.

5. Reinventing retry logic in every consumer

We had a few different ways of handling consumer errors across our services. The simplest was to lean on the broker and requeue forever. A more sophisticated but ultimately brittle approach was to inspect RabbitMQ's x-death header to count retries: the broker compresses entries by {queue, reason} rather than per-redelivery, the semantics have shifted between versions and on quorum queues x-delivery-count is now the modern equivalent. The third option was to have no retry handling at all and to simply let failed messages drop on the floor.

Every messaging system has its own primitives (Kafka has consumer groups and offset commits; SQS has visibility timeouts and built-in DLQs; AMQP has manual acks and dead-letter exchanges), but anyone running on top of any of them eventually finds themselves answering the same handful of questions. How many times do we retry? How do we delay between retries without blocking the consumer thread? Where do messages we cannot process go? And how do we know when the parking lot is filling up?

What we converged on, in our specific AMQP world:

• Manual acknowledgments only, with no autoack on production queues.
• Errors classified as retryable (transient failures like network timeouts or downstream blips) or as fatal (bad data and logic errors that are never going to succeed), with a different handling path for each.
• Broker-side retry through a delayed retry queue with message-ttl and a dead-letter-exchange pointing back to the original. The consumer just nacks; the broker handles the wait, leaving listener threads free.
• A delivery-limit on every queue, defaulting to 20 in RabbitMQ 4.x.
• A dead-letter or parking-lot queue for anything that exceeds the limit, with monitoring on the depth.

None of this is novel; it is reasonably close to vanilla AMQP best practice. The same checklist exists in Kafka and SQS shapes, just with different vocabulary attached. The lesson here, for us at least, was less about building anything clever and more about putting an end to every team inventing its own version.

6. Watching queue depth instead of message age

For the longest time our primary RabbitMQ alert was a threshold on queue depth, set at 80,000 messages. Looking at it now, this is a deeply ambiguous signal. 80,000 messages might represent a 1-minute lag on a high-throughput queue, or a 3-week backlog on a low-throughput one. The number itself tells you almost nothing about whether anyone is being affected. The Kafka equivalent, consumer lag in offsets, suffers from exactly the same problem.

What actually matters is freshness: how long does it take a message to get from publish to consume? That is the property the user feels. Today we stamp every message at publish time and measure the average broker transit time on the consume side, with thresholds set per tier: real-time queues get 100 ms, near-real-time queues get a second and batch queues get 30. Queue depth has not disappeared as a signal, but it now serves as a diagnostic rather than a primary SLI.

The broader point, which applies across messaging systems generally: every signal in your system needs to map back to "is this affecting the user." Resource utilization, queue depth, consumer lag and connection count are causes. Latency, error rate and freshness are symptoms. Symptoms are what you should page on; causes are what you should investigate with.

What ties it all together

The thread running through all six is contracts: between the producer and the broker; between the broker and the consumer; between the platform team and the service team. When contracts are implicit, every team makes its own assumptions, and incidents are how you find out where those assumptions diverge. This pattern is not specific to AMQP, RabbitMQ, or message brokers in particular. It is the failure mode of any distributed system that shares state through unwritten conventions.

Most of our get-well program, when we got down to it, was about writing those contracts down: explicit message size limits, explicit idempotency requirements, explicit retry topology, and explicit SLOs with error budgets. The infrastructure changes mattered, but in the end the documentation mattered more. Code built on assumptions eventually requires a rewrite; code built on documented contracts rarely needs to change.

If these anti-patterns sound uncomfortably familiar, don't try to fix everything at once: pick the worst offender and start there. Distributed systems are messy, and most messaging stacks turn out to be far less reliable than we think. The only way to find out where your system stands is to stop relying on implicit assumptions and start defining your explicit contracts.