Questions tagged [message-queue]
A message queue is a service that receives events or messages from producers and provides them to consumers.
259 questions
3votes
3answers
414views
Design question for exactly-once processing in a message-driven system using a unique ID
To achieve exactly-once processing where messages are consumed from a queue with at-least-once delivery, many sources (e.g. here and here and here) suggest attaching a unique ID to messages in the ...
- 139
0votes
0answers
141views
How does each application replica read a unique message from the message broker
In RabbitMQ streams or Kafka, messages are not deleted after being consumed. If you have a consumer application replicated across multiple Kubernetes pods, how can you ensure that each pod picks up a ...
0votes
5answers
137views
How can I measure which messages are causing queue backups?
Consider the following multi-producer, multi-consumer system: For simplicity, assume that we only have one queue Messages are dequeued on a first-in-first-out basis Oldest-message-pickup-latency has ...
- 119
1vote
1answer
148views
Background thread processing vs queue based processing for relatively short tasks (max 30-40 seconds)
I need suggestion / recommendation on the design approaches mentioned below. UseCase: I have a usecase where a client uses my system to generate some recommendations. Now, these recommendations are ...
- 19
3votes
3answers
212views
Handling rate limits / delays in consumers without affecting performance of other operations
I have a producer that generates a batch job that consists of multiple operations (approx. 100 - 10000). These operations can be processed in any order, ideally as fast as possible. The processing of ...
- 141
0votes
1answer
108views
How to rebalance data across nodes?
I am implementing a message queue where messages are distributed across nodes in a cluster. The goal is to design a system to be able to auto-scale without needing to keep a global map of each message ...
- 115
7votes
5answers
402views
What can I do to get a message processor to slow down the rate of writes that it is making to a database?
We have this architecture: queue -> message processor (horizontal scaling) -> RDBMS Sometimes external systems dump 10k messages onto the queue and the message processor of course dutifully ...
- 229
0votes
3answers
1kviews
Backend to client communication: WebSocket or Message Queue?
I have a hobby project, for simplicity imagine a poker game, consisting of a backend (Spring) and a client application (Flutter) where the client can join rooms in which he can take a seat. The client ...
- 9
2votes
4answers
231views
Ensure concurrent processing based on a message key
I've been wondering about the following: Say you have an async message/task and you want that message only processed one at the time. for example: Only process one order for each customer at the time. ...
- 174
0votes
2answers
275views
How to implement event payload isolation in an event driven architecture?
I have an event, let's say order.placed The payload consists of some general information about the order and 3 confidential fields. For example part of the message should only be visible to consumer A,...
0votes
1answer
85views
Design a sequential processing of records
About 10 to 15 records are processed per day with the time interval of 5 minutes between each record. System A inserts a record in DB and sends id of that record to active mq. System B Listener ...
- 119
1vote
0answers
116views
golang: pattern for handling message queues? Are named functions anti-idiomatic somehow?
Had a discussion today in how to implement services that work with messages coming in from event queues. We call these services processors. One of us argues for using several functions, while the ...
2votes
0answers
118views
parent correlation id in message with rabbit queue and open telemetry
I have a rabbit queue that I use to communicate between applications. I have a chain of three message. App 1 generate and push a message in rabbit(message 1) App 2 catch the message and generate ...
- 307
3votes
2answers
420views
Can multiple producers cause events with the same key to be enqueued out of order within a specific partition of a topic?
I'm aiming to create a simple event-driven system where each microservice operates with its own database. The idea is to share database changes across microservices through events. To ensure proper ...
0votes
1answer
127views
Is RabbitMQ suitable for allowing SaaS customers to subscribe to events?
I work on a SaaS system which is generally a single-tenant environment. Customers (whom the SaaS instances are for) would like a way to subscribe to events, such as updates of certain data entities. ...
- 101
