Questions tagged [distributed-system]
This tag can be used by questions on distributed system concept, design, and implementations.
211 questions
0votes
0answers
69views
For distributed workflows - tradeoffs having stateless workers operate on central state, versus stateful workers?
I'm working on a problem right now that processes incoming data at a very high rate. Each event that flows in has an association ID, and each group of associated events will affect behaviour over time ...
- 121
4votes
2answers
131views
Lock management for isolation in SAGA Pattern
System Overview: The architecture involves two services with separate databases: User-Order Service: Handles user actions such as placing, canceling, and returning orders. Seller-Order Service: ...
- 149
3votes
3answers
414views
Design question for exactly-once processing in a message-driven system using a unique ID
To achieve exactly-once processing where messages are consumed from a queue with at-least-once delivery, many sources (e.g. here and here and here) suggest attaching a unique ID to messages in the ...
- 139
2votes
1answer
232views
How to track external media (binaries) that multiple engineers will update?
I'm not sure if this is the right forum for this--if not, please point me in the right direction. This will be a little long-winded due to the specific nature of my question, so I apologize in advance....
- 137
1vote
3answers
112views
Does quorum protocols circumvent fundamental limitations posed by the CAP theorem?
I can have multiple replicas of my database to ensure high availability. Then I can have a quorum such that R+W > N to ensure consistency. So does strategies like quorum base read/write or ...
- 1,450
0votes
1answer
108views
How to rebalance data across nodes?
I am implementing a message queue where messages are distributed across nodes in a cluster. The goal is to design a system to be able to auto-scale without needing to keep a global map of each message ...
- 115
1vote
2answers
91views
Subscribing to a range of topics per instance of a service [closed]
If I have a system where my distributed service is sending live scores of 1000 s of football games from some hypothetical event to millions of clients. The service subscribes to the games from a ...
- 244
0votes
1answer
113views
how about using kubernetes statefulset to mapping the snowflake datacener id and worker id
I am developing a distribution id project, now using the twitter snowflake id as the fundation of the distribution id. In kubernetes cluster, to fetch the uniq and non-conflict datacenter id and ...
- 103
2votes
1answer
303views
Problem with data consistency in Microservice architecture(2pc transaction) when external dependency envolved
I have microservice architecture. I implement a distributed transaction flow (2-phase commit), kafka, and 1 coordinator service. So, my problem is in each step of preparation I am calling an outside ...
-2votes
1answer
184views
What is the difference between Eventual Consistency, Strong Eventual Consistency, and Causal Consistency?
I read articles about CRDTs and frequently come across the terms 'Strong Eventual Consistency,' 'Eventual Consistency,' and 'Causal Consistency.' What is the relationship between these consistency ...
- 11
-1votes
1answer
86views
Designing a Distributed System for Indigenous Data Sovereignty Across Nations [closed]
I'm looking for some quick "back-of-the-napkin" thoughts from systems engineers on the following scenario: A collection of Indigenous groups (in the US, Canada, and Australia) want to design ...
6votes
4answers
2kviews
Protecting against malicious duplicate IDs in a distributed environment
Let's say we have multiple (somewhat autonomous) (micro-)services, and when entities are created, the ID (UUIDs or whatever) can be set externally. How can we ensure that an ID remains unique across ...
- 71
0votes
0answers
68views
Distributed caching system that scales and enables real-time UI updates?
I have a NextJS app for which I want to cache data from our backend APIs to reduce the load when we have high volumes of concurrent users scrolling through (or search) years-worth of historical data. ...
0votes
1answer
172views
Architectural decision regarding at-most-once semantics cron jobs in distributed systems
please note that while I refer to a specific web application framework in the following, the problem also arises with most other web application frameworks I know so please don't be afraid to reply ...
- 109
0votes
2answers
211views
Fault tolerance in aggregated distributed state
I have a scheduling system that is horizontally scaled, and stores shared state in a redis key. The purpose of the system is to implement something similar to classic rate limiting, but a bit ...
