Questions tagged [queue]
The queue tag has no summary.
69 questions
0votes
1answer
109views
Matching supersets
In my application I have requests and items, each of which are associated with property tuples (key value pairs, where keys can be repeated) stored in a Postgres database. The goal is to find a single ...
- 159
-1votes
1answer
346views
Airflow DAG vs custom worker to process queue of videos
I need to process a queue of videos in a scalable way. Processing includes number of tasks that can depend on each other. Some tasks are computationally expensive some not (e.g. transcribe audio, ...
- 1
0votes
2answers
4kviews
How to terminate python queue and instruct all consumer-threads to finish their tasks?
I have a multi-threaded application. There is 1 thread that produces a resource and puts it into a queue and many (not constant amount) consumer-threads that get the resources from the queue. When ...
- 111
0votes
1answer
234views
Are self-written queues and linked lists really worth it (better than built-in arrays)?
Let's say I need a queue for my project and the language I use does not have a built-in queue structure. So I decided to implement one myself. So I create an object with elements positions as the ...
- 39
3votes
4answers
2kviews
Iterating through stacks and queues?
Are stacks and queues allowed to be iterated over, conceptually? It “feels wrong” for some reason but I can't come up with a reason why it would be bad to see through the whole thing that's stacked/...
1vote
1answer
506views
What kind of queue should I use for processing large volume of data?
I have a segment of customers which may range between 1 and 5000. For each customers, I need to do some processing by making some query to the database using the data from the customer. The query is ...
- 121
2votes
2answers
1kviews
Is Kafka a good idea for connecting a REST API with Worker which runs large computations per job in a queue?
We currently have a REST API service (let's call it as A) written in Python which triggers various heavy calculation jobs for Worker (W) written in Python as well. Both services are interconnected ...
0votes
1answer
1kviews
Scheduling asynchronous jobs with efficient round-robin algorithm using Redis
We are trying to redesign our microservice architecture-based application to support multiple tenants. We have a simple queue service that will be utilized by other services to queue any asynchronous ...
- 101
-1votes
1answer
36views
Options to maintain queue of resources that will be uniquely returned
I need to maintain a queue of resources in a cache such that when multiple clients try to pop elements from the queue concurrently, each element would be uniquely returned to only one of the clients. ...
- 157
-1votes
1answer
141views
Design approach to decouple DB write to reduce application latency
I want to use DynamoDB to save application operation data at the end of a session for auditing purposes without affecting the actual application latency. Because this date is only for auditing so we ...
0votes
0answers
30views
Is there a way to add impending state changes to a queue on a node server, and is it necessary?
I'm writing an express/socket.io-powered game server for a web game. I have a central map of game state objects, each representing an ongoing match, like so: // map of gameId -> game (primary ...
- 1,124
1vote
1answer
1kviews
Pattern to limit message queue processing based on external API throttling
I have a project where there's an external API which implements throttling. Roughly speaking, I'm allowed to perform N requests per minute. I also have a message queue (Apache Kafka) whose consumers ...
- 167
-2votes
1answer
113views
What is the proper way to define a generic cron-job
Am going to handle a CRON job module which will be a generic module, on the job execution time shall I push the events to a queue so let the respective module will pick and do the respective operation....
6votes
2answers
773views
Design question for handling large volumes of messages in multi-tenant queue
I have a system with two applications interacting via a message queue. Let's call them Producer and Consumer. Some key context is that this a multi-tenancy scenario. Producer produces events based ...
- 169
0votes
1answer
106views
Distributed processing of a large number of queues
I have a rather basic application hosted on Kubernetes, which connects to a Mongo database. The app has a wallet feature. A user can put money in their wallet using real-world payments (e.g. via ...
- 407
