![]() ![]() ![]() These are just the basics, but for a proper tutorial please check out the amazing Procrastinate documentation! Our learnings Monitoring doing: The job is currently being processed by a worker.todo: The job is ready to be picked by a worker.Procrastinate jobs have one of four states: append ( await wait_for_order (guest ) )īy deferring a task, a job is written to the queue and a worker will pick it up: async def main ( ) :Īwait take_order. taskįor guest in guests_at_table (table_number ) : You can define a task by decorating a function. Procrastinate is asynchronous at core, which is great as our service is also async and we don't have to mix synchronous and async code.It supports PostgreSQL as a message queue, which is already in use in our service.Besides the great naming this has two reasons: These are great packages, but we went with procrastinate. dramatiq simpler alternative that supports Redis and RabbitMQ.celery popular package that supports different backends (e.g.Now that we decided to switch to a task queue, we need to choose one of the many available frameworks in python. For the task queue waiter we just add another task to serve food and add it to the queue as soon as it's ready. With the cronjob waiter we would need to integrate serving food into the existing task or create a new cronjob that runs every 10 minutes. Instead, they can call the waiter whenever they want to order something.īy now our customers can order food and drinks, but the waiter doesn't have a task to serve food yet. For our restaurant service this would mean that guests don't need to wait for the waiter to come to the table every 15 minutes anymore. It also enables eventual consistency by listening to events or webhooks and triggering tasks accordingly. This is great if you need to manually or automatically execute a single task or a subset instead of all tasks. If the waiter needs more than 15 minutes to process all tasks and the number of tasks in the queue keeps increasing, we can just increase the number of waiters without assigning specific tables to waiters.Īnother advantage is that tasks can be triggered independently from another. After 15 minutes the task to ask table 4 for orders is written to the queue again. ask table 4 for orders, and processes it. ![]() The waiter picks a task from the queue, e.g. In order to enable horizontal scaling we decide to switch to a task queue. But if the number of tables (tasks) keeps increasing, the waiter will need more than 15 minutes to process all tasks at some point. As long as the waiter needs less than 15 minutes you're fine. Every 15 minutes the waiter goes to every table and asks for an order. We start out with a cronjob waiter (worker). Let's assume our new service is a restaurant. In order to restart the whole job, you just need one kubernetes console command. We catch if a task fails and send a notification once all tasks are completed. Since this is essentially one job looping over every task, it's fast to implement and easy to monitor. Whenever that's the case, our go to solution is to create a kubernetes cronjob, that handles all tasks. Most of my team's services need to crawl data or do computations regularly. I'm part of a data science team at LeanIX that builds python based micro services. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |