A message queue provides a way to send data messages between servers, but instead of delivering them directly and immediately (synchronously), we first pass them to an intermediary queue which saves the message and reliably sends it on to a single endpoint (asynchronously) ensuring it is definitely delivered once.
What kinds of data can be sent?
That depends on the provider:
- RabbitMQ - binary only
- Google Pub/Sub - binary only
- Amazon SQS - binary only
- Zeplo - send data as a HTTPS request (which can include binary, JSON, etc)
What are message queues good for?
- Long running execution
- Throttling requests (to balance load)
- 3rd Party APIs
- Keeping databases/services in sync
- Delayed requests
- Regular interval based requests
We'll go into each one in more detail below.
Long running execution
If you need to execute code that will take a long time to run, it's better to provide the user with an instant response to say that you've received their request, and then to continue the work in the background. You can then notify the user when the job is complete (perhaps with an e-mail or a loading screen).
Depending on your setup or use case, you may have large bursts of traffic or requests (for example, during a sports event you may get lots of image uploads). Instead of rejecting those requests, you can simply pass them to your message queue (which is much less resource intensive), and then process them in batches based on how much CPU and memory is available.
3rd Party APIs
It can be difficult working with 3rd party APIs. Here are
- Changed API - if a 3rd party changes their API design, your requests will start failing. By adding messages to a queue, you can store messages until a fix is implemented.
- Rate limits - it's easy to run into rate limits, and every app has a different way of implementing them. Using a queue you can retry messages after the required rate limit reset duration.
- Downtime - you can't control downtime on external apps, but your customers still expect their requests to be processed eventually, so when their app comes back you'll need to make sure their requests are honoured
If you have a number of microservices, then you need a way to communicate between them. It's common to use HTTPS for this, but sometimes a service will be temporarily down or overloaded with requests. By pushing into a queue, you can make sure that every request is handled when the server comes back up.
Keeping databases/services in sync
It's common to have two databases or services that you need to keep in sync. For example, you might have a Redis cache and a Postgres database or you might want to keep your DB in sync with a 3rd pary app. If you update one, but then the request to the other fails your data will be left in an inconsistent state. This can lead to extremely hard to diagnose bugs and is time consuming to resolve.
Sometimes you need a request to be slightly delayed before sending it. There are a number of reasons why you might need this.
Here are some example use cases:
- Send an e-mail 30 minutes after a user signs up
- Delete a users account data after 7 days
- Send notifications to subscribed users, after 1m (in case the of user undo)
Regular interval based requests
There are a number of use cases where running code on an a regular interval can be useful.
- Reset a customers usage every 30 days
- Send a Slack message to your team with key metrics
- Send an activity summary e-mail every day
Do you send webhooks to your users based on activity on your app? If you do, then what happens when their service is not responding? Do you simply drop the event? In this case, it's best practise to push the event into the queue, to ensure that the event is definitely sent to the webhook recipient.