There are two main approaches to retrieving queued messages when using a message queue.

  • Push (using HTTPS*) - messages are pushed to the server to be processed
  • Pull  (using streams) - messages are pulled by server doing the processing

TLDR; Use push - it provides better visibility, scalability and reduces developer complexity. Make sure you use a HTTPS proxy message queue, like Zeplo.

Pull doesn't work with serverless

The most obvious reason to use push instead of pull is if you're using a serverless architecture. Pull can't be used in serverless, because the pull method requires the server (processing the message) to be always running. Pull needs a connection to the queue server to be constantly active, incase a new message comes in. As almost all serverless providers limit execution time to a maximum of 15 minutes, you simply can't use pull with serverless.

Pull is harder to scale

You've most likely already spent time and effort ensuring that your existing API service is scaleable. It can already dynamically respond to changes in the number of incoming HTTPS requests, creating and destroying resources as needed. With push over HTTPS, you can re-use all this existing logic.

However, if you use a pull approach, you have to duplicate your entire scaling and monitoring infrastructure to support scaling nodes based on CPU, memory and queue backlog pressure. Doubling the devops work required.

Pull is not more performant

It's 2021 and we now have HTTPS/2.

Pull creates complexity*

If your service already has a HTTPS API, then using pull requires that you create  a secondary interface into your service (with a different API) creating additional complexity and increasing the likelihood of bugs.

  1. Interface 1 (Sync) - HTTPS (you've already implemented)
  2. Interface 2 (Async) -  Custom message queue format

With push (over HTTPS)*, you can simply re-use the first interface, deciding at runtime whether to run asynchronously or synchronously.

  1. Interface 1 (Sync or Async) - HTTPS (you've already implemented)

* This benefit is only possible when you use a message queue that supports HTTPS proxy jobs (see below)

Pull makes testing harder*

It's very easy to test HTTPS endpoints, there are lots of frameworks and tooling for every language. It has a clear and defined API, and once again, you are probably already doing HTTPS testing. With pull you have to start over, mocking the queue logic and designing a new test framework.

* This benefit is only possible when you use a message queue that supports HTTPS proxy jobs (see below)

Use a message queue that supports HTTPS proxy jobs

Some services, like Google Cloud Pub/Sub and AWS SQS allow for HTTP push, but use a propriety format for sending the HTTPS push jobs. This means you can't re-use your existing API and testing infrastructure.

Services like Zeplo, relay/proxy your HTTPS request exactly as you pushed it (including headers, method and body). That means you can re-use your existing API and testing frameworks exactly as-is. In fact, you can use the exact same endpoint for both asynchronous and synchronous requests!