Tag: Message queue

Cloud Message Queues

~ Steve ~ Leave a comment

More and more of my personal work utilizes message queues (MQ) to integrate systems or to propagate longer running work across pools of workers. AMQP is the 300 pound gorilla in the room when it comes to message queuing. Implementations of RabbitMQ, Qpid, Red Hat MRG, etc.. abound. However, when you are the little guy on the field it can be economical to use a cloud service so you can focus directly on your product. Can cloud message queues be a good replacement for running a MQ yourself?

What I Expect

I’m going to make some assumptions that the service will be available, messages will not disappear (unless I set it to do so) and minor network latency is acceptable.

These are features I expect in priority order:

  1. Central connection point (+5)
  2. FIFO support (+4)
  3. Delivery to first available consumer (+3)
  4. Basic publish/subscribe support (+2)
  5. Push message support (+1)

Options

As it turns out there are more players in the cloud messaging space than I would have thought! A quick search turned up the obvious Amazon SQS along with IronMQ, stormmq, Softlayer Message Queue and Marconi.

Amazon SQS

I tend to think Amazon’s SQS is probably the default MQ as a Service. So many people use AWS and it’s right there ready to be used.

  1. Central connection point: Yes (+5)
  2. FIFO support: No
  3. Delivery to first available consumer: Sort of… (+1)
  4. Basic publish/subscribe support: Yes (+2)
  5. Push message support: Sort of… (+0)

There is no doubt that Amazon’s SQS is a great system but right off the bat it’s obvious it doesn’t meet what I expect. According to the FAQ:

Q: Does Amazon SQS provide first-in-first-out (FIFO) access to messages?

No, Amazon SQS does not guarantee FIFO access to messages in Amazon SQS queues, mainly because of the distributed nature of the Amazon SQS. If you require specific message ordering, you should design your application to handle it.

The first consumer who makes an API request will get the next message. That is sort of delivery to the first available consumer but not exactly.

Push messaging is not directly supported but long polling is available. This is close enough that I’d consider it.

Q: What is SQS Long Polling?

SQS long polling is a new way to retrieve messages from your SQS queues. While the traditional SQS short polling returns immediately, even if the queue being polled is empty, SQS long polling doesn’t return a response until a message arrives in the queue, or the long poll times out. SQS long polling makes it easy and inexpensive to retrieve messages from your SQS queue as soon as they are available.

Feature Result: 8

Result: 8/15

IronMQ

Comparing IronMQ to Amazon SQS was interesting. Unlike SQS, IronMQ uses a REST interface which I feel simplifies MQ as a web service. I played a little bit with the service and it was much speedier than I thought it would be! I even tried the beanstalkd support but wasn’t able to get it to fully work.

  1. Central connection point: Yes (+5)
  2. FIFO support: Yes (+4)
  3. Delivery to first available consumer: Sort of… (+1)
  4. Basic publish/subscribe support: Yes (+2)
  5. Push message support: Not really…

Again there is a hiccup on delivery to the first available consumer. Just like SQS, IronMQ is based off requests from the clients. It could match if a client requests only when it can fully dedicate itself to the next message.

Unfortunately, the push support in IronMQ doesn’t cut it for me. If I’m reading the documentation correctly it makes an assumption that the consumers are all listening HTTP servers. I see the use case for this but I also wouldn’t want to make some or all of my consumers publicly listening on the Internet and spinning off work in another thread or process. I’d rather long polling.

Feature Result: 12

Result: 12/15

stormmq

This didn’t get any testing whatsoever. According to the features page it will have a GA of Q1 2013 (which was earlier this year..). I sent a message via Twitter to find out if I’m just seeing old data on their site. For now I will assume that the statement that it’s AMQP 1-0 is true.

  1. Central connection point: Yes (+5)
  2. FIFO support: Yes (+4)
  3. Delivery to first available consumer: Yes (+3)
  4. Basic publish/subscribe support: Yes (+2)
  5. Push message support: Yes (+1)

This was the first service which would meet all of my wants. However, I can’t sign up for it or use it so it kind of knocks it off the list.

Feature Result: 15

Penalty for not being available: -15

Result: 0/15

Softlayer Message Queue

  1. Central connection point: Yes (+5)
  2. FIFO support: No
  3. Delivery to first available consumer: No
  4. Basic publish/subscribe support: Yes (+2)
  5. Push message support: No

Like SQS, Softlayer Message Queue notes that FIFO is not supported:

Does SoftLayer Message Queue provide first-in-first-out (FIFO) message access?

While the system does it’s best to return messages in FIFO order, it is not guaranteed. A number of factors can influence message ordering, including message timeouts and updated visibility intervals.

The FAQ also notes that it is possible to have conditions where a consumer (or consumers?) may get the exact same message multiple times.

How can multiple readers access the same message queue, without losing messages or processing them many times?

Queues can be accessed by any number of consumers/clients. When a consumer requests messages from the queue, each message is marked invisible—this prevents other consumers from receiving the same message. However, the distributed nature of the message queue cannot guarantee single delivery of any one message. While the system makes a best effort, clients should expect the possibility of receiving the same message multiple times.

Since the MQ service is web service (REST) based there is no pushing of messages, only pulling. I didn’t see anything noting long polling or even push to HTTP servers like IronMQ.

Feature Result: 7

Result: 7/15

Marconi

  1. Central connection point: Yes (+5)
  2. FIFO support: Yes (+4)
  3. Delivery to first available consumer: Sort of.. (+1)
  4. Basic publish/subscribe support: Yes (+2)
  5. Push message support: In progress…

Since Marconi is primarily web service based (REST) it has the same issues as Amazon SQS and IronMQ.

Push messaging is not currently available from what I can tell but there is ZMQ bindings in the work with AMQP work coming later.

The negative part to using Marconi is that one either needs to be using an OpenStack based service already or they will need to set up the service themselves. This does add some overhead to using it. Also I keep wanting to say macaroni.

Feature result: 12

Bonus for being Open Source: +2

Penalty for needing specific software: -5

Result: 9/15

Conclusion

So can cloud message queuing be a good replacement for running your own service? It highly depends on what you are doing, but if you don’t need all the features of modern MQ systems then you can run with a cloud MQ and cut some time/cost.

It may seem unfair, but I am wary of using stormmq because of the old GA info on the site even if they went GA today. At a later date after GA I’d consider trying them but I’d want to see a bit better messaging (no pun intended) via their site.

For me it makes the most sense to use IronMQ until I end up on an OpenStack based system at which point, if ZMQ or AMQP is available, I’d likely switch over. Having ZMQ/AMQP’s push ability would be worth the move.