STOMP and CoilMQ

I've just pushed out my third release (v0.3) of CoilMQ, a simple STOMP server written in Python.  (You can see the roadmap or the issue tracker to learn about the changes.)  It's been a fun project, so I decided to write about why I've been working on it and why you might want to consider Stomp for message passing.


Project Background: A Twisted Snake-y Road

A few months back I started a simple project to learn more about writing a basic network server in Python.  I decided I would create a STOMP server in Python, because there weren't any -- well, none that I could get to actually work.  And there was this fantastic & simple stompserver Ruby project out there just taunting Python with its simple, concise implementation.

So, I plowed into the project and immediately resolved, as all aspiring Python developers do, that this was perfectly suited to Twisted.  Afterall, the Ruby Stompserver project was built on EventMachine. Nevermind that MorbidQ seemed to have had the same idea (and was firmly on my "not-working" list).  Nevermind that I'd been there before, waist-deep in Twisted entrails trying to grab some slimy Deffered; I made myself write "Twisted is harder than dataReceived" two thousand times.  Nevermind that Ted Dziuba wrote a brilliant tirade on Twisted which anyone that's had to wrestle this Gorgon can appreciate.  Nevermind all that.

Well, that didn't last long.  Sure, I had an original version written using Twisted.  Very simple.  Everything in memory; no blocking I/O.  It looked architecturally like the Ruby EventMachine code.  Then I thought, what about using a database for queue storage?  And that's when the romance ended.  Well, I don't have a database API that returns Deffereds; I just have one that'll stop my app while the query runs.  I know this was really just an educational project & proof of concept, but I didn't feel right doing it wrong.

So, I decided that I'd write a my server using threads for concurrency.  I know it's not what the cool kids are doing, but -- despite the GIL -- it's a great way to handle I/O concurrency in Python and I feel it's a really important thing for developers to understand -- because it comes up all the time.  (I'm no expert, don't get me wrong; this was a learning exercise.)



Why Stomp?

Stomp is a really simple protocol for publisher-subscriber message passing.  It's loosely HTTP-like in its syntax and it works for passing text-based or binary data -- so you can pass whatever data type your consumers care about (JSON, XML, AMF, pickled objects, etc.).  The protocol itself doesn't explicitly prescribe topic or queue implementations, but there is a convention that destinations that begin with "/topic/" are topics (broadcast to all subscribers, no persistence or reliability) and destinations that begin with "/queue/" are queues (sent only to one subscriber, persisted).  Stomp does go beyond the basics to provide support for transactions and reliable subscribers (that must ack receipt).  Some implementations (e.g. ActiveMQ) have also added the concept of durable topics.

There are lots of uses for Stomp because it's been implemented in lots of different languages.  For example, you could use simply use this as a way to pass data (e.g. pickled objects) between disparate web applications written in Python (e.g. using stompy client).  Or you could use Stomp to provide for passing messages between Python application servers and Flash/Flex clients (e.g. using as3-stomp client); that's an easy way to implement server-push for Flex.  To build on that example, you could just as easily push AMF-encoded (binary) messages between different Flash clients.  So Stomp is really quite basic, and very versatile.

Why CoilMQ?

This is a tricker question to answer.  Honestly, if you've got a project out there that needs to push messages at scale, you probably want to look at some of the enterprise brokers.  I've had great luck using ActiveMQ and I've heard people that need to scale to massive throughput suggest using RabbitMQ.  But if you don't need to handle thousands of requests per second, and you do want to understand the software you're using, then I would suggest that a project like CoilMQ is a great choice.

Plus, it'd be great to have some feedback!  I've had a number of downloads & no bug reports (or complaints).  I'd love to think that means it just works perfectly for everyone, but I suspect the truth is that I just need more users.  I should add that I'd also be very pleased to accept any suggestions or patches (the former is greatly helped by the latter).