Addicted to Ti: Habanero Cycles

My first road bike (of my adult life) was a 2002 Lemond Victoire.  It was a wonderful bike, but it had one fatal flaw: it didn't fit me.  Of course it took me awhile (about 4 years, I guess) to fully appreciate this; after all, I had nothing to compare it to.  I didn't realize that hitting ones knees on the front bars when pedaling out of the saddle wasn't the norm, that fairly significant (> 1") toe-front-wheel overlap wasn't a requirement, or that 12.5cm disparity between seat and handlebar height was as appropriate a position for a bike as a Snoop Dogg music video. Apparently I was a little naive.  And by "was", I mean up until a couple weeks ago when I sold that bike and bought a new Habanero Cycles frame.

Habanero Cycles, judging from their site, is a few guys down in Florida that act as a (super-lightweight) middle man for titanium frames made in China.  I think this graphic on the landing page pretty much sums up the shop:

This simple GIF, with its jagged pixel edges and MS-Paint-era aesthetic, eschews modern web design fashions and technologies.  Habanero Cycles with their straight tubes, traditional triangles, and encouragement of 1" head tubes seems to do much the  does the equivalent in the cycling world.

But, perhaps unlike the GIF image above, I think a lot of people would agree that these are really timeless and attractive  bicycles.  I certainly think so.  After all, this is my second Habanero Cycles frame.

Last year, I decided to build a cyclocross bike for commuting, to replace my single speed.  I like riding a single speed, but I wanted a bike that could fit fenders and one that had gears since I was anticipating pulling a baby trailer with it on the weekends.  Here was the bike in "commuter mode" after building it and before actually making it practical for commuting (adding fenders, lights, switching to compact crankset, etc.):

As it turns out, it also works for riding as a cyclocross bike.  I raced in my first cx race this fall and plan to do another in a couple weeks.

My new bike is a road frame, which I have to explain to my non-cycling friends and family is very, very different from a cyclocross frame -- you just can't really tell by looking at it.

(Yes, I removed the down-tube decals; I wasn't a big fan of the orange lettering on this model.)

I think that I would be betraying the "NO HYPE" mantra of Habanero Cycles if I started weaving word tapestries (or at least potholders) from the cycling reviewer lexicon: "lateral stiffness", "vertical compliance", "soak up road chatter", "effortless climbing", etc.  (but not too many cetera, this is a shockingly shallow vocabulary pool).  I do, however, have a few general statements that I would like to make about both frames:

• They are very well built.  The craftsmanship is excellent; the finish fantastic; the welds are beautiful.  (I tried explaining this value of "beautiful welds" to my wife, but she seemed unconvinced that this is why it should cost as much as a personal computer.)
• They are built like (light) tanks.  Strength and durability are values that Habanero Cycles extols and these frames implement those values.  And as a result ...
• They're not super-lightweight frames.  I'd guess that you could probably find high-end designer steel frames around the same weight, but I can't imagine they would be anywhere near as strong or stiff.  My Lemond Victoire frame probably weighed almost 1 lb less than the Habanero that replaced it, but it was very flexy (wheels would rub brakes on out of seat climbing).  I would rather have a solid frame.  And you can save weight in other places.  My 60cm Habanero build weighted in at 17 lbs 3oz (compared to 18 lbs 11oz for my Lemond) -- and didn't break the bank.
• They ride like bicycles.  In truth, I think both frames feel fantastic.  If I had to elaborate, I would say that the ride is similar to steel bikes I've ridden (e.g. my SE Lager single speed or the '80s Bianchi I rented last winter): the frame feels solid & mutes texture on paved surfaces.  To borrow another buzz phrase, these are also laterally stiff frames.  I don't notice them wagging side to side when standing on the pedals.  They both just kinda ride like bikes should.

Perhaps on par with the quality of the frames themselves, though, is the the customer service. I have worked mostly with Mark (I think he does the online stuff for them, and I tend to do more by email than phone) and the customer support has been superb.  We had many back and forth emails where he helped me choose frame size and suggestion other component dimensions to get the geometry I wanted; he's worked with me through some follow-up issues with seatpost slipping (diagnosis is bad seatpost binder bold; he's sending a replacement) -- answering emails on weekends, etc.  I have certainly never experienced such excellent customer service in the cycling industry.

After writing this, I think it's less that I'm addicted to titanium, and more that I'm addicted to the ideals embodied by that "NO HYPE" GIF image.  I like the fact that these bikes don't claim to be anything more than well-built bicycles: they embrace the fundamental triangle design that all other frame designs circle around, use traditional dimensions and geometries, favor strength and durability over weight savings, and represent a really well-crafted product.  I do like the stress-free qualities (low-maintenance finish, no rusting) and ride quality, but I think this is just part of that same bigger picture here.  There's nothing wrong with innovation, but there's often nothing wrong with the original edition either.

Commuter tires: the Continental 4 Seasons review

Choosing tires always feels like such an important decision.  They're obviously an important aspect of a bike's setup, the interface to the environment, and a powerful fashion statement.  It's not a fleeting decision; tires last a long time -- or they're supposed to -- and while you can change them if they're wrong it's a pain and it makes your hands black.  I can only imagine that the weight of this decision is akin to drafting a fantasy football team -- or maybe even a real football team.

And probably much like a football team, you wouldn't want to use the same tires from year to year.  No, you'd want to toss out the thing that worked great (in my case the Panaracer T-Serv tires that I sold with my single speed) and change it up.

Continental Grand Prix 4 Seasons

MSRP $80 (!!!), street price $60+.

I love my (700x23c) Continental Gatorskin tires on my road bike so much I decided that the Continental 4 Seasons were the tires for me.  After all, I do ride for 4 seasons.  The marketing blurb struck a chord in my heart: 

A tire meant for road riders who do it the real way - ride all the time, on all types of roads, in all types of conditions. [...] Get the speed and dependability you need with the 4 Season, built for the long haul road rider.

Perfect! At the time I found them on sale for $47 so the price wasn't just stupid.  I was starting off riding in the winter, so I figured it was a perfect test.  These are lightweight tires at 240g for the 700x25c.

Well, my first rear tire only lasted 300 miles or so before the bead blew out on an otherwise serene ride into work.  Luckily it blew out right near work, so I could walk the bike in the last couple blocks.

I made a halfhearted attempt to get Continental to warranty the tire, but they never got back to me.  I couldn't find my receipt.  I ordered from the internet.  So I walked into my LBS with my head hung low and paid full price for a replacement.  I figured that by paying the price of a one-way airplane ticket across the country, I'd probably be able to at least get that many miles out of it.

Well, after approximately 3,000 miles this is what my rear tire looks like.  (The danger of having full fenders on your bike is that you can forget to check your rear tire.) 

I have to say that I find it a little ironic they're called "4 seasons", since they only seem to last two.  Maybe by "long haul road rider" they actually meant, "someone that takes a short spin on the weekends -- weather permitting".

Maybe 3,000 miles is expected (though it looks like I should have replaced these hundreds of miles ago), but it seems a little premature to me.

I will say that the tires have good grip and they did as well as could be expected on icy trails.  Probably the softness of the rubber is what's accounting for the wear.  I did get a flat once when I drove over glass on the nearby trail, but one can only expect so much of rubber.

So, your mileage may vary.  (Hopefully your mileage is greater than mine.)

CyanogenMod, you saved my Captivate.

I've decided to start writing a few short blog posts again.  I think my problem with "contributing to the internet" is that it does require a certain sense of confidence in the value of one's thoughts or opinions.  I typically don't think mine are worth sharing, but every now and then I do think "this might help others avoid misery".  This is one of those times.  Oh, Samsung Captivate, it is of you I sing.

If you were, as I was, lured away from the iPhone to the Samsung Captivate (the Galaxy S model for AT&T), I am sorry for what you have had to endure.  Don't let this phone drive you away from Android.  There's a light at the end of this tunnel; it's cyan-colored.

To be fair here, there are some good things about the Captivate.

1. The screen.  The Super AMOLED screen on the Captivate is beautiful.  Maybe it's because Samsung makes TVs, but they know how to do color, contrast, etc.
2. Photo and video quality.  The camera is a 5mp camera which takes good pictures (for a phone).  The HD video recording is also really great (for a phone).  It's made it possible to take lots of decent quality, impromptu baby videos and that's really valuable.  The recording app itself is not that great, but the products are nice.
3. And of course, there's Android.  This phone has definitely done a poor job of "porting" it over, but I do love the Android OS.  I can run Python scripts on my phone, build apps without paying for a developer license, run Samba for sharing files to my desktop, view content in Flash, use the latest & greatest Google apps, etc.  And I believe in open software.

... but there are some things that are not so great:

1. Performance.  This phone with the official Android builds is SLOW!  The stock phone came with Eclaire (2.1), which was acceptable but missing enough capabilities that it was a little hard to be slow at anything.  Everyone was waiting eagerly for Froyo (2.2).  I ran a "leaked" Froyo for a little bit, but it was horribly slow.  For example, it would take multiple attempts with the swipe pattern to unlock my phone because it was not recognizing touches; answering calls was hit or miss for the same reason.  Swiping between screens was jerky.  Apps would get stuck in unresponsive states.  It was a disaster.  I tried one of the "aftermarket" kernels that did improve speed but at the cost of a battery that only lasted half a day.  I downgraded to Eclaire.  Then official Froyo came out and I upgraded.  It was better than the leaked version, but not a lot.  Less jerky movements, but still lots of applications hanging for a long time.  Games like Need for Speed were jerky while playing.  If I was honest, it was pretty crappy.
2. Bluetooth.  So probably half of my phone calls are made in the car and I use my car's Bluetooth so that I don't have to be driving dangerously (or illegally) .  Well, the Samsung pairs fine with my car, but it would disconnect after being used for a couple of minutes.  Then it would reconnect.  So imagine driving down the highway at 65mph and your car saying "the bluetooth connection has been lost".  I'm trying to get the phone of out my pocket and yelling "hang on a sec" so that the other party doesn't say anything important that only my nether region could hear. As soon as I get it out of my pocket, the car picks up the link again.  What a piece of crap.
3. Samsung's Android skin and crapware.  Samsung made some sort of iPhone-like skin.  It's not really all that surprising that Apple is suing them.  Their app list was a cheap knock-off of the iPhone screens.  They provided a bunch of really junky applications that you couldn't uninstall.  The camera and video app worked but were pretty annoying to use. There are ways to work around this and replace some of the stock behaviors with Android (the beauty of Android), but the out-of-the-box OS experience was pretty disheartening.  If I was a less technically savvy or less adventurous user, I'm sure I would have returned this phone within a week.
4. Abysmal update schedule.  Samsung is way behind with their software updates.  Gingerbread was released in Dec 2010 and still hasn't been released for the Captivate (it's now Oct 2011).
5. GPS is horrible.  This is probably partly hardware related, but the functionality of the GPS has also been greatly affected by the OS version.  Some versions of the OS have worked better than others (Froyo was an improvement), but it's bad enough that it makes Google Navigation frequently useless (it keeps thinking I've left the route when I haven't, for example).
6. Locked down by AT&T.  They disabled "Untrusted Sources" for installing apps.   So you can only use the official market.  You can obviously work around this with rooting, but one shouldn't have to void their warranty to install something from Amazon's market or from an open-source google code project.

On the whole, I would say that the negatives outweighed the positives quite significantly.  So if you had any less self-discipline-- or faster automatic windows in your car -- your Captivate is probably in a river or the ditch alongside some stretch of highway.  I'm sure you don't regret that decision.  If you do still have your Captivate, though, I have some good news: CyanogenMod.

CyanogenMod is an open-source, aftermarket firmware for numerous smartphones; and as of version 7.1.0, the Samsung Captivate is on the list of supported phones.  (In theory the official ROMs are also open-source, since they have to be by [GPL] license, but they are not developed as open-source projects.)  CyanogenMod was not trivial to install (more below) but was well worth it.  It is a huge improvement and basically makes this feel like a new, and vastly superior phone.  (I've been running this for around a month now.)

• The Bluetooth now works.  I was worried since I assumed it uses the BlueZ Linux BT stack, which doesn't have the best reputation, but it works flawlessly in my car.  No more rummaging in my pocket, while yelling at the person on the other end.
• The camera/video recorder app is vastly superior to Samsung's.
• I have the latest (Gingerbread) version of the Android OS, with all the improvements that brings (not least of which is a much better keyboard).
• Performance is fantastic.  Feels twice as fast as stock Froyo build.  No more jerky unlocking or call answering; no more jerkiness in Need for Speed car racing :)
• CyanogenMod uses ADWLauncher as it's "skin" which is  also vastly superior to Samsung's.
• No more crap applications I can't delete.
• GPS seems to work better than any other OS/version I've tried.  It is still slow to lock on to exact position, but seems to track correctly and not randomly jump a few neighborhoods over.  It is definitely usable now.
• Lots of developer-friendly features.  Obviously things like "Unsigned Sources" can be enabled.  There's also tethering and a while lot more. I haven't scratched the surface of this yet.
• Battery life did not degrade. I would have expected that a phone that runs much faster would suck down battery faster, but I get the same battery life as before (which admittedly is only a day of few calls and moderate network use).

The downside is that you'll void your warranty.  Of course, at this point Samsung's warranty is likely expired for me and even it it was still active, it was only going to guarantee me the crappy performance, poor hardware support, and inferior OS spin of the stock system.  Yeah, that was a pretty easy decision.

Getting it installed is not hard if you have some patience, a strong stomach (you may need to descend into the bowels of the internet or wade through troll-infested forums for help or obscure workarounds). If things go wrong, it may help to have some understanding of Linux systems, but you're going to be following some forum steps either way.  There is a good installation guide on Cyanogen's wiki; follow that and it may just work for you without a hitch.  Basically you will need to install a new kernel which supports ClockworkMod Recovery using Heimdall, then install ClockworkMod Recovery itself, and finally install the CyanogenMod ROM using ClockworkMod recovery application.  So you will first have a stock-ish system running on ClockworkMod Recovery before you then install CyanogenMod.  The beauty of this is that ClockworkMod will let you backup your current ROM completely in case something goes wrong.  Before you start you'll also want to make sure your apps (and their data) are all backed up.  I use Titanium Backup; it does this well.

I had one significant problem following the instructions.  If you are running (stock) Froyo, as I was, you will need to ensure that your "recovery" application is downgraded to version 2e.  The 3e recovery utility will only run/install signed packages, which prevented me from getting the ClockworkMod Recovery boot loader installed.  This thread may help.  Or Google for "2e recovery Captivate" and wade through the results.  When I originally had this problem I was able to find a thread where someone had posted just the 2e recovery utility, and I was able to overwrite the utility through the phone (needed root and a file utility that supported remounting partitions rw -- I told you this wasn't for weak stomachs) -- I'm not able to find that thread right now.  I was happy that I didn't have to boot into Windows to do any of the upgrade steps (I'm running Ubuntu 64-bit).

Anyway, hopefully this proves useful to someone.  If you haven't already thrown away your Captivate, don't; rescue it with CyanogenMod!

stompclient: another STOMP client for the Python community

Astute readers (as if I have enough readers to actually differentiate) may remember that I've previously entered the python STOMP world with my CoilMQ project. And now I'm at it again, but this time from the client side with my (perhaps presumptuously named) stompclient project. Yes, there are other clients out there -- stomp.py, stompy, and stomper to name the main players. So why a new one?  Well, we've been using a couple of these in various stages of production, and based on my experiences I'd decided that there was room for an additional player in this game.

1. I wanted a STOMP client that would be easy to use without getting in the way.  For example, spawning listener threads from within the class (as stomp.py does) is easy; however, if you want to exercise some more control over how received messages are handled, it's in the way.
2. I wanted a STOMP client that would support a publish-only usage model.  Most of my needs to interact with a stomp server from Python have involved writing clients that need to just push messages onto topics/queues (e.g. from python WSGI web applications).
3. I wanted to also provide a helpful set of STOMP library utilities for other projects. Specifically, I wanted to flesh out & clean up the Frame classes that I had started implementing for CoilMQ and add in my fixed version of stomper's FrameBuffer that would traffic natively in frames.
4. I really wanted a better-documented, better-tested, and generally cleaner, more pythonic (pep-8) codebase.
5. And finally, this was really another opportunity to learn more about sockets and multi-threaded application design (& testing) in Python.
   

Unlike HTTP, the STOMP protocol is not a serial request-response protocol.  This actually makes it non-trivial to write a client that can both send and receive messages, since you can't simply sock.send() a frame and then sock.recv() a frame and expect that to the response to your sent frame.    Of course, this makes sense since a subscribing client also needs to be able to receive message frames from the server (without being in response to any request).  So to have a client that can both send and receive messages, there needs to be some sort of receiver loop constantly running. I chose to take inspiration from the way that the stompy does this and use queues.  My approach was a little simpler in that there is simply a listener loop (expected to be run in its own frame) that enqueues any received frames on the appropriate queue (e.g. message frames go on the message queue, receipt frames on the receipt queue, etc.).  Very simple, but seems quite effective. This approach is also flexible, since it means you could create pool of worker threads that all pull from the appropriate queue(s) to process messages concurrently.  (It probably wouldn't be too difficult to also provide a multiprocessing implementation for the worker pool.) Here's the simple publish-only example that pushes some binary content (a pickled python object) onto a queue:

import pickle
from datetime import datetime

from stompclient import PublishClient

client = PublishClient('127.0.0.1', 61613)
client.connect()
payload = {'key': 'value', 'counter': 0, 'list': ['a', 'b', 'c'], 'date': datetime.now()}
client.send('/queue/example', pickle.dumps(payload, protocol=pickle.HIGHEST_PROTOCOL))
client.disconnect()

Head on over to the project website for more examples, documentation, and download links.

CPython Threading: Interrupting

I've decided to kick off a return to blogging with a series on multi-threaded development in Python (CPython, to be specific).  Yes, we all know there's a GIL in the water, but multi-threading is still an extremely useful concurrency strategy in Python for i/o-bound activities ... which tends to characterize most of my use cases for concurrency.  But there are lots of things that make multi-threaded programming tricky and there aren't quite as many resources out there for Python (as there are for Java, say).

Disclaimer: I am not an expert at multi-threaded programming in Python (or any other language). Most of this has been trial & error, some help from the Google, and a lot of foundation from the excellent book on the subject by Brian Goetz: Java Concurrency in Practice (despite the title, the principles in the book apply to Python too). If you know of a better way or better explanation, please leave a comment so we can all benefit.

After getting over some of the challenges of mutable state and atomicity of operations, I think one of the things that probably bit me next in Python specifically was handling of asynchronous exceptions (like KeyboardInterrupt and in some cases SystemExit) -- and specifically how one goes about actually stopping a multi-threaded application.  Too many times I would end up with a script that would just hang when I hit CTRL-C (and I'd have to explicitly kill it).  So let's start there.

Asynchronous Exceptions

The KeyboardInterrupt exception is actually an OS signal; specifically, the signal module translates SIGINT into the KeyboardInterrupt exception. The rule is that on platforms where this signal module is present, these signal exceptions will be raised in the main thread. The SystemExit exception is similar, in that no matter which thread raises it, it will always be raised in the main thread.  On other platforms, apparently they may be raised anywhere (see the "Caveats" section of the thread module reference documentation for more info); for the sake of focus here, we will assume that you are working on a platform with the signal module present.

Let's start out with a simple example of a multi-threaded program that you cannot abort with CTRL-C:

import time
import threading

def dowork():
  while True:
    time.sleep(1.0)

def main():
  t = threading.Thread(target=dowork, args=(), name='worker')
  t.start()

  # Block until the thread completes.
  t.join()

if __name__ == '__main__':
  main()

The problem here is that the worker thread will not exit when the main thread receives the KeyboardInterrupt "signal".  So even though the KeyboardInterrupt will be raised (almost certainly in the t.join() call), there's nothing to make the activity in the worker thread stop. As a result, you'll have to go kill that python process manually, sorry.

Stopping Worker Threads

Solution 1: Kill with Prejudice

So the quick fix here is to make the worker thread a daemon thread.  From the threading reference documentation:

A thread can be flagged as a “daemon thread”. The significance of this flag is that the entire Python program exits when only daemon threads are left.

So in practice here, if you stop your main thread, your daemon thread will just stop in the middle of whatever it was doing & exit. In many cases this abrupt termination of any worker threads may be appropriate; however, there may also be cases where you actually want to manage what happens when your threads terminate; maybe they need to commit (or rollback) a transaction, save their state, etc. For this a more thoughtful approach is required.

Solution 2: Instruct Politely

The alternative to just killing them is to instruct the thread to stop using some agreed-upon system. You are probably aware (or have guessed) by now that there is no Thread.stop() method in Python (and the one in Java is deprecated and generally considered a Bad Idea™).  So what you must do is to implement a "thread interruption policy" which in our case is basically a signaling mechanism that the main thread can use to tell the worker thread to stop.  Python provides a threading.Event class that is for exactly this type of inter-thread signaling.

The threading.Event objects are very simple two-state (on/off) flags that can be used without any additional locking to pass "messages" between threads. Here is a basic stratagy for using a threading.Event to communicate a 'shutdown' message to a worker thread:

1. You share a "shutdown" threading.Event instance between the threads (i.e. you either pass it to the threads or put it in a mutually accessible place).
2. You set the event from the main thread when you receive the appropriate signal. Here we're focused on KeyboardInterrupt, but presumably users could also take some action within your application (e.g. "stop" button) to stop your application, i.e.
   

   shutdown_event.set()

3. You check it (frequently) in another thread and take the appropriate action once it has been set.

   while not shutdown_event.is_set():
      do_some_work()
   do_some_cleanup()
   

It is probably worth pointing out here that this system is really just some conventions that you've established between your main thread and the workers. If the workers don't periodically check the shutdown event, then they won't stop their work -- and CTRL-C still won't work.

Putting it Together

After applying the threading.Event model to our example, we are able to have our CTRL-C respected relatively quickly (as quickly as the worker thread gets around to checking the event).

import time
import threading

shutdown_event = threading.Event()

def dowork():
  while not shutdown_event.is_set():
    time.sleep(1.0)

def main():
  """ Start some threads & stuff. """

  t = threading.Thread(target=dowork, args=(), name='worker')
  t.start()

  try:
    while t.is_alive():
      t.join(timeout=1.0)
  except (KeyboardInterrupt, SystemExit):
    shutdown_event.set()

if __name__ == '__main__':
  main()

Working around uninterruptable Thread.join()

You may have noticed that we changed how we called Thread.join(). Calling the join() method on a thread without a timeout will block until that thread returns/completes. As I understand it, this is due to a mutex in the join() method which has the implication that you cannot interrupt it with KeyboardInterrupt.  You can work around this, though, by essentially checking in a loop until the thread does exit:

while t.is_alive():
      t.join(timeout=0.1)

Other Events and Exceptions

You may notice that in the compiled example that I am also catching the SystemExit for sake of completeness. In a more complex app, you would need to make sure that other exceptions were also handled so that they would result in the shutdown message going to the worker threads.

You could also choose to register a signal handler (in your main thread) for other OS signals and raise an appropriate exception (e.g. SystemExit) or take other actions. The important point here is that these would all need to be handled in your main thread and communicated by some sort of convention to the worker thread(s).

In Summary

Dealing with these "asynchronous events" in multi-threaded applications can be a little confusing (and sometimes a little frustrating when your app refuses to exit). Understanding the key points here will hopefully help make this a bit clearer:

1. Signals are handled by the main thread. This means that KeyboardInterrupt is always raised in the main thread.
2. Daemon threads will exit automatically when the main thread exits.
3. For cases where you need more control over thread termination, use threading.Event objects to "signal" threads to exit.
4. Be aware that Thread.join() calls will block and cannot be interrupted! Use an alternative while-loop strategy for joining instead.

STOMP and CoilMQ

I've just pushed out my third release (v0.3) of CoilMQ, a simple STOMP server written in Python.  (You can see the roadmap or the issue tracker to learn about the changes.)  It's been a fun project, so I decided to write about why I've been working on it and why you might want to consider Stomp for message passing.


Project Background: A Twisted Snake-y Road

A few months back I started a simple project to learn more about writing a basic network server in Python.  I decided I would create a STOMP server in Python, because there weren't any -- well, none that I could get to actually work.  And there was this fantastic & simple stompserver Ruby project out there just taunting Python with its simple, concise implementation.

So, I plowed into the project and immediately resolved, as all aspiring Python developers do, that this was perfectly suited to Twisted.  Afterall, the Ruby Stompserver project was built on EventMachine. Nevermind that MorbidQ seemed to have had the same idea (and was firmly on my "not-working" list).  Nevermind that I'd been there before, waist-deep in Twisted entrails trying to grab some slimy Deffered; I made myself write "Twisted is harder than dataReceived" two thousand times.  Nevermind that Ted Dziuba wrote a brilliant tirade on Twisted which anyone that's had to wrestle this Gorgon can appreciate.  Nevermind all that.

Well, that didn't last long.  Sure, I had an original version written using Twisted.  Very simple.  Everything in memory; no blocking I/O.  It looked architecturally like the Ruby EventMachine code.  Then I thought, what about using a database for queue storage?  And that's when the romance ended.  Well, I don't have a database API that returns Deffereds; I just have one that'll stop my app while the query runs.  I know this was really just an educational project & proof of concept, but I didn't feel right doing it wrong.

So, I decided that I'd write a my server using threads for concurrency.  I know it's not what the cool kids are doing, but -- despite the GIL -- it's a great way to handle I/O concurrency in Python and I feel it's a really important thing for developers to understand -- because it comes up all the time.  (I'm no expert, don't get me wrong; this was a learning exercise.)



Why Stomp?

Stomp is a really simple protocol for publisher-subscriber message passing.  It's loosely HTTP-like in its syntax and it works for passing text-based or binary data -- so you can pass whatever data type your consumers care about (JSON, XML, AMF, pickled objects, etc.).  The protocol itself doesn't explicitly prescribe topic or queue implementations, but there is a convention that destinations that begin with "/topic/" are topics (broadcast to all subscribers, no persistence or reliability) and destinations that begin with "/queue/" are queues (sent only to one subscriber, persisted).  Stomp does go beyond the basics to provide support for transactions and reliable subscribers (that must ack receipt).  Some implementations (e.g. ActiveMQ) have also added the concept of durable topics.

There are lots of uses for Stomp because it's been implemented in lots of different languages.  For example, you could use simply use this as a way to pass data (e.g. pickled objects) between disparate web applications written in Python (e.g. using stompy client).  Or you could use Stomp to provide for passing messages between Python application servers and Flash/Flex clients (e.g. using as3-stomp client); that's an easy way to implement server-push for Flex.  To build on that example, you could just as easily push AMF-encoded (binary) messages between different Flash clients.  So Stomp is really quite basic, and very versatile.

Why CoilMQ?

This is a tricker question to answer.  Honestly, if you've got a project out there that needs to push messages at scale, you probably want to look at some of the enterprise brokers.  I've had great luck using ActiveMQ and I've heard people that need to scale to massive throughput suggest using RabbitMQ.  But if you don't need to handle thousands of requests per second, and you do want to understand the software you're using, then I would suggest that a project like CoilMQ is a great choice.

Plus, it'd be great to have some feedback!  I've had a number of downloads & no bug reports (or complaints).  I'd love to think that means it just works perfectly for everyone, but I suspect the truth is that I just need more users.  I should add that I'd also be very pleased to accept any suggestions or patches (the former is greatly helped by the latter).

Notes from PyCon 2010

PyCon just wrapped up; it was geektastic!  This was certainly the largest group of Python developers I'd ever witnessed, and I came away with some startling realizations about Pythonistas.  For example, I learned that, like Samon's strength, a Python developer's social standing is directly proportional to the length of his (or her) locks and beard.  I have stopped shaving and today I had an oprah-ah-ha moment with metaclasses and slots.  If I can make it through this awkward in-between phase, I can only imagine the revelations that lie ahead.

Atlanta was interesting.  I mean we were in a very corporate, rich downtown: lots of big buildings housing banks and lots of people rolling on chromed 20s.  I'd never seen so much chrome.  The juxtaposition of Atlanta with Pycon was probably at its most poignant on Saturday night, when apparently the Hyatt hosts an Old School Saturdays dance party in their downstairs ball rooms.  That happened to be right next to the Python open spaces.  They had some ropes to keep the groups separate.  I'm sure mayhem would have ensued if party goers had wandered into the Unladen Swallow open space.

Anyway, back to the conference.  Some of the videos and slides are being aggregated online.  The talks were great; I'm hoping that selection becomes more comprehensive.

I definitely came away with a bunch of notes, things to try, etc

• Python in the browser (Silverlight) looked cool, but it's not really a viable option for Linux users.  Even the latest preview of Moonlight doesn't run the IronPython REPL without errors.  (Stable version doesn't run it at all.)  Oh well, I'll try back later.
• The GIL on multi-core isn't fixed yet in 3.2 -- and this problem is relevant even when dealing with IO-bound instead of CPU-bound apps.  David Beazley's presentation was fantastic.  Of course, threading still makes sense for handling IO load, but it's good to have this limitation commonly understood.
• Donovan Preston's talk definitely made me want to check out Eventlet.  One of the biggest drawbacks of frameworks like Twisted is that you have to write code in a special way (i.e. using deffereds, or yield) and the implication of this is that you also cannot just take other people's code and include it in your project.  That's a huge problem.  However, even with Eventlet you do have the drawbacks of it being a non-blocking server.  So I was even more excited to learn about Spawning.  From what I understood, Spawning combines the power of the async server with actual processes, that make it easy to offload those blocking IO calls -- i.e. your database queries won't pause your entire application.
• The State of Packaging (slides aren't up at time of writing) was a great look at improvements that are being made to the packaging metadata in Python.  Tarek Ziadé is to be commended for tackling this problem.  And oh what a problem it is!
• The new unittest  has some really useful improvements (like multi-line text diffs for assertEqual!).  And it's been backported as unittest2 for us folks still stuck on 2.6.
• I will use Dozer!  (It's memory profiling WSGI middleware.  How simple!)
• I will use repoze.profile!  (It's a performance profiling middleware.)
• When I think that I have a problem best solved by Nginx, I will also evaluate HAProxy.
• Apparently people love Munin.  They say it's much better than Nagios.

I'm sure there was a lot more.  I got bored of scouring down my notes and looking up links.  It was a fantastic conference, though.  If you didn't go this year, go next year.  I hear it's in Atlanta again; bring a purple suit and they might let you into Old School Saturdays.