Flask-Track-Usage 1.1.0 Released

A few years ago the initial Flask-Track-Usage release was announced via my blog. At the time I thought I’d probably be the one user. I’m glad to say I was wrong! Today I’m happy to announce the release of Flask-Track-Usage 1.1.0 which sports a number enhancements and bug fixes.

Unfortunately, some changes are not backwards compatible. However, I believe the backwards incompatible changes make the overall experience better. If you would like to stick with the previous version of Flask-Track-Usage make sure to version pin in your requirements file/section:

flask_track_usage==1.0.1

Version 1.1.0 has made changes requested by the community as well as a few bug fixes. These include:

  • Addition of the X-Forwarded-For header as xforwardedfor in storage. Requested by jamylak.
  • Configurable GeoIP endpoint support. Requested by jamylak.
  • Migration from pymongo.Connection to pymongo.MongoClient.
  • Better SQLStorage metadata handling. Requested by gouthambs.
  • SQLStorage implementation redesign. Requested and implemented by gouthambs.
  • Updated documentation for 1.1.0.
  • Better unittesting.

I’d like to thank Gouthaman Balaraman who has been a huge help authoring the SQLStorage based on the SQLAlchemy ORM and providing feedback and support on Flask-Track-Usage design.

As always, please report bugs and feature requests on the GitHub Issues Page.

Cloud Message Queues

More and more of my personal work utilizes message queues (MQ) to integrate systems or to propagate longer running work across pools of workers. AMQP is the 300 pound gorilla in the room when it comes to message queuing. Implementations of RabbitMQ, Qpid, Red Hat MRG, etc.. abound. However, when you are the little guy on the field it can be economical to use a cloud service so you can focus directly on your product. Can cloud message queues be a good replacement for running a MQ yourself?

What I Expect

I’m going to make some assumptions that the service will be available, messages will not disappear (unless I set it to do so) and minor network latency is acceptable.

These are features I expect in priority order:

  1. Central connection point (+5)
  2. FIFO support (+4)
  3. Delivery to first available consumer (+3)
  4. Basic publish/subscribe support (+2)
  5. Push message support (+1)

Options

As it turns out there are more players in the cloud messaging space than I would have thought! A quick search turned up the obvious Amazon SQS along with IronMQ, stormmqSoftlayer Message Queue and Marconi.

Amazon SQS

I tend to think Amazon’s SQS is probably the default MQ as a Service. So many people use AWS and it’s right there ready to be used.

  1. Central connection point: Yes (+5)
  2. FIFO support: No
  3. Delivery to first available consumer: Sort of… (+1)
  4. Basic publish/subscribe support: Yes (+2)
  5. Push message support: Sort of… (+0)

There is no doubt that Amazon’s SQS is a great system but right off the bat it’s obvious it doesn’t meet what I expect. According to the FAQ:

Q: Does Amazon SQS provide first-in-first-out (FIFO) access to messages?

No, Amazon SQS does not guarantee FIFO access to messages in Amazon SQS queues, mainly because of the distributed nature of the Amazon SQS. If you require specific message ordering, you should design your application to handle it.

The first consumer who makes an API request will get the next message. That is sort of delivery to the first available consumer but not exactly.

Push messaging is not directly supported but long polling is available. This is close enough that I’d consider it.

Q: What is SQS Long Polling?

SQS long polling is a new way to retrieve messages from your SQS queues. While the traditional SQS short polling returns immediately, even if the queue being polled is empty, SQS long polling doesn’t return a response until a message arrives in the queue, or the long poll times out. SQS long polling makes it easy and inexpensive to retrieve messages from your SQS queue as soon as they are available.

Feature Result: 8

Result: 8/15

IronMQ

Comparing IronMQ to Amazon SQS was interesting. Unlike SQS, IronMQ uses a REST interface which I feel simplifies MQ as a web service. I played a little bit with the service and it was much speedier than I thought it would be! I even tried the beanstalkd support but wasn’t able to get it to fully work.

  1. Central connection point: Yes (+5)
  2. FIFO support: Yes (+4)
  3. Delivery to first available consumer: Sort of… (+1)
  4. Basic publish/subscribe support: Yes (+2)
  5. Push message support: Not really…

Again there is a hiccup on delivery to the first available consumer. Just like SQS, IronMQ is based off requests from the clients. It could match if a client requests only when it can fully dedicate itself to the next message.

Unfortunately, the push support in IronMQ doesn’t cut it for me. If I’m reading the documentation correctly it makes an assumption that the consumers are all listening HTTP servers. I see the use case for this but I also wouldn’t want to make some or all of my consumers publicly listening on the Internet and spinning off work in another thread or process. I’d rather long polling.

Feature Result: 12

Result: 12/15

stormmq

This didn’t get any testing whatsoever. According to the features page it will have a GA of Q1 2013 (which was earlier this year..). I sent a message via Twitter to find out if I’m just seeing old data on their site. For now I will assume that the statement that it’s AMQP 1-0 is true.

  1. Central connection point: Yes (+5)
  2. FIFO support: Yes (+4)
  3. Delivery to first available consumer: Yes (+3)
  4. Basic publish/subscribe support: Yes (+2)
  5. Push message support: Yes (+1)

This was the first service which would meet all of my wants. However, I can’t sign up for it or use it so it kind of knocks it off the list.

Feature Result: 15

Penalty for not being available: -15

Result: 0/15

Softlayer Message Queue

  1. Central connection point: Yes (+5)
  2. FIFO support: No
  3. Delivery to first available consumer: No
  4. Basic publish/subscribe support: Yes (+2)
  5. Push message support: No

Like SQS, Softlayer Message Queue notes that FIFO is not supported:

Does SoftLayer Message Queue provide first-in-first-out (FIFO) message access?

While the system does it’s best to return messages in FIFO order, it is not guaranteed. A number of factors can influence message ordering, including message timeouts and updated visibility intervals.

The FAQ also notes that it is possible to have conditions where a consumer (or consumers?) may get the exact same message multiple times.

How can multiple readers access the same message queue, without losing messages or processing them many times?

Queues can be accessed by any number of consumers/clients. When a consumer requests messages from the queue, each message is marked invisible—this prevents other consumers from receiving the same message. However, the distributed nature of the message queue cannot guarantee single delivery of any one message. While the system makes a best effort, clients should expect the possibility of receiving the same message multiple times.

Since the MQ service is web service (REST) based there is no pushing of messages, only pulling. I didn’t see anything noting long polling or even push to HTTP servers like IronMQ.

Feature Result: 7

Result: 7/15

Marconi

  1. Central connection point: Yes (+5)
  2. FIFO support: Yes (+4)
  3. Delivery to first available consumer: Sort of.. (+1)
  4. Basic publish/subscribe support: Yes (+2)
  5. Push message support: In progress…

Since Marconi is primarily web service based (REST) it has the same issues as Amazon SQS and IronMQ.

Push messaging is not currently available from what I can tell but there is ZMQ bindings in the work with AMQP work coming later.

The negative part to using Marconi is that one either needs to be using an OpenStack based service already or they will need to set up the service themselves. This does add some overhead to using it. Also I keep wanting to say macaroni.

Feature result: 12

Bonus for being Open Source: +2

Penalty for needing specific software: -5

Result: 9/15

Conclusion

So can cloud message queuing be a good replacement for running your own service? It highly depends on what you are doing, but if you don’t need all the features of modern MQ systems then you can run with a cloud MQ and cut some time/cost.

It may seem unfair, but I am wary of using stormmq because of the old GA info on the site even if they went GA today. At a later date after GA I’d consider trying them but I’d want to see a bit better messaging (no pun intended) via their site.

For me it makes the most sense to use IronMQ until I end up on an OpenStack based system at which point, if ZMQ or AMQP is available, I’d likely switch over. Having ZMQ/AMQP’s push ability would be worth the move.

Python IDE Woes

I love  the Python programming language. I could spend hours explaining why it’s generally my go to language when coding something new but that’s not what I really want to touch on today. Today it’s about IDEs and editors.

IDE or editor selection is almost a cultish exercise. Developers break out into small societies around coding tools and banish those exhibiting wandering eyes towards  tool lust. In some cases I’ve fallen into those patterns when I find a tool that I enjoy. It can be frustrating to have to uproot your development workflow to accommodate a new tool even when it’s obvious that after the initial learning curve the tool will make life easier. With all that said I am not stuck to any IDE or editor. I’m an IDE/editor swinger.

There are plenty of choices for Python, many of which I’ve used throughout the years. My problem is that all of those I’ve tried I always end up moving away from to one of the default editors: vim or emacs (in my case vim). For the heck of it this post is to go through the main reasons for the most recent movements back to vim.

IDEs and Editors

Eclipse

I used to joke that every 6 months I’d give Eclipse a try. It’s very popular in the Java programming world and has a great Python plugin called PyDev. However, Eclipse itself always feels sluggish even when I’m writing Java. Keep in mind I’m not on some old hardware which is limiting the IDE nor am I doing some corner case kind of usage. The sluggishness is not terrible either, but it’s noticeable reminding me that I’m using a very large piece of software. One of my thoughts is that if I notice the IDE/editor for anything other than a helpful feature then there is likely a problem.

NetBeans

NetBeans is not all that different than Eclipse in my mind. It’s a Java IDE which is extendable to other languages. Last time I tried NetBeans it was less sluggish than Eclipse. My main issue with NetBeans is I can never tell what is going one with the IDE. There was a pretty big push for first class Python support in NetBeans. I played with that IDE version for a while and, overall, liked it. Now searching python on the NetBeans site returns nothing. It seems like there are random community efforts to bring Python back into NetBeans but I want something that works well right now (no offense to any of the efforts!).

PyCharm

Another similar IDE is PyCharm. I have at least one fellow Python developer which swears by PyCharm and I can see why. It doesn’t seem sluggish even though it’s pretty large. It has a good feature set. But it has confusing proprietary licensing and the IDE is specific to the language. If you need to get some Ruby coding done then it’s a different JetBrains IDE and, I assume, another purchase and license agreement. That’s a bit frustrating!

Komodo IDE

Komodo IDE was one of the earlier Python IDEs I tried.  I have to admit not trying it in a while, but last time I played with it I felt it was sluggish in a similar way to Eclipse. Like Eclipse it has a lot of plugins/add-ons available which is great but, then again, it’s proprietary and costs $382. Sort of a long term deal breaker right there.

Komodo Edit

The younger brother of Komodo IDE, Komodo Edit actually feels more useable to me. I believe it’s just the basic core of Komodo IDE, but that works in it’s favor. It seemed faster and kept out of my way. The initial start screen always feels clunky. However, the license is weird and is not listed on the FSF or OSI list. I guess it’s proprietary? Confusion is not a good thing.

Sublime Text

This one kills me. Sublime Text is a fantastic editor. I really like it! The editor is very fast and has some unique features. It’s available on all three major computer platforms (Linux, Windows and OS X). The license is proprietary but is simple, understandable. I believe this may be a first. Though the fact it’s proprietary makes it a harder editor to make my default. If Sublime Text was Free Software or Open Source it would be my programming default editor.

Scribes

This was a nice editor. I say was for Scribes because it looks like it is no longer actively developed. It’s a shame because it was a powerful and fast editor with an Open license. Even though it’s an older editor than Sublime Text I liked it for a lot of the same reasons. If this was still being worked on it would be my default programming editor.

PIDA

PIDA was my default IDE for a while. If I was using an IDE it was likely to be PIDA. It was Open, had good plugins, fast, embedded Vim as the editor, said it loved me, etc.. However, the PIDA web site has disappeared and the last stable release was about 3 years ago. It seems like a8 may be the replacement but I’ve not had time to run with it yet.

Others To Look At

As I stated before I’d like to spend some time with a8. It looks like it’s X based only (Linux, BSD, etc..) which is fine for me since most all of my dev is on Linux. I’ve also seen a lot of talk about Ninja IDE and it looks promising though I’m generally not a fan of specific language IDEs since I am not always using the same language.

What Do You Use?

Seriously. If you are doing Python development regularly what are you using and why? What features have you found to be amazing and which ones are overrated marketing dribble? Am I the only guy who continues to go back to vim in this day and age?

Why I Chose NewsBlur

Not all that long ago Google Reader closed it doors pushing millions of users off the platform. Many users were frustrated to lose their long time place to get their news not all that different from someone in yesteryear losing their favorite newspaper.  The whole thing was far from ideal but did go to teach users that you can’t expect cloud services to last forever (which is a good wake up call). But in the fall of Google Reader came many possible replacements which added their own spins on how one reads news. Feedly, The Old Reader and NetVibes were a few of the popular replacements. But I settled on NewsBlur and eventually became a paid user.

NewsBlur is mainly written by Samuel Clay (more on why I say mainly later).  He seems like a friendly, hard working fellow. He responds to bug reports and is active in his products community.  While this may seem like common sense just take a few minutes to look at random SaaS products on the Internet. You’ll find many of the developers are hidden behind customer service groups who, at worst, are outsourced and are more of a dead end than a way to get things fixed. Long story short, it seems like Samuel really cares about his product.

It is possible to have a Free account on NewsBlur. While you are limited to a specific amount of feeds many people will find the limits are higher than the feed counts they had in Google Reader. At the time of writing the limit is 64 sites.

There are some social features provided by NewsBlur yet these features are not required nor forced into general workflow. For instance, there is a concept of the BlurBlog which looks like it could be fun. But I tend to read the news and share elsewhere. If I ever decide to use the BlurBlog functionality it’s there. Otherwise I can just use NewsBlur as a fantastic reader.

NewsBlur is Open Source under the MIT license (also known as the Expat License). This gives me peace of mind knowing if Samuel ever decided that he was done with NewsBlur I could export my feeds, setup my own instance, and continue using the product on my own infrastructure. Yeah, it’s not trivial but it’s possible which is a huge advantage given the last reader I used shut down.

No software is without it’s bugs but Samuel does a good job bug squashing. And if you are developer who wants to give a hand you can patch the issue yourself and submit the fix (another win for Open Source). At the time of writing there are 43 development contributors to NewsBlur. This is a much better solution than waiting for a customer service representative to reinterpret your bug submission to a developer so that the fix may be done someday in the future.

If you are still looking for a replacement for Google Reader give NewsBlur a chance even if it’s a second chance as the application seems to be enhanced weekly. If you like it, consider becoming a paid user. Can you can’t say no to Shiloh:

Introducing Flask-Track-Usage

A little while ago one of the guys on a project I work one was asking about how many people were using the projects public web service. My first thought was to go grepping through logs. After all, the requests are right there and pretty consumable with a bit of Unix command line magic. But after a little discussion it became clear that would get old after a while. What about a week from now? How about a month or year? Few people want to go run commands and then manually correlate them. This lead to us looking around for some common solutions. The most obvious one was Google Analytics. To be honest I don’t much care about those systems. While that one may not (or may be) intrusive on users I just don’t feel all that comfortable forcing people to be subjected to a third party of a third party unless there is no other good choice. Luckily, being that the metrics are service related, the javascript/cookie/pixel based transaction wouldn’t have worked very well anyway.

So it was off to look at what others have made with a heavy eye towards Flask based solutions so it matched the same framework we were already using. Flask-Analytics came up in a search. The simple design was something I liked but the extension was more so aimed at using cookies to track users through an application while we want to track overall usage. I figured it was time to roll something ourselves and provide it back out to the community if they could use it as well.

Here it is in all it’s simplistic glory: Flask-Track-Usage. It doesn’t use cookies nor javascript and can store the results into any system which you provide a callable or Storage object. There is also FreeGeoIP integration for those what want to track where users are coming from. The code comes with a MongoDB Storage object for those who want to store the content back into their MongoDB. Want to know a bit more of the technical details? Check out the README or the project page. Patches welcome!

Simplicity Over Beauty. Functionality Over Features.

I’m a fan of many of the services that allow the legal purchase of music online. One service that I’ve come to enjoy over the years is emusic which, for quite some time, didn’t provide a way to download music other than through their website and desktop downloader.

As seems common, a developer decided to scratch his own itch and release a mobile downloader application. If it wasn’t for his application I could have easily moved over to Amazon MP3 or Google Music just based on the ease of purchase and growing selections. Fast forward a year and emusic figured out people wanted a mobile downloader. A beta was released.

And it was buggy and had features that, as a user, I really don’t care about while having oddly implemented features that seemed core to the downloader experience. Let me explain: I wanted to search, purchase and download music. I think that those three actions are nearly universal for a music purchase and download app. Here is the list of features for the official emusic application:

  • Listen to any MP3 files stored on your phone through the music player
  • Get song recommendations based on your listening habits
  • Create playlists tailored to your every mood
  • Browse an artist’s dossier (discographies, biographies, photos, features, and related artists) while you listen to music
  • Access eMusic’s unique charts
  • Search eMusic’s catalog and listen to track samples
  • Download music
  • Listen to eMusic Radio (available to U.S. members only)
  • Save for Later
  • Access account details

(Source)

This is how I ended up feeling about the emusic application: it searched, purchased (or crashed), downloaded in a weird filename format (or crashed), wanted to search my existing library to give recommendations (sort of weird) and listen to samples (or crash). When the application crashed there was a possibility that emusic would think I downloaded the song when I didn’t receive it meaning I would need to contact to support and ask for a redownload.

The ‘weird format’ may sound, well, weird but it’s true. I’m someone who likes to listen to music in many settings. The two most common ways I listen to music are from a laptop or via my phone. Obviously this means I’m at least using two different music players (right now it’s Nightingale and Player Pro) and the need to organize music is important. When I download via the emusic website or via the indie developers app I get a nice directory structure with artist and songs named in a way that makes sense to me. When I download from the emusic mobile application, well, see for yourself:

In fairness many of the crashing bugs have been fixed and it’s a better client than it was but I still buy and download using the third party app developers application. It’s simpler, stable and let’s me do the three tasks I need out of a downloader.

So why didn’t emusic decide to license/buy the downloader that the developer released? I’m not totally sure but I have a an inkling that they wanted to show they could outdo the simple app. Pretty interface, more features, samples, etc. As a user I care less about how it looks and much more about how functional the application is.

Today I still use the indie developers emusic downloader instead of the official app and I recommend it’s use over emusics official app. I want an application that let’s me get the music I want and delivers a consistent user experience (read: simple flow and doesn’t crash). If it’s not pretty or doesn’t have every single cool feature under the sun, well, I’m ok with that.