In My Mind It’s Time For Go

I have been, and continue to be, a fan of the Python programming language. It’s clean looking, portable, quick to write, has tons of libraries and a great community. However, I, like many other software developers, don’t think that one must be married to a language. Go has received a lot of attention as of late and it finally makes sense to me as to why.

I’ve gone through, and added to my tool belt, many languages before Python becoming my go to star. The first was Perl. Being so versatile and having a large community of both professional and hobby developers made it an easy early choice for me. The biggest issue for me with Perl was I started to learn how much I liked simple, easy to read code which follows a coding standard. PHP became my fall back web language. It was so simple to write an application it was almost dumb not to use it at the time. No language is perfect and I found many developers at the time didn’t know how to write safe PHP or easy to follow code. I slowly drifted away from PHP as a mainstay. C, while fast and being, well, C, never became a go to mainly because of the time to get things done.

Java just never really did it for me. University tried to shove it down my throat and it didn’t work. Funny thing is I really tried to like it. I gave it multiple chances but would always walk away feeling better with my hands untied from the JVM. I also found myself loathing IDE’s due to people’s insistence that a Java developer needs to use one to be productive. Like many cool kids I became a Java hater for a while and did everything I could to keep friends away from it. But enough of this, let’s talk about Go.

Go has been on my radar since its initial announcement. If I remember right my first thoughts were not positive ones. Everyone and their brother seemed to have their own languages coming out or, at the very least, a DSL which would be the next big thing. I decided to stay back for a bit and see what would happen with the language rather than diving in or slamming the door. Since then there has been a good amount of libraries created for the language, some pretty interesting users such as Docker and DropBox and this post which sums up why Go is a good option when considering Node.js.

What I’ve found is that no tutorial, video nor explanation from a Go fan could convince me to actually see why Go could be so great. After all, the idea behind Go’s OO support sounds half hearted. So one night I did a Google search for what features make people fawn over Go and channels and goroutines came up quite a lot. The next step was to write a Hello World like application that would utilize both features and see how I felt. This is what I came up with:

package main

import "fmt"


// Coms turns into a communication instance. in/out are channels.
type Coms struct {
    in    chan string
    out   chan string
}

// Closes both in and out channels in a Coms instance.
func (c Coms) Close() error {
    fmt.Println("Closing channels")
    close(c.in)
    close(c.out)
    return nil
}
// -------------------


// Main function which sends ping and pong back in two goroutines
// using a Coms instance.
func main() {
    // Creating an "object"
    coms := Coms{make(chan string), make(chan string)}
    // At the end of the function run coms.Close.
    defer coms.Close()

    // First goroutine which ping's out and responds
    // back to pong's with a ping.
    go func() {
        coms.in <- "ping"
        for {
            data := <- coms.out
            fmt.Println(data)
            coms.in <- "ping"
        }
    }()

    // Second goroutine which pong's in response to ping's.
    go func() {
        for {
            data := <- coms.in
            fmt.Println(data)
            coms.out <- "pong"
        }
    }()

    // To exit the application hit enter
    var input string
    fmt.Scanln(&input)
}

 

The result of this code was:

$ go run pingpong.go
ping
pong
ping
pong
ping
....
⏎
Closing channels
$

 

It’s a simple program but it pulled me in. The OO style was not nearly as clunky as I thought it would be and the goroutines were so simple to use I was almost shocked. I also got first hand feel for a non-intrusive strongly typed system. It felt almost like a dynamic language. For things that need speed I felt hooked!

Will I continue to use Python? Most surely! But for things that need to be super fast I think Go will be my default. If someone told me to greenfield a SaaS service tier I’d probably lobby for using Go while keeping the web tier Python or something(s) similar. uWSGI server anyone?.

If you are still on the fence with go take a quick look at it’s feature set and take the 20 minutes to write a simple application utilizing them. Examples are nice but there is nothing better than trying the syntax and structures yourself. Writing some code should tell you if Go is for you.

Advertisement

Red Hat Developer Blog: Git Bonsai, or Keeping Your Branches Well Pruned

Code repositories are the final resting place for code, acting as equal parts bank vault, museum, and graveyard. Unlike a vault, content is almost always added faster than it is removed, but much like a graveyard, there is a definite miasma when a repository becomes too full. That smell is developer frustration, as they have to search through dozens, or eventually, hundreds of branches to find the one they want.

We’ve had sporadic cases where branches did not get merged into masters (and sometimes fixes were overwritten in later releases) and have wasted collectively hundreds of developer hours on “which branch is this in?” exchanges.

Sam and I talk about a simple yet helpful git tool to squash bad branches over at the Red Hat Developer Blog.

Slack Isn’t New, It’s New

New tech tools show up daily. You’ve probably heard about Slack already as it’s being talked about a lot, but just in case you haven’t I’ll give you the tl;dr: Slack integrates your development and infrastructure notifications, chat and documents from different providers into one chat like interface. It’s pretty much the same as Hipchat. All in all slack is a pretty cool and sleek system which allows for easy chatting within a group. But as I tested the service I kept having a feeling of deja vu. This cool new service feels familiar to me. Then it hit me, I have seen this before, haven’t I?

For a long time many engineers, especially in the Free/open arena, have utilized IRC as a way of communication while working. It’s efficient, simple, client agnostic and supports chat rooms as well as private messages. Slack’s main chat interface is very similar to IRC. It’s a chat room with a list of people present and the ability to send private messages. Just like IRC people post messages to the chat room and everyone in the room is able to read them. In a way it’s a little funny to think about chat rooms being “new” but, then again, people have been using instant messaging and SMS style message systems for so long that the concept of the chat room may seem fresh. So the chat interface is similar to IRC, but what about the integration? Aren’t they new?

With Slack the integrations are set up via the web interface. Each integration can send information to a channel with an icon and message. Obviously IRC does not have this functionality directly, but IRC bots do! Many developers set up bots like Supybot with integration with their external development tools. Announcements of new builds, code pushes, deployments, support requests, etc.. show up in channel from the bot. While it’s not as flashy as Slack the same basic integration idea occurs.

Don’t get me wrong, the point of this post isn’t to say that Slack is dumb or simply a copy of something “better”. The point of this post is that, while Slack isn’t really something brand new, it is quite cool. There is a reason developers and ops folks have been setting up things similar to Slack in their own chat rooms for years! The ability to see the development process actually flow can be pretty exciting and empowering. Those who have or will not be able to set up their own integrations have an option to use Slack as a pre-baked set up which, depending on team/company may be more user friendly for the less technical minded. And letting the non-technical see how much is happening day to day can open their eyes to just how much a team is getting done. Probably way more than they realize.

Cloud Message Queues

More and more of my personal work utilizes message queues (MQ) to integrate systems or to propagate longer running work across pools of workers. AMQP is the 300 pound gorilla in the room when it comes to message queuing. Implementations of RabbitMQ, Qpid, Red Hat MRG, etc.. abound. However, when you are the little guy on the field it can be economical to use a cloud service so you can focus directly on your product. Can cloud message queues be a good replacement for running a MQ yourself?

What I Expect

I’m going to make some assumptions that the service will be available, messages will not disappear (unless I set it to do so) and minor network latency is acceptable.

These are features I expect in priority order:

  1. Central connection point (+5)
  2. FIFO support (+4)
  3. Delivery to first available consumer (+3)
  4. Basic publish/subscribe support (+2)
  5. Push message support (+1)

Options

As it turns out there are more players in the cloud messaging space than I would have thought! A quick search turned up the obvious Amazon SQS along with IronMQ, stormmqSoftlayer Message Queue and Marconi.

Amazon SQS

I tend to think Amazon’s SQS is probably the default MQ as a Service. So many people use AWS and it’s right there ready to be used.

  1. Central connection point: Yes (+5)
  2. FIFO support: No
  3. Delivery to first available consumer: Sort of… (+1)
  4. Basic publish/subscribe support: Yes (+2)
  5. Push message support: Sort of… (+0)

There is no doubt that Amazon’s SQS is a great system but right off the bat it’s obvious it doesn’t meet what I expect. According to the FAQ:

Q: Does Amazon SQS provide first-in-first-out (FIFO) access to messages?

No, Amazon SQS does not guarantee FIFO access to messages in Amazon SQS queues, mainly because of the distributed nature of the Amazon SQS. If you require specific message ordering, you should design your application to handle it.

The first consumer who makes an API request will get the next message. That is sort of delivery to the first available consumer but not exactly.

Push messaging is not directly supported but long polling is available. This is close enough that I’d consider it.

Q: What is SQS Long Polling?

SQS long polling is a new way to retrieve messages from your SQS queues. While the traditional SQS short polling returns immediately, even if the queue being polled is empty, SQS long polling doesn’t return a response until a message arrives in the queue, or the long poll times out. SQS long polling makes it easy and inexpensive to retrieve messages from your SQS queue as soon as they are available.

Feature Result: 8

Result: 8/15

IronMQ

Comparing IronMQ to Amazon SQS was interesting. Unlike SQS, IronMQ uses a REST interface which I feel simplifies MQ as a web service. I played a little bit with the service and it was much speedier than I thought it would be! I even tried the beanstalkd support but wasn’t able to get it to fully work.

  1. Central connection point: Yes (+5)
  2. FIFO support: Yes (+4)
  3. Delivery to first available consumer: Sort of… (+1)
  4. Basic publish/subscribe support: Yes (+2)
  5. Push message support: Not really…

Again there is a hiccup on delivery to the first available consumer. Just like SQS, IronMQ is based off requests from the clients. It could match if a client requests only when it can fully dedicate itself to the next message.

Unfortunately, the push support in IronMQ doesn’t cut it for me. If I’m reading the documentation correctly it makes an assumption that the consumers are all listening HTTP servers. I see the use case for this but I also wouldn’t want to make some or all of my consumers publicly listening on the Internet and spinning off work in another thread or process. I’d rather long polling.

Feature Result: 12

Result: 12/15

stormmq

This didn’t get any testing whatsoever. According to the features page it will have a GA of Q1 2013 (which was earlier this year..). I sent a message via Twitter to find out if I’m just seeing old data on their site. For now I will assume that the statement that it’s AMQP 1-0 is true.

  1. Central connection point: Yes (+5)
  2. FIFO support: Yes (+4)
  3. Delivery to first available consumer: Yes (+3)
  4. Basic publish/subscribe support: Yes (+2)
  5. Push message support: Yes (+1)

This was the first service which would meet all of my wants. However, I can’t sign up for it or use it so it kind of knocks it off the list.

Feature Result: 15

Penalty for not being available: -15

Result: 0/15

Softlayer Message Queue

  1. Central connection point: Yes (+5)
  2. FIFO support: No
  3. Delivery to first available consumer: No
  4. Basic publish/subscribe support: Yes (+2)
  5. Push message support: No

Like SQS, Softlayer Message Queue notes that FIFO is not supported:

Does SoftLayer Message Queue provide first-in-first-out (FIFO) message access?

While the system does it’s best to return messages in FIFO order, it is not guaranteed. A number of factors can influence message ordering, including message timeouts and updated visibility intervals.

The FAQ also notes that it is possible to have conditions where a consumer (or consumers?) may get the exact same message multiple times.

How can multiple readers access the same message queue, without losing messages or processing them many times?

Queues can be accessed by any number of consumers/clients. When a consumer requests messages from the queue, each message is marked invisible—this prevents other consumers from receiving the same message. However, the distributed nature of the message queue cannot guarantee single delivery of any one message. While the system makes a best effort, clients should expect the possibility of receiving the same message multiple times.

Since the MQ service is web service (REST) based there is no pushing of messages, only pulling. I didn’t see anything noting long polling or even push to HTTP servers like IronMQ.

Feature Result: 7

Result: 7/15

Marconi

  1. Central connection point: Yes (+5)
  2. FIFO support: Yes (+4)
  3. Delivery to first available consumer: Sort of.. (+1)
  4. Basic publish/subscribe support: Yes (+2)
  5. Push message support: In progress…

Since Marconi is primarily web service based (REST) it has the same issues as Amazon SQS and IronMQ.

Push messaging is not currently available from what I can tell but there is ZMQ bindings in the work with AMQP work coming later.

The negative part to using Marconi is that one either needs to be using an OpenStack based service already or they will need to set up the service themselves. This does add some overhead to using it. Also I keep wanting to say macaroni.

Feature result: 12

Bonus for being Open Source: +2

Penalty for needing specific software: -5

Result: 9/15

Conclusion

So can cloud message queuing be a good replacement for running your own service? It highly depends on what you are doing, but if you don’t need all the features of modern MQ systems then you can run with a cloud MQ and cut some time/cost.

It may seem unfair, but I am wary of using stormmq because of the old GA info on the site even if they went GA today. At a later date after GA I’d consider trying them but I’d want to see a bit better messaging (no pun intended) via their site.

For me it makes the most sense to use IronMQ until I end up on an OpenStack based system at which point, if ZMQ or AMQP is available, I’d likely switch over. Having ZMQ/AMQP’s push ability would be worth the move.

Red Hat Developer Blog: Feeling Developer Pain

The rest of this post describes our journey from initially trying to implement a simple solution to improve the day-to-day lives of developers, through the technical limitations we experienced along the way, and finally arrives at the empathy for our developers we’ve gained from that experience. We’ll wrap up with a note on how Red Hat Software Collections (announced as GA in September) would’ve simplified our development process.

Read the whole post Tim and I wrote over at the Red Hat Develope Blog.

More Editors/IDE’s

In my last post I talked about not being able to find a good Python editor/IDE other than vim. Nothing has really changed since then but there was another editor and IDE that was brought to my attention which I failed to point out. Let’s talk about them!

Editors/IDE’s

Emacs

While talking with Tim Bielawa I was reminded about Emacs since it’s what Tim uses. Emacs has such amazing integration that people sometimes say it’s an operating system itself! I’ve really only ever given Emacs two real shots at being my main editor. The first time was when I was just starting to get into programming and was reading a lot about what other programmers and system administrators used. At first it seemed a lot easier than vi but I ended up running with vi/vim since, at the very least, vi seemed to always be present on every Linux/BSD box I worked with by default. The second attempt was after I promised a few Emacs users at work that I’d give it another fair shot. I said I would use Emacs when I’d normally use vim for a full week knowing that it would force me to learn more about the editor. It wasn’t easy to stop trying to use mode editing but I was able to code without feeling too contained by the editor (Note: the contained feeling was me not knowing the editor well, not Emacs itself. It was really obvious that Emacs is a powerful editor). My biggest gripes from the week long test were around needing to install emacs on systems for use (which is sort of a silly one, I admit) and I felt some of the commands were way to long. I don’t remember the last one off the top of my head but I do remember that one command had a bunch of dashes and was frustrating every time I needed to use it. I think it’s about time I try it again and see if I can overcome my hurdles trying to use it efficiently. Who knows, maybe third time’s the charm!

Cloud9

Cloud9 is really cool. It’s not your grandfather’s IDE by any stretch. As the name implies the application exists out in the cloud (They use Openshift). I love the idea that I can start editing code with full IDE features from any machine I’m currently occupying. I have not tried using Cloud9 with a tablet (with keyboard of course!) yet but if that works then this thing would have rockstar status in my mind. I’ve used it for a few projects for both ease of use and to test out some of the features Cloud9 boasts. Being that it is on Openshift the IDE has it’s own platform letting you install tools and dependencies. There are also some collaboration features which I have not tried out yet mainly because I’m not sure how that works when you are using Github (IE: can they push code under your name or are they restricted?). I would use Cloud9 a lot more if it wasn’t for the Internet. While Cloud9 is pretty responsive in most cases but due to some point in the network connection between “here” and “there” things slow down or stop responding for a second or two. If this was something other than coding I probably wouldn’t care that much … but this is coding. Any hiccups while editing breaks concentration and slows down progress. One other issue I noticed was the lack of preferences across ones instances. Say you have two projects in Cloud9. Each project has it’s own IDE instance. If you want to set preferences for both IDE instances you will have to open each on it’s own and set them. You can not set any kind of global default preferences for all your instances. Hopefully they will add that functionality as I’m pretty positive I’m not the only person who finds that a bit weird. Over time Cloud9 and similar IDE’s will find ways to speed up and add better preference support but until then, in my mind, Cloud9 is straddling the line between full contender and really cool tech preview to keep my eye on.

Python IDE Woes

I love  the Python programming language. I could spend hours explaining why it’s generally my go to language when coding something new but that’s not what I really want to touch on today. Today it’s about IDEs and editors.

IDE or editor selection is almost a cultish exercise. Developers break out into small societies around coding tools and banish those exhibiting wandering eyes towards  tool lust. In some cases I’ve fallen into those patterns when I find a tool that I enjoy. It can be frustrating to have to uproot your development workflow to accommodate a new tool even when it’s obvious that after the initial learning curve the tool will make life easier. With all that said I am not stuck to any IDE or editor. I’m an IDE/editor swinger.

There are plenty of choices for Python, many of which I’ve used throughout the years. My problem is that all of those I’ve tried I always end up moving away from to one of the default editors: vim or emacs (in my case vim). For the heck of it this post is to go through the main reasons for the most recent movements back to vim.

IDEs and Editors

Eclipse

I used to joke that every 6 months I’d give Eclipse a try. It’s very popular in the Java programming world and has a great Python plugin called PyDev. However, Eclipse itself always feels sluggish even when I’m writing Java. Keep in mind I’m not on some old hardware which is limiting the IDE nor am I doing some corner case kind of usage. The sluggishness is not terrible either, but it’s noticeable reminding me that I’m using a very large piece of software. One of my thoughts is that if I notice the IDE/editor for anything other than a helpful feature then there is likely a problem.

NetBeans

NetBeans is not all that different than Eclipse in my mind. It’s a Java IDE which is extendable to other languages. Last time I tried NetBeans it was less sluggish than Eclipse. My main issue with NetBeans is I can never tell what is going one with the IDE. There was a pretty big push for first class Python support in NetBeans. I played with that IDE version for a while and, overall, liked it. Now searching python on the NetBeans site returns nothing. It seems like there are random community efforts to bring Python back into NetBeans but I want something that works well right now (no offense to any of the efforts!).

PyCharm

Another similar IDE is PyCharm. I have at least one fellow Python developer which swears by PyCharm and I can see why. It doesn’t seem sluggish even though it’s pretty large. It has a good feature set. But it has confusing proprietary licensing and the IDE is specific to the language. If you need to get some Ruby coding done then it’s a different JetBrains IDE and, I assume, another purchase and license agreement. That’s a bit frustrating!

Komodo IDE

Komodo IDE was one of the earlier Python IDEs I tried.  I have to admit not trying it in a while, but last time I played with it I felt it was sluggish in a similar way to Eclipse. Like Eclipse it has a lot of plugins/add-ons available which is great but, then again, it’s proprietary and costs $382. Sort of a long term deal breaker right there.

Komodo Edit

The younger brother of Komodo IDE, Komodo Edit actually feels more useable to me. I believe it’s just the basic core of Komodo IDE, but that works in it’s favor. It seemed faster and kept out of my way. The initial start screen always feels clunky. However, the license is weird and is not listed on the FSF or OSI list. I guess it’s proprietary? Confusion is not a good thing.

Sublime Text

This one kills me. Sublime Text is a fantastic editor. I really like it! The editor is very fast and has some unique features. It’s available on all three major computer platforms (Linux, Windows and OS X). The license is proprietary but is simple, understandable. I believe this may be a first. Though the fact it’s proprietary makes it a harder editor to make my default. If Sublime Text was Free Software or Open Source it would be my programming default editor.

Scribes

This was a nice editor. I say was for Scribes because it looks like it is no longer actively developed. It’s a shame because it was a powerful and fast editor with an Open license. Even though it’s an older editor than Sublime Text I liked it for a lot of the same reasons. If this was still being worked on it would be my default programming editor.

PIDA

PIDA was my default IDE for a while. If I was using an IDE it was likely to be PIDA. It was Open, had good plugins, fast, embedded Vim as the editor, said it loved me, etc.. However, the PIDA web site has disappeared and the last stable release was about 3 years ago. It seems like a8 may be the replacement but I’ve not had time to run with it yet.

Others To Look At

As I stated before I’d like to spend some time with a8. It looks like it’s X based only (Linux, BSD, etc..) which is fine for me since most all of my dev is on Linux. I’ve also seen a lot of talk about Ninja IDE and it looks promising though I’m generally not a fan of specific language IDEs since I am not always using the same language.

What Do You Use?

Seriously. If you are doing Python development regularly what are you using and why? What features have you found to be amazing and which ones are overrated marketing dribble? Am I the only guy who continues to go back to vim in this day and age?

Introducing Flask-Track-Usage

A little while ago one of the guys on a project I work one was asking about how many people were using the projects public web service. My first thought was to go grepping through logs. After all, the requests are right there and pretty consumable with a bit of Unix command line magic. But after a little discussion it became clear that would get old after a while. What about a week from now? How about a month or year? Few people want to go run commands and then manually correlate them. This lead to us looking around for some common solutions. The most obvious one was Google Analytics. To be honest I don’t much care about those systems. While that one may not (or may be) intrusive on users I just don’t feel all that comfortable forcing people to be subjected to a third party of a third party unless there is no other good choice. Luckily, being that the metrics are service related, the javascript/cookie/pixel based transaction wouldn’t have worked very well anyway.

So it was off to look at what others have made with a heavy eye towards Flask based solutions so it matched the same framework we were already using. Flask-Analytics came up in a search. The simple design was something I liked but the extension was more so aimed at using cookies to track users through an application while we want to track overall usage. I figured it was time to roll something ourselves and provide it back out to the community if they could use it as well.

Here it is in all it’s simplistic glory: Flask-Track-Usage. It doesn’t use cookies nor javascript and can store the results into any system which you provide a callable or Storage object. There is also FreeGeoIP integration for those what want to track where users are coming from. The code comes with a MongoDB Storage object for those who want to store the content back into their MongoDB. Want to know a bit more of the technical details? Check out the README or the project page. Patches welcome!

Raspberry Pi and Arduino: Good Friends

I have a Raspberry Pi and it’s pretty great. I have an Arduino Esplora and an Arduino Micro and they are fun. Though, as I’ve played more with the Arduino I’ve found one totally understandable drawback: it’s more or less local only. I mean that the data that comes back through sensors or the items being controlled only send response back over serial or USB/serial. It makes sense but it also limits what can be done with it when used all by itself. One of my early ideas for a project was to use a light and a temperature sensor to keep an eye on aging homebrew beer. Nothing super fancy, just records the information for viewing and alerting when the sensors see data outside of the accepted norms. I could do this with some LED’s, buzzer or a display that would notify me when things were off but that isn’t really the type of alerting I’d like to see. That type of alerting would require me to go look at the box for information. I might as well do the sensor gathering manually with my eyes and feeling inside the box. It also means the data would be lost on every iteration. Data from 10 minutes ago could only be gathered if I was present 10 minutes back. Enter Raspberry Pi.

The Raspberry Pi is able to power the Arduino Esplora and, likely, the Arduino Micro. Since communication is over USB/serial the Raspberry Pi can collect the data from the sensors and provide a networked view into the data. For instance, a web interface showing temperature and light graphs. And, best yet, it’s simple to add a USB wireless adapter to the Pi to avoid running an ethernet cable back to the network.

Now, from what I read, it’s possible to use Raspberry Pi itself without an Arduino to collect data and control devices but it requires an ADC for analog input/output. But there is something that seems more proper about separating the physical logic (using C in Arduino) from the notification and reporting logic (using Python on a Raspberry Pi). It feels almost MVC like.

In any case, if you are looking at doing some analog and digital stuff with a Raspberry Pi do know that adding a small Arduino makes life easy and, if you decide to change to another device for providing network views it should be a simple switch over.

“Security? That’s the OS’s/Networks Job!”

I spend a good amount of my time doing software development. I’m one of those guys that has a bad habit of starting projects, getting half or three fourths of the way through and then coming up with another project to do (leaving the original out on in the cold). Needless to say I end up playing with a lot of tools and libraries to help with projects but I’ve started to notice a pattern. The assumption that behind the firewall everyone is friends.

In a more recent project I was working on it became apparent that a queuing system of some kind was going to be needed. Instead of running out and picking the most popular flavor of the month I figured the best move would be to give a few different systems used for queuing a run and see how they worked out. In general I was impressed with their abilities but found the security lacking greatly in a number of them.

Applications

Please be aware I’m not trying to discount any of these applications.The two I tried directly I really liked from a development point of view.

Redis

One of the earlier ones I checked out was Redis. It was blazing fast but the security model is interesting.

Redis is designed to be accessed by trusted clients inside trusted environments. This means that usually it is not a good idea to expose the Redis instance directly to the internet or, in general, to an environment where untrusted clients can directly access the Redis TCP port or UNIX socket.

(Source)

To make matters even more interesting it has support for a single password passed plainly over the wire. Granted, it’s possible to use an SSL proxy as the guide points out but with one user non-repudiation could be a serious problem (especially if logs go back to NAT’d addresses). In effect the security model of Redis seems to require a single tenant, well logged (at network, host and app level) and heavily ACL’d environment. With cloud hosting I’m not so sure how well one could ensure this is the case at all times. Granted, if it’s a single developer running his own infrastructure or a very small company/group/team then it could be possible that the model would work well enough. Honestly I couldn’t get over the fact I’d have to tell friends who wanted to play with the project they’d have to make a special environment before installing.

Beanstalk

I didn’t end up trying beanstalk but did notice it had similar pitfalls. As Kurt Seifried points out in his blog:

The major downside to beanstalkd is that it doesn’t provide any encryption of network traffic or authentication capabilities, so any client with access to a running instance of a beanstalkd server will have access to all the queues in that running instance. This can be gotten around by wrapping beanstalkd in SSL (using stunnel or a similar service) to secure communications and limiting access to beanstalkd queues based on either IP address or by requiring SSL client authentication.

(Source)

So again, if you want to use the service you must either setup extra hoops and/or have an incredibly locked down infrastructure.

ZeroMQ

ZeroMQ is really cool. But you end up with similar problems of network ACL’s providing all of your protection unless you write your own authentication and authorization mechanisms.

What security features does ØMQ support?

None at the moment. ØMQ does not deal with security by design but concentrates on getting your bytes over the network as fast as possible. Solutions exist for security at the transport layer which are well understood and have had many man-years of development invested in them, such as IPsec or OpenVPN.

(Source)

Granted zmq is a bit lower level and used as a building block instead of a solution so it is understandable why some things are pushed back upon the developer to implement as needed.

But Who Cares?

It’s more about being aware.

  • Can anyone promise that network ACL’s won’t be modified to enable a shiny new application?
  • Can you be sure that the other side of the SSL connection will remain safe  and trustworthy?
  • Is any data making it’s way through which can have an effect on process inside the firewall guaranteed safe (example)?
  • If the hosts are multi tenant or in the cloud are you sure everyone who has access to the VM’s or networks are trustworthy?

You and/or the developers of these apps wouldn’t have come up with some kind of security solution if it was OK for any random Joe to play with the service. If someone is able to interact with a service which is “soft on the inside” then it’s likely that service would be an early target.

Simple Examples

For example, let’s imagine an attacker gets access to the service because he is able to take control of an approved host. If the service on the other side is Redis then the attacker could sit and gain information painlessly before copying work from that point forward. If it is a zmq port then an attacker could attach another process to it and get either a copy of everything (SUB, ”) or a subset of data (PULL). Beanstalk probably has similar abilities. The security on the other side of the connection, whether inside or outside the firewall, ends up being as important as the security on the inside as the level of access to the service is more or less the same. All or nothing.

Using an SSL tunnel and only allowing specific hosts may constitue as defense in depth on paper it doesn’t seem to be enough. Maybe I’m to paranoid but if there was authentication and basic authorization in or around the service an intruder would need to gain further information or perform more attacks to gain access.