I Have To Make Things

We are all consumers of things. These things are everything from food to software based services. We are trained to want more things, use more things and find that one thing that will finally make us not want any other things but it never ends up working that way. Over the last 10 years I figured out that making something is way more fulfilling.

Years ago I figured out that I enjoy writing code. Specifically FLOSS. While I always heard FLOSS was great because you “scratch your own itch” I found myself looking at what others were looking for and trying to figure out a way to get it done. That didn’t keep me from coming up with my own ideas, but I found trying to implement someone’s idea of things was more of a challenge — like a puzzle. Imprinting my own way of thinking in code is “easy”, but trying to wrap my mind around someone else’s way of tackling problems isn’t so straight forward.

I believe my want to create has also shaded my view on things like tablets as laptop replacements. I can’t produce things I consider valuable with a tablet (with exceptions to adding a keyboard and having an ssh client). I can produce communications and consume but that doesn’t cut it. I want tools to create that let me produce well crafted results I can feel satisfied with.

Over the last few years I’ve found my want to create things does not stop at producing software. I’ve picked up brewing which has really been a challenge I’ve enjoyed. I still have so far to go but with each attempt I find things I could do better and improve my results. I’ve also picked up more baking which I had done a bit of before. For some reason making bread is a very relaxing process for me.

Over the weekend I found myself with nothing on my plate to make. I felt bored, frustrated and found myself grasping at things to do. For instance, I start to rewrite some code in a different language just to do it. Of course this didn’t actually make me feel any better as it didn’t really serve any real purpose. I turned to cook some food to eat later in the week which helped, but didn’t really do it for me. All this reminded me that I am one of those folks who has to actually make things. I need to create to things. Maybe it’s a way of expression or maybe it just proves I have “value” (Hi Tim) but no matter what I need to make things.

In My Mind It’s Time For Go

I have been, and continue to be, a fan of the Python programming language. It’s clean looking, portable, quick to write, has tons of libraries and a great community. However, I, like many other software developers, don’t think that one must be married to a language. Go has received a lot of attention as of late and it finally makes sense to me as to why.

I’ve gone through, and added to my tool belt, many languages before Python becoming my go to star. The first was Perl. Being so versatile and having a large community of both professional and hobby developers made it an easy early choice for me. The biggest issue for me with Perl was I started to learn how much I liked simple, easy to read code which follows a coding standard. PHP became my fall back web language. It was so simple to write an application it was almost dumb not to use it at the time. No language is perfect and I found many developers at the time didn’t know how to write safe PHP or easy to follow code. I slowly drifted away from PHP as a mainstay. C, while fast and being, well, C, never became a go to mainly because of the time to get things done.

Java just never really did it for me. University tried to shove it down my throat and it didn’t work. Funny thing is I really tried to like it. I gave it multiple chances but would always walk away feeling better with my hands untied from the JVM. I also found myself loathing IDE’s due to people’s insistence that a Java developer needs to use one to be productive. Like many cool kids I became a Java hater for a while and did everything I could to keep friends away from it. But enough of this, let’s talk about Go.

Go has been on my radar since its initial announcement. If I remember right my first thoughts were not positive ones. Everyone and their brother seemed to have their own languages coming out or, at the very least, a DSL which would be the next big thing. I decided to stay back for a bit and see what would happen with the language rather than diving in or slamming the door. Since then there has been a good amount of libraries created for the language, some pretty interesting users such as Docker and DropBox and this post which sums up why Go is a good option when considering Node.js.

What I’ve found is that no tutorial, video nor explanation from a Go fan could convince me to actually see why Go could be so great. After all, the idea behind Go’s OO support sounds half hearted. So one night I did a Google search for what features make people fawn over Go and channels and goroutines came up quite a lot. The next step was to write a Hello World like application that would utilize both features and see how I felt. This is what I came up with:

package main

import "fmt"


// Coms turns into a communication instance. in/out are channels.
type Coms struct {
    in    chan string
    out   chan string
}

// Closes both in and out channels in a Coms instance.
func (c Coms) Close() error {
    fmt.Println("Closing channels")
    close(c.in)
    close(c.out)
    return nil
}
// -------------------


// Main function which sends ping and pong back in two goroutines
// using a Coms instance.
func main() {
    // Creating an "object"
    coms := Coms{make(chan string), make(chan string)}
    // At the end of the function run coms.Close.
    defer coms.Close()

    // First goroutine which ping's out and responds
    // back to pong's with a ping.
    go func() {
        coms.in <- "ping"
        for {
            data := <- coms.out
            fmt.Println(data)
            coms.in <- "ping"
        }
    }()

    // Second goroutine which pong's in response to ping's.
    go func() {
        for {
            data := <- coms.in
            fmt.Println(data)
            coms.out <- "pong"
        }
    }()

    // To exit the application hit enter
    var input string
    fmt.Scanln(&input)
}

 

The result of this code was:

$ go run pingpong.go
ping
pong
ping
pong
ping
....
⏎
Closing channels
$

 

It’s a simple program but it pulled me in. The OO style was not nearly as clunky as I thought it would be and the goroutines were so simple to use I was almost shocked. I also got first hand feel for a non-intrusive strongly typed system. It felt almost like a dynamic language. For things that need speed I felt hooked!

Will I continue to use Python? Most surely! But for things that need to be super fast I think Go will be my default. If someone told me to greenfield a SaaS service tier I’d probably lobby for using Go while keeping the web tier Python or something(s) similar. uWSGI server anyone?.

If you are still on the fence with go take a quick look at it’s feature set and take the 20 minutes to write a simple application utilizing them. Examples are nice but there is nothing better than trying the syntax and structures yourself. Writing some code should tell you if Go is for you.

Red Hat Developer Blog: Git Bonsai, or Keeping Your Branches Well Pruned

Code repositories are the final resting place for code, acting as equal parts bank vault, museum, and graveyard. Unlike a vault, content is almost always added faster than it is removed, but much like a graveyard, there is a definite miasma when a repository becomes too full. That smell is developer frustration, as they have to search through dozens, or eventually, hundreds of branches to find the one they want.

We’ve had sporadic cases where branches did not get merged into masters (and sometimes fixes were overwritten in later releases) and have wasted collectively hundreds of developer hours on “which branch is this in?” exchanges.

Sam and I talk about a simple yet helpful git tool to squash bad branches over at the Red Hat Developer Blog.

Slack Isn’t New, It’s New

New tech tools show up daily. You’ve probably heard about Slack already as it’s being talked about a lot, but just in case you haven’t I’ll give you the tl;dr: Slack integrates your development and infrastructure notifications, chat and documents from different providers into one chat like interface. It’s pretty much the same as Hipchat. All in all slack is a pretty cool and sleek system which allows for easy chatting within a group. But as I tested the service I kept having a feeling of deja vu. This cool new service feels familiar to me. Then it hit me, I have seen this before, haven’t I?

For a long time many engineers, especially in the Free/open arena, have utilized IRC as a way of communication while working. It’s efficient, simple, client agnostic and supports chat rooms as well as private messages. Slack’s main chat interface is very similar to IRC. It’s a chat room with a list of people present and the ability to send private messages. Just like IRC people post messages to the chat room and everyone in the room is able to read them. In a way it’s a little funny to think about chat rooms being “new” but, then again, people have been using instant messaging and SMS style message systems for so long that the concept of the chat room may seem fresh. So the chat interface is similar to IRC, but what about the integration? Aren’t they new?

With Slack the integrations are set up via the web interface. Each integration can send information to a channel with an icon and message. Obviously IRC does not have this functionality directly, but IRC bots do! Many developers set up bots like Supybot with integration with their external development tools. Announcements of new builds, code pushes, deployments, support requests, etc.. show up in channel from the bot. While it’s not as flashy as Slack the same basic integration idea occurs.

Don’t get me wrong, the point of this post isn’t to say that Slack is dumb or simply a copy of something “better”. The point of this post is that, while Slack isn’t really something brand new, it is quite cool. There is a reason developers and ops folks have been setting up things similar to Slack in their own chat rooms for years! The ability to see the development process actually flow can be pretty exciting and empowering. Those who have or will not be able to set up their own integrations have an option to use Slack as a pre-baked set up which, depending on team/company may be more user friendly for the less technical minded. And letting the non-technical see how much is happening day to day can open their eyes to just how much a team is getting done. Probably way more than they realize.

Cloud Message Queues

More and more of my personal work utilizes message queues (MQ) to integrate systems or to propagate longer running work across pools of workers. AMQP is the 300 pound gorilla in the room when it comes to message queuing. Implementations of RabbitMQ, Qpid, Red Hat MRG, etc.. abound. However, when you are the little guy on the field it can be economical to use a cloud service so you can focus directly on your product. Can cloud message queues be a good replacement for running a MQ yourself?

What I Expect

I’m going to make some assumptions that the service will be available, messages will not disappear (unless I set it to do so) and minor network latency is acceptable.

These are features I expect in priority order:

  1. Central connection point (+5)
  2. FIFO support (+4)
  3. Delivery to first available consumer (+3)
  4. Basic publish/subscribe support (+2)
  5. Push message support (+1)

Options

As it turns out there are more players in the cloud messaging space than I would have thought! A quick search turned up the obvious Amazon SQS along with IronMQ, stormmqSoftlayer Message Queue and Marconi.

Amazon SQS

I tend to think Amazon’s SQS is probably the default MQ as a Service. So many people use AWS and it’s right there ready to be used.

  1. Central connection point: Yes (+5)
  2. FIFO support: No
  3. Delivery to first available consumer: Sort of… (+1)
  4. Basic publish/subscribe support: Yes (+2)
  5. Push message support: Sort of… (+0)

There is no doubt that Amazon’s SQS is a great system but right off the bat it’s obvious it doesn’t meet what I expect. According to the FAQ:

Q: Does Amazon SQS provide first-in-first-out (FIFO) access to messages?

No, Amazon SQS does not guarantee FIFO access to messages in Amazon SQS queues, mainly because of the distributed nature of the Amazon SQS. If you require specific message ordering, you should design your application to handle it.

The first consumer who makes an API request will get the next message. That is sort of delivery to the first available consumer but not exactly.

Push messaging is not directly supported but long polling is available. This is close enough that I’d consider it.

Q: What is SQS Long Polling?

SQS long polling is a new way to retrieve messages from your SQS queues. While the traditional SQS short polling returns immediately, even if the queue being polled is empty, SQS long polling doesn’t return a response until a message arrives in the queue, or the long poll times out. SQS long polling makes it easy and inexpensive to retrieve messages from your SQS queue as soon as they are available.

Feature Result: 8

Result: 8/15

IronMQ

Comparing IronMQ to Amazon SQS was interesting. Unlike SQS, IronMQ uses a REST interface which I feel simplifies MQ as a web service. I played a little bit with the service and it was much speedier than I thought it would be! I even tried the beanstalkd support but wasn’t able to get it to fully work.

  1. Central connection point: Yes (+5)
  2. FIFO support: Yes (+4)
  3. Delivery to first available consumer: Sort of… (+1)
  4. Basic publish/subscribe support: Yes (+2)
  5. Push message support: Not really…

Again there is a hiccup on delivery to the first available consumer. Just like SQS, IronMQ is based off requests from the clients. It could match if a client requests only when it can fully dedicate itself to the next message.

Unfortunately, the push support in IronMQ doesn’t cut it for me. If I’m reading the documentation correctly it makes an assumption that the consumers are all listening HTTP servers. I see the use case for this but I also wouldn’t want to make some or all of my consumers publicly listening on the Internet and spinning off work in another thread or process. I’d rather long polling.

Feature Result: 12

Result: 12/15

stormmq

This didn’t get any testing whatsoever. According to the features page it will have a GA of Q1 2013 (which was earlier this year..). I sent a message via Twitter to find out if I’m just seeing old data on their site. For now I will assume that the statement that it’s AMQP 1-0 is true.

  1. Central connection point: Yes (+5)
  2. FIFO support: Yes (+4)
  3. Delivery to first available consumer: Yes (+3)
  4. Basic publish/subscribe support: Yes (+2)
  5. Push message support: Yes (+1)

This was the first service which would meet all of my wants. However, I can’t sign up for it or use it so it kind of knocks it off the list.

Feature Result: 15

Penalty for not being available: -15

Result: 0/15

Softlayer Message Queue

  1. Central connection point: Yes (+5)
  2. FIFO support: No
  3. Delivery to first available consumer: No
  4. Basic publish/subscribe support: Yes (+2)
  5. Push message support: No

Like SQS, Softlayer Message Queue notes that FIFO is not supported:

Does SoftLayer Message Queue provide first-in-first-out (FIFO) message access?

While the system does it’s best to return messages in FIFO order, it is not guaranteed. A number of factors can influence message ordering, including message timeouts and updated visibility intervals.

The FAQ also notes that it is possible to have conditions where a consumer (or consumers?) may get the exact same message multiple times.

How can multiple readers access the same message queue, without losing messages or processing them many times?

Queues can be accessed by any number of consumers/clients. When a consumer requests messages from the queue, each message is marked invisible—this prevents other consumers from receiving the same message. However, the distributed nature of the message queue cannot guarantee single delivery of any one message. While the system makes a best effort, clients should expect the possibility of receiving the same message multiple times.

Since the MQ service is web service (REST) based there is no pushing of messages, only pulling. I didn’t see anything noting long polling or even push to HTTP servers like IronMQ.

Feature Result: 7

Result: 7/15

Marconi

  1. Central connection point: Yes (+5)
  2. FIFO support: Yes (+4)
  3. Delivery to first available consumer: Sort of.. (+1)
  4. Basic publish/subscribe support: Yes (+2)
  5. Push message support: In progress…

Since Marconi is primarily web service based (REST) it has the same issues as Amazon SQS and IronMQ.

Push messaging is not currently available from what I can tell but there is ZMQ bindings in the work with AMQP work coming later.

The negative part to using Marconi is that one either needs to be using an OpenStack based service already or they will need to set up the service themselves. This does add some overhead to using it. Also I keep wanting to say macaroni.

Feature result: 12

Bonus for being Open Source: +2

Penalty for needing specific software: -5

Result: 9/15

Conclusion

So can cloud message queuing be a good replacement for running your own service? It highly depends on what you are doing, but if you don’t need all the features of modern MQ systems then you can run with a cloud MQ and cut some time/cost.

It may seem unfair, but I am wary of using stormmq because of the old GA info on the site even if they went GA today. At a later date after GA I’d consider trying them but I’d want to see a bit better messaging (no pun intended) via their site.

For me it makes the most sense to use IronMQ until I end up on an OpenStack based system at which point, if ZMQ or AMQP is available, I’d likely switch over. Having ZMQ/AMQP’s push ability would be worth the move.

Red Hat Developer Blog: Feeling Developer Pain

The rest of this post describes our journey from initially trying to implement a simple solution to improve the day-to-day lives of developers, through the technical limitations we experienced along the way, and finally arrives at the empathy for our developers we’ve gained from that experience. We’ll wrap up with a note on how Red Hat Software Collections (announced as GA in September) would’ve simplified our development process.

Read the whole post Tim and I wrote over at the Red Hat Develope Blog.

More Editors/IDE’s

In my last post I talked about not being able to find a good Python editor/IDE other than vim. Nothing has really changed since then but there was another editor and IDE that was brought to my attention which I failed to point out. Let’s talk about them!

Editors/IDE’s

Emacs

While talking with Tim Bielawa I was reminded about Emacs since it’s what Tim uses. Emacs has such amazing integration that people sometimes say it’s an operating system itself! I’ve really only ever given Emacs two real shots at being my main editor. The first time was when I was just starting to get into programming and was reading a lot about what other programmers and system administrators used. At first it seemed a lot easier than vi but I ended up running with vi/vim since, at the very least, vi seemed to always be present on every Linux/BSD box I worked with by default. The second attempt was after I promised a few Emacs users at work that I’d give it another fair shot. I said I would use Emacs when I’d normally use vim for a full week knowing that it would force me to learn more about the editor. It wasn’t easy to stop trying to use mode editing but I was able to code without feeling too contained by the editor (Note: the contained feeling was me not knowing the editor well, not Emacs itself. It was really obvious that Emacs is a powerful editor). My biggest gripes from the week long test were around needing to install emacs on systems for use (which is sort of a silly one, I admit) and I felt some of the commands were way to long. I don’t remember the last one off the top of my head but I do remember that one command had a bunch of dashes and was frustrating every time I needed to use it. I think it’s about time I try it again and see if I can overcome my hurdles trying to use it efficiently. Who knows, maybe third time’s the charm!

Cloud9

Cloud9 is really cool. It’s not your grandfather’s IDE by any stretch. As the name implies the application exists out in the cloud (They use Openshift). I love the idea that I can start editing code with full IDE features from any machine I’m currently occupying. I have not tried using Cloud9 with a tablet (with keyboard of course!) yet but if that works then this thing would have rockstar status in my mind. I’ve used it for a few projects for both ease of use and to test out some of the features Cloud9 boasts. Being that it is on Openshift the IDE has it’s own platform letting you install tools and dependencies. There are also some collaboration features which I have not tried out yet mainly because I’m not sure how that works when you are using Github (IE: can they push code under your name or are they restricted?). I would use Cloud9 a lot more if it wasn’t for the Internet. While Cloud9 is pretty responsive in most cases but due to some point in the network connection between “here” and “there” things slow down or stop responding for a second or two. If this was something other than coding I probably wouldn’t care that much … but this is coding. Any hiccups while editing breaks concentration and slows down progress. One other issue I noticed was the lack of preferences across ones instances. Say you have two projects in Cloud9. Each project has it’s own IDE instance. If you want to set preferences for both IDE instances you will have to open each on it’s own and set them. You can not set any kind of global default preferences for all your instances. Hopefully they will add that functionality as I’m pretty positive I’m not the only person who finds that a bit weird. Over time Cloud9 and similar IDE’s will find ways to speed up and add better preference support but until then, in my mind, Cloud9 is straddling the line between full contender and really cool tech preview to keep my eye on.