Programmatic Ansible Middle Ground

 

Forward

About a year ago serversforhackers posted a great article on how to run Ansible programamatically  Since then Ansible has had a major release which introduced changes within the Python API.

Simulating The CLI

Not that long ago Jason  DeTiberus and I were talking about how to use Ansible from within other Python packages. One of the things he said was that it should be possible to reuse the command line code instead of the internal API if you hook into the right place. I finally had some time to take a look and it seems he’s right!

If you took a look at the 2.0 API you’ll see there is a lot more power handed over to you as the developer but with that comes a lot of code. Code that for many will be nearly copy/paste style code directly from command-line interface code. So when there is not a need for the extra power why not just reuse code that already exists?

import os  # Used for expanding paths

from ansible.cli.playbook import PlaybookCLI
from ansible.errors import AnsibleOptionsError, AnsibleParserError

def execute_playbook(playbook, hosts, args=[]):
    """
    :param playbook: Full path to the playbook to execute.
    :type playbook: str
    :param hosts: A host or hosts to target the playbook against.
    :type hosts: str, list, or tuple
    :param args: Other arguments to pass to the run.
    :type args: list
    :returns: The TaskQueueHandler for the run.
    :rtype: ansible.executor.task_queue_manager.TaskQueueManager.
    """
    # Set hosts args up right for the ansible parser. It likes to have trailing ,'s
    if isinstance(hosts, basestring):
        hosts = hosts + ','
    elif hasattr(hosts, '__iter__'):
        hosts = ','.join(hosts) + ','
    else:
        raise AnsibleParserError('Can not parse hosts of type {}'.format(
            type(hosts)))

    # Create the cli object
    cli_args = ['playbook'] + args + ['-i', hosts, os.path.realpath(playbook)]
    print('Executing: {}'.format(' '.join(cli_args)))
    cli = PlaybookCLI(cli_args)
    # Parse args and run it
    try:
        cli.parse()
        # Return the result:
        # 0: Success
        # 1: "Error"
        # 2: Host failed
        # 3: Unreachable
        # 4: Parser Error
        # 5: Options error
        return cli.run()
    except (AnsibleParserError, AnsibleOptionsError) as error:
        print('{}: {}'.format(type(error), error))
        raise error

 

Breaking It Down

The function starts off with some hosts parsing. This is not really needed but it does make the function easier to work with. On the command line Ansible likes to have a comma at the end of hosts passed in. This chunk of code makes sure that if a list or string is given for a host that the resulting host string is properly formatted.

    # Set hosts args up right for the ansible parser. It likes to have trailing ,'s
    if isinstance(hosts, basestring):
        hosts = hosts + ','
    elif hasattr(hosts, '__iter__'):
        hosts = ','.join(hosts) + ','
    else:
        raise AnsibleParserError('Can not parse hosts of type {}'.format(type(hosts)))

The Real Code

This chunk of code is what is actually calling Ansible. It creates the command line argument list, creates a PlaybookCLI instance, has it parsed, and then executes the playbook.

    # Create the cli object
    cli_args = ['playbook'] + args + ['-i', hosts, os.path.realpath(playbook)]
    print('Executing: {}'.format(' '.join(cli_args)))
    cli = PlaybookCLI(cli_args)
    # Parse args and run it
    try:
        cli.parse()
        # Return the result:
        # 0: Success
        # 1: "Error"
        # 2: Host failed
        # 3: Unreachable
        # 4: Parser Error
        # 5: Options error
        return cli.run()
    except (AnsibleParserError, AnsibleOptionsError) as error:
        print('{}: {}'.format(type(error), error))
        raise error

Using The Function

# Execute /tmp/test.yaml with 2 hosts
result = execute_playbook('/tmp/test.yaml', ['192.168.152.100', '192.168.152.101'])

# Execute /tmp/test.yaml with 1 host and add the -v flag
result = execute_playbook('/tmp/test.yaml', '192.168.152.101', ['-v'])

Intercepting The Output

One drawback of using the command-line interface code directly is that the output is expected to go to the user in the standard way. That is to say, it’s sent to the screen and colorized. This will probably be fine for some, but others may want to grab the output and use it in some form. While it is possible to change output through the configuration options it is also possible to monkey patch display and intercept the output for your own use cases. As an example, here is a Display class which forwards all output that is not meant for the screen only to our logging.info method.

# MONKEY PATCH to catch output. This must happen at the start of the code!
import logging

from ansible.utils.display import Display

# Set up our logging
logger = logging.getLogger('transport')
logger.setLevel(logging.INFO)
handler = logging.StreamHandler()
handler.formatter = logging.Formatter('%(name)s - %(message)s')
logger.addHandler(handler)

class LogForward(Display):
    """
    Quick hack of a log forwarder
    """

    def display(self, msg, screen_only=None, *args, **kwargs):
        """
        Pass display data to the logger.
        :param msg: The message to log.
        :type msg: str
        :param args: All other non-keyword arguments.
        :type args: list
        :param kwargs: All other keyword arguments.
        :type kwargs: dict
        """
        # Ignore if it is screen only output
        if screen_only:
            return
        logging.getLogger('transport').info(msg)

    # Forward it all to display
    info = display
    warning = display
    error = display
    # Ignore debug
    debug = lambda s, *a, **k: True

# By simply setting display Ansible will slurp it in as the display instance
display = LogForward()
# END MONKEY PATCH. Add code after this line.

Putting It All Together

If you want to use it all together it should look like this:

# MONKEY PATCH to catch output. This must happen at the start of the code!
import logging

from ansible.utils.display import Display

# Set up our logging
logger = logging.getLogger('transport')
logger.setLevel(logging.INFO)
handler = logging.StreamHandler()
handler.formatter = logging.Formatter('%(name)s - %(message)s')
logger.addHandler(handler)

class LogForward(Display):
    """
    Quick hack of a log forwarder
    """

    def display(self, msg, screen_only=None, *args, **kwargs):
        """
        Pass display data to the logger.
        :param msg: The message to log.
        :type msg: str
        :param args: All other non-keyword arguments.
        :type args: list
        :param kwargs: All other keyword arguments.
        :type kwargs: dict
        """
        # Ignore if it is screen only output
        if screen_only:
            return
        logging.getLogger('transport').info(msg)

    # Forward it all to display
    info = display
    warning = display
    error = display
    # Ignore debug
    debug = lambda s, *a, **k: True

# By simply setting display Ansible will slurp it in as the display instance
display = LogForward()
# END MONKEY PATCH. Add code after this line.

import os  # Used for expanding paths

from ansible.cli.playbook import PlaybookCLI
from ansible.errors import AnsibleOptionsError, AnsibleParserError

def execute_playbook(playbook, hosts, args=[]):
    """
    :param playbook: Full path to the playbook to execute.
    :type playbook: str
    :param hosts: A host or hosts to target the playbook against.
    :type hosts: str, list, or tuple
    :param args: Other arguments to pass to the run.
    :type args: list
    :returns: The TaskQueueHandler for the run.
    :rtype: ansible.executor.task_queue_manager.TaskQueueManager.
    """
    # Set hosts args up right for the ansible parser. It likes to have trailing ,'s
    if isinstance(hosts, basestring):
        hosts = hosts + ','
    elif hasattr(hosts, '__iter__'):
        hosts = ','.join(hosts) + ','
    else:
        raise AnsibleParserError('Can not parse hosts of type {}'.format(
            type(hosts)))

    # Create the cli object
    cli_args = ['playbook'] + args + ['-i', hosts, os.path.realpath(playbook)]
    logger.info('Executing: {}'.format(' '.join(cli_args)))
    cli = PlaybookCLI(cli_args)
    # Parse args and run it
    try:
        cli.parse()
        # Return the result:
        # 0: Success
        # 1: "Error"
        # 2: Host failed
        # 3: Unreachable
        # 4: Parser Error
        # 5: Options error
        return cli.run()
    except (AnsibleParserError, AnsibleOptionsError) as error:
        logger.error('{}: {}'.format(type(error), error))
        raise error

 

Pros and Cons

Of course nothing is without drawbacks. Here are some negatives with this method:

  • No direct access to “TaskQueueManager“
  • If the CLI changes the code must change
  • Monkey patching …. ewww

But the positives seem to be worth it so far:

  • You don’t have to deal with “TaskQueueManager“ and all of the construction code
  • The CLI doesn’t seem to change often
  • The same commands one would run on the CLI can easily be extrapolated and even run manually
Advertisements

In My Mind It’s Time For Go

I have been, and continue to be, a fan of the Python programming language. It’s clean looking, portable, quick to write, has tons of libraries and a great community. However, I, like many other software developers, don’t think that one must be married to a language. Go has received a lot of attention as of late and it finally makes sense to me as to why.

I’ve gone through, and added to my tool belt, many languages before Python becoming my go to star. The first was Perl. Being so versatile and having a large community of both professional and hobby developers made it an easy early choice for me. The biggest issue for me with Perl was I started to learn how much I liked simple, easy to read code which follows a coding standard. PHP became my fall back web language. It was so simple to write an application it was almost dumb not to use it at the time. No language is perfect and I found many developers at the time didn’t know how to write safe PHP or easy to follow code. I slowly drifted away from PHP as a mainstay. C, while fast and being, well, C, never became a go to mainly because of the time to get things done.

Java just never really did it for me. University tried to shove it down my throat and it didn’t work. Funny thing is I really tried to like it. I gave it multiple chances but would always walk away feeling better with my hands untied from the JVM. I also found myself loathing IDE’s due to people’s insistence that a Java developer needs to use one to be productive. Like many cool kids I became a Java hater for a while and did everything I could to keep friends away from it. But enough of this, let’s talk about Go.

Go has been on my radar since its initial announcement. If I remember right my first thoughts were not positive ones. Everyone and their brother seemed to have their own languages coming out or, at the very least, a DSL which would be the next big thing. I decided to stay back for a bit and see what would happen with the language rather than diving in or slamming the door. Since then there has been a good amount of libraries created for the language, some pretty interesting users such as Docker and DropBox and this post which sums up why Go is a good option when considering Node.js.

What I’ve found is that no tutorial, video nor explanation from a Go fan could convince me to actually see why Go could be so great. After all, the idea behind Go’s OO support sounds half hearted. So one night I did a Google search for what features make people fawn over Go and channels and goroutines came up quite a lot. The next step was to write a Hello World like application that would utilize both features and see how I felt. This is what I came up with:

package main

import "fmt"


// Coms turns into a communication instance. in/out are channels.
type Coms struct {
    in    chan string
    out   chan string
}

// Closes both in and out channels in a Coms instance.
func (c Coms) Close() error {
    fmt.Println("Closing channels")
    close(c.in)
    close(c.out)
    return nil
}
// -------------------


// Main function which sends ping and pong back in two goroutines
// using a Coms instance.
func main() {
    // Creating an "object"
    coms := Coms{make(chan string), make(chan string)}
    // At the end of the function run coms.Close.
    defer coms.Close()

    // First goroutine which ping's out and responds
    // back to pong's with a ping.
    go func() {
        coms.in <- "ping"
        for {
            data := <- coms.out
            fmt.Println(data)
            coms.in <- "ping"
        }
    }()

    // Second goroutine which pong's in response to ping's.
    go func() {
        for {
            data := <- coms.in
            fmt.Println(data)
            coms.out <- "pong"
        }
    }()

    // To exit the application hit enter
    var input string
    fmt.Scanln(&input)
}

 

The result of this code was:

$ go run pingpong.go
ping
pong
ping
pong
ping
....
⏎
Closing channels
$

 

It’s a simple program but it pulled me in. The OO style was not nearly as clunky as I thought it would be and the goroutines were so simple to use I was almost shocked. I also got first hand feel for a non-intrusive strongly typed system. It felt almost like a dynamic language. For things that need speed I felt hooked!

Will I continue to use Python? Most surely! But for things that need to be super fast I think Go will be my default. If someone told me to greenfield a SaaS service tier I’d probably lobby for using Go while keeping the web tier Python or something(s) similar. uWSGI server anyone?.

If you are still on the fence with go take a quick look at it’s feature set and take the 20 minutes to write a simple application utilizing them. Examples are nice but there is nothing better than trying the syntax and structures yourself. Writing some code should tell you if Go is for you.

Red Hat Developer Blog: Feeling Developer Pain

The rest of this post describes our journey from initially trying to implement a simple solution to improve the day-to-day lives of developers, through the technical limitations we experienced along the way, and finally arrives at the empathy for our developers we’ve gained from that experience. We’ll wrap up with a note on how Red Hat Software Collections (announced as GA in September) would’ve simplified our development process.

Read the whole post Tim and I wrote over at the Red Hat Develope Blog.

Python IDE Woes

I love  the Python programming language. I could spend hours explaining why it’s generally my go to language when coding something new but that’s not what I really want to touch on today. Today it’s about IDEs and editors.

IDE or editor selection is almost a cultish exercise. Developers break out into small societies around coding tools and banish those exhibiting wandering eyes towards  tool lust. In some cases I’ve fallen into those patterns when I find a tool that I enjoy. It can be frustrating to have to uproot your development workflow to accommodate a new tool even when it’s obvious that after the initial learning curve the tool will make life easier. With all that said I am not stuck to any IDE or editor. I’m an IDE/editor swinger.

There are plenty of choices for Python, many of which I’ve used throughout the years. My problem is that all of those I’ve tried I always end up moving away from to one of the default editors: vim or emacs (in my case vim). For the heck of it this post is to go through the main reasons for the most recent movements back to vim.

IDEs and Editors

Eclipse

I used to joke that every 6 months I’d give Eclipse a try. It’s very popular in the Java programming world and has a great Python plugin called PyDev. However, Eclipse itself always feels sluggish even when I’m writing Java. Keep in mind I’m not on some old hardware which is limiting the IDE nor am I doing some corner case kind of usage. The sluggishness is not terrible either, but it’s noticeable reminding me that I’m using a very large piece of software. One of my thoughts is that if I notice the IDE/editor for anything other than a helpful feature then there is likely a problem.

NetBeans

NetBeans is not all that different than Eclipse in my mind. It’s a Java IDE which is extendable to other languages. Last time I tried NetBeans it was less sluggish than Eclipse. My main issue with NetBeans is I can never tell what is going one with the IDE. There was a pretty big push for first class Python support in NetBeans. I played with that IDE version for a while and, overall, liked it. Now searching python on the NetBeans site returns nothing. It seems like there are random community efforts to bring Python back into NetBeans but I want something that works well right now (no offense to any of the efforts!).

PyCharm

Another similar IDE is PyCharm. I have at least one fellow Python developer which swears by PyCharm and I can see why. It doesn’t seem sluggish even though it’s pretty large. It has a good feature set. But it has confusing proprietary licensing and the IDE is specific to the language. If you need to get some Ruby coding done then it’s a different JetBrains IDE and, I assume, another purchase and license agreement. That’s a bit frustrating!

Komodo IDE

Komodo IDE was one of the earlier Python IDEs I tried.  I have to admit not trying it in a while, but last time I played with it I felt it was sluggish in a similar way to Eclipse. Like Eclipse it has a lot of plugins/add-ons available which is great but, then again, it’s proprietary and costs $382. Sort of a long term deal breaker right there.

Komodo Edit

The younger brother of Komodo IDE, Komodo Edit actually feels more useable to me. I believe it’s just the basic core of Komodo IDE, but that works in it’s favor. It seemed faster and kept out of my way. The initial start screen always feels clunky. However, the license is weird and is not listed on the FSF or OSI list. I guess it’s proprietary? Confusion is not a good thing.

Sublime Text

This one kills me. Sublime Text is a fantastic editor. I really like it! The editor is very fast and has some unique features. It’s available on all three major computer platforms (Linux, Windows and OS X). The license is proprietary but is simple, understandable. I believe this may be a first. Though the fact it’s proprietary makes it a harder editor to make my default. If Sublime Text was Free Software or Open Source it would be my programming default editor.

Scribes

This was a nice editor. I say was for Scribes because it looks like it is no longer actively developed. It’s a shame because it was a powerful and fast editor with an Open license. Even though it’s an older editor than Sublime Text I liked it for a lot of the same reasons. If this was still being worked on it would be my default programming editor.

PIDA

PIDA was my default IDE for a while. If I was using an IDE it was likely to be PIDA. It was Open, had good plugins, fast, embedded Vim as the editor, said it loved me, etc.. However, the PIDA web site has disappeared and the last stable release was about 3 years ago. It seems like a8 may be the replacement but I’ve not had time to run with it yet.

Others To Look At

As I stated before I’d like to spend some time with a8. It looks like it’s X based only (Linux, BSD, etc..) which is fine for me since most all of my dev is on Linux. I’ve also seen a lot of talk about Ninja IDE and it looks promising though I’m generally not a fan of specific language IDEs since I am not always using the same language.

What Do You Use?

Seriously. If you are doing Python development regularly what are you using and why? What features have you found to be amazing and which ones are overrated marketing dribble? Am I the only guy who continues to go back to vim in this day and age?