Programmatic Ansible Middle Ground

 

Forward

About a year ago serversforhackers posted a great article on how to run Ansible programamatically  Since then Ansible has had a major release which introduced changes within the Python API.

Simulating The CLI

Not that long ago Jason  DeTiberus and I were talking about how to use Ansible from within other Python packages. One of the things he said was that it should be possible to reuse the command line code instead of the internal API if you hook into the right place. I finally had some time to take a look and it seems he’s right!

If you took a look at the 2.0 API you’ll see there is a lot more power handed over to you as the developer but with that comes a lot of code. Code that for many will be nearly copy/paste style code directly from command-line interface code. So when there is not a need for the extra power why not just reuse code that already exists?

import os  # Used for expanding paths

from ansible.cli.playbook import PlaybookCLI
from ansible.errors import AnsibleOptionsError, AnsibleParserError

def execute_playbook(playbook, hosts, args=[]):
    """
    :param playbook: Full path to the playbook to execute.
    :type playbook: str
    :param hosts: A host or hosts to target the playbook against.
    :type hosts: str, list, or tuple
    :param args: Other arguments to pass to the run.
    :type args: list
    :returns: The TaskQueueHandler for the run.
    :rtype: ansible.executor.task_queue_manager.TaskQueueManager.
    """
    # Set hosts args up right for the ansible parser. It likes to have trailing ,'s
    if isinstance(hosts, basestring):
        hosts = hosts + ','
    elif hasattr(hosts, '__iter__'):
        hosts = ','.join(hosts) + ','
    else:
        raise AnsibleParserError('Can not parse hosts of type {}'.format(
            type(hosts)))

    # Create the cli object
    cli_args = ['playbook'] + args + ['-i', hosts, os.path.realpath(playbook)]
    print('Executing: {}'.format(' '.join(cli_args)))
    cli = PlaybookCLI(cli_args)
    # Parse args and run it
    try:
        cli.parse()
        # Return the result:
        # 0: Success
        # 1: "Error"
        # 2: Host failed
        # 3: Unreachable
        # 4: Parser Error
        # 5: Options error
        return cli.run()
    except (AnsibleParserError, AnsibleOptionsError) as error:
        print('{}: {}'.format(type(error), error))
        raise error

 

Breaking It Down

The function starts off with some hosts parsing. This is not really needed but it does make the function easier to work with. On the command line Ansible likes to have a comma at the end of hosts passed in. This chunk of code makes sure that if a list or string is given for a host that the resulting host string is properly formatted.

    # Set hosts args up right for the ansible parser. It likes to have trailing ,'s
    if isinstance(hosts, basestring):
        hosts = hosts + ','
    elif hasattr(hosts, '__iter__'):
        hosts = ','.join(hosts) + ','
    else:
        raise AnsibleParserError('Can not parse hosts of type {}'.format(type(hosts)))

The Real Code

This chunk of code is what is actually calling Ansible. It creates the command line argument list, creates a PlaybookCLI instance, has it parsed, and then executes the playbook.

    # Create the cli object
    cli_args = ['playbook'] + args + ['-i', hosts, os.path.realpath(playbook)]
    print('Executing: {}'.format(' '.join(cli_args)))
    cli = PlaybookCLI(cli_args)
    # Parse args and run it
    try:
        cli.parse()
        # Return the result:
        # 0: Success
        # 1: "Error"
        # 2: Host failed
        # 3: Unreachable
        # 4: Parser Error
        # 5: Options error
        return cli.run()
    except (AnsibleParserError, AnsibleOptionsError) as error:
        print('{}: {}'.format(type(error), error))
        raise error

Using The Function

# Execute /tmp/test.yaml with 2 hosts
result = execute_playbook('/tmp/test.yaml', ['192.168.152.100', '192.168.152.101'])

# Execute /tmp/test.yaml with 1 host and add the -v flag
result = execute_playbook('/tmp/test.yaml', '192.168.152.101', ['-v'])

Intercepting The Output

One drawback of using the command-line interface code directly is that the output is expected to go to the user in the standard way. That is to say, it’s sent to the screen and colorized. This will probably be fine for some, but others may want to grab the output and use it in some form. While it is possible to change output through the configuration options it is also possible to monkey patch display and intercept the output for your own use cases. As an example, here is a Display class which forwards all output that is not meant for the screen only to our logging.info method.

# MONKEY PATCH to catch output. This must happen at the start of the code!
import logging

from ansible.utils.display import Display

# Set up our logging
logger = logging.getLogger('transport')
logger.setLevel(logging.INFO)
handler = logging.StreamHandler()
handler.formatter = logging.Formatter('%(name)s - %(message)s')
logger.addHandler(handler)

class LogForward(Display):
    """
    Quick hack of a log forwarder
    """

    def display(self, msg, screen_only=None, *args, **kwargs):
        """
        Pass display data to the logger.
        :param msg: The message to log.
        :type msg: str
        :param args: All other non-keyword arguments.
        :type args: list
        :param kwargs: All other keyword arguments.
        :type kwargs: dict
        """
        # Ignore if it is screen only output
        if screen_only:
            return
        logging.getLogger('transport').info(msg)

    # Forward it all to display
    info = display
    warning = display
    error = display
    # Ignore debug
    debug = lambda s, *a, **k: True

# By simply setting display Ansible will slurp it in as the display instance
display = LogForward()
# END MONKEY PATCH. Add code after this line.

Putting It All Together

If you want to use it all together it should look like this:

# MONKEY PATCH to catch output. This must happen at the start of the code!
import logging

from ansible.utils.display import Display

# Set up our logging
logger = logging.getLogger('transport')
logger.setLevel(logging.INFO)
handler = logging.StreamHandler()
handler.formatter = logging.Formatter('%(name)s - %(message)s')
logger.addHandler(handler)

class LogForward(Display):
    """
    Quick hack of a log forwarder
    """

    def display(self, msg, screen_only=None, *args, **kwargs):
        """
        Pass display data to the logger.
        :param msg: The message to log.
        :type msg: str
        :param args: All other non-keyword arguments.
        :type args: list
        :param kwargs: All other keyword arguments.
        :type kwargs: dict
        """
        # Ignore if it is screen only output
        if screen_only:
            return
        logging.getLogger('transport').info(msg)

    # Forward it all to display
    info = display
    warning = display
    error = display
    # Ignore debug
    debug = lambda s, *a, **k: True

# By simply setting display Ansible will slurp it in as the display instance
display = LogForward()
# END MONKEY PATCH. Add code after this line.

import os  # Used for expanding paths

from ansible.cli.playbook import PlaybookCLI
from ansible.errors import AnsibleOptionsError, AnsibleParserError

def execute_playbook(playbook, hosts, args=[]):
    """
    :param playbook: Full path to the playbook to execute.
    :type playbook: str
    :param hosts: A host or hosts to target the playbook against.
    :type hosts: str, list, or tuple
    :param args: Other arguments to pass to the run.
    :type args: list
    :returns: The TaskQueueHandler for the run.
    :rtype: ansible.executor.task_queue_manager.TaskQueueManager.
    """
    # Set hosts args up right for the ansible parser. It likes to have trailing ,'s
    if isinstance(hosts, basestring):
        hosts = hosts + ','
    elif hasattr(hosts, '__iter__'):
        hosts = ','.join(hosts) + ','
    else:
        raise AnsibleParserError('Can not parse hosts of type {}'.format(
            type(hosts)))

    # Create the cli object
    cli_args = ['playbook'] + args + ['-i', hosts, os.path.realpath(playbook)]
    logger.info('Executing: {}'.format(' '.join(cli_args)))
    cli = PlaybookCLI(cli_args)
    # Parse args and run it
    try:
        cli.parse()
        # Return the result:
        # 0: Success
        # 1: "Error"
        # 2: Host failed
        # 3: Unreachable
        # 4: Parser Error
        # 5: Options error
        return cli.run()
    except (AnsibleParserError, AnsibleOptionsError) as error:
        logger.error('{}: {}'.format(type(error), error))
        raise error

 

Pros and Cons

Of course nothing is without drawbacks. Here are some negatives with this method:

  • No direct access to “TaskQueueManager“
  • If the CLI changes the code must change
  • Monkey patching …. ewww

But the positives seem to be worth it so far:

  • You don’t have to deal with “TaskQueueManager“ and all of the construction code
  • The CLI doesn’t seem to change often
  • The same commands one would run on the CLI can easily be extrapolated and even run manually
Advertisements

Flask-Track-Usage 1.1.0 Released

A few years ago the initial Flask-Track-Usage release was announced via my blog. At the time I thought I’d probably be the one user. I’m glad to say I was wrong! Today I’m happy to announce the release of Flask-Track-Usage 1.1.0 which sports a number enhancements and bug fixes.

Unfortunately, some changes are not backwards compatible. However, I believe the backwards incompatible changes make the overall experience better. If you would like to stick with the previous version of Flask-Track-Usage make sure to version pin in your requirements file/section:

flask_track_usage==1.0.1

Version 1.1.0 has made changes requested by the community as well as a few bug fixes. These include:

  • Addition of the X-Forwarded-For header as xforwardedfor in storage. Requested by jamylak.
  • Configurable GeoIP endpoint support. Requested by jamylak.
  • Migration from pymongo.Connection to pymongo.MongoClient.
  • Better SQLStorage metadata handling. Requested by gouthambs.
  • SQLStorage implementation redesign. Requested and implemented by gouthambs.
  • Updated documentation for 1.1.0.
  • Better unittesting.

I’d like to thank Gouthaman Balaraman who has been a huge help authoring the SQLStorage based on the SQLAlchemy ORM and providing feedback and support on Flask-Track-Usage design.

As always, please report bugs and feature requests on the GitHub Issues Page.

I Have To Make Things

We are all consumers of things. These things are everything from food to software based services. We are trained to want more things, use more things and find that one thing that will finally make us not want any other things but it never ends up working that way. Over the last 10 years I figured out that making something is way more fulfilling.

Years ago I figured out that I enjoy writing code. Specifically FLOSS. While I always heard FLOSS was great because you “scratch your own itch” I found myself looking at what others were looking for and trying to figure out a way to get it done. That didn’t keep me from coming up with my own ideas, but I found trying to implement someone’s idea of things was more of a challenge — like a puzzle. Imprinting my own way of thinking in code is “easy”, but trying to wrap my mind around someone else’s way of tackling problems isn’t so straight forward.

I believe my want to create has also shaded my view on things like tablets as laptop replacements. I can’t produce things I consider valuable with a tablet (with exceptions to adding a keyboard and having an ssh client). I can produce communications and consume but that doesn’t cut it. I want tools to create that let me produce well crafted results I can feel satisfied with.

Over the last few years I’ve found my want to create things does not stop at producing software. I’ve picked up brewing which has really been a challenge I’ve enjoyed. I still have so far to go but with each attempt I find things I could do better and improve my results. I’ve also picked up more baking which I had done a bit of before. For some reason making bread is a very relaxing process for me.

Over the weekend I found myself with nothing on my plate to make. I felt bored, frustrated and found myself grasping at things to do. For instance, I start to rewrite some code in a different language just to do it. Of course this didn’t actually make me feel any better as it didn’t really serve any real purpose. I turned to cook some food to eat later in the week which helped, but didn’t really do it for me. All this reminded me that I am one of those folks who has to actually make things. I need to create to things. Maybe it’s a way of expression or maybe it just proves I have “value” (Hi Tim) but no matter what I need to make things.

Python IDE Woes

I love  the Python programming language. I could spend hours explaining why it’s generally my go to language when coding something new but that’s not what I really want to touch on today. Today it’s about IDEs and editors.

IDE or editor selection is almost a cultish exercise. Developers break out into small societies around coding tools and banish those exhibiting wandering eyes towards  tool lust. In some cases I’ve fallen into those patterns when I find a tool that I enjoy. It can be frustrating to have to uproot your development workflow to accommodate a new tool even when it’s obvious that after the initial learning curve the tool will make life easier. With all that said I am not stuck to any IDE or editor. I’m an IDE/editor swinger.

There are plenty of choices for Python, many of which I’ve used throughout the years. My problem is that all of those I’ve tried I always end up moving away from to one of the default editors: vim or emacs (in my case vim). For the heck of it this post is to go through the main reasons for the most recent movements back to vim.

IDEs and Editors

Eclipse

I used to joke that every 6 months I’d give Eclipse a try. It’s very popular in the Java programming world and has a great Python plugin called PyDev. However, Eclipse itself always feels sluggish even when I’m writing Java. Keep in mind I’m not on some old hardware which is limiting the IDE nor am I doing some corner case kind of usage. The sluggishness is not terrible either, but it’s noticeable reminding me that I’m using a very large piece of software. One of my thoughts is that if I notice the IDE/editor for anything other than a helpful feature then there is likely a problem.

NetBeans

NetBeans is not all that different than Eclipse in my mind. It’s a Java IDE which is extendable to other languages. Last time I tried NetBeans it was less sluggish than Eclipse. My main issue with NetBeans is I can never tell what is going one with the IDE. There was a pretty big push for first class Python support in NetBeans. I played with that IDE version for a while and, overall, liked it. Now searching python on the NetBeans site returns nothing. It seems like there are random community efforts to bring Python back into NetBeans but I want something that works well right now (no offense to any of the efforts!).

PyCharm

Another similar IDE is PyCharm. I have at least one fellow Python developer which swears by PyCharm and I can see why. It doesn’t seem sluggish even though it’s pretty large. It has a good feature set. But it has confusing proprietary licensing and the IDE is specific to the language. If you need to get some Ruby coding done then it’s a different JetBrains IDE and, I assume, another purchase and license agreement. That’s a bit frustrating!

Komodo IDE

Komodo IDE was one of the earlier Python IDEs I tried.  I have to admit not trying it in a while, but last time I played with it I felt it was sluggish in a similar way to Eclipse. Like Eclipse it has a lot of plugins/add-ons available which is great but, then again, it’s proprietary and costs $382. Sort of a long term deal breaker right there.

Komodo Edit

The younger brother of Komodo IDE, Komodo Edit actually feels more useable to me. I believe it’s just the basic core of Komodo IDE, but that works in it’s favor. It seemed faster and kept out of my way. The initial start screen always feels clunky. However, the license is weird and is not listed on the FSF or OSI list. I guess it’s proprietary? Confusion is not a good thing.

Sublime Text

This one kills me. Sublime Text is a fantastic editor. I really like it! The editor is very fast and has some unique features. It’s available on all three major computer platforms (Linux, Windows and OS X). The license is proprietary but is simple, understandable. I believe this may be a first. Though the fact it’s proprietary makes it a harder editor to make my default. If Sublime Text was Free Software or Open Source it would be my programming default editor.

Scribes

This was a nice editor. I say was for Scribes because it looks like it is no longer actively developed. It’s a shame because it was a powerful and fast editor with an Open license. Even though it’s an older editor than Sublime Text I liked it for a lot of the same reasons. If this was still being worked on it would be my default programming editor.

PIDA

PIDA was my default IDE for a while. If I was using an IDE it was likely to be PIDA. It was Open, had good plugins, fast, embedded Vim as the editor, said it loved me, etc.. However, the PIDA web site has disappeared and the last stable release was about 3 years ago. It seems like a8 may be the replacement but I’ve not had time to run with it yet.

Others To Look At

As I stated before I’d like to spend some time with a8. It looks like it’s X based only (Linux, BSD, etc..) which is fine for me since most all of my dev is on Linux. I’ve also seen a lot of talk about Ninja IDE and it looks promising though I’m generally not a fan of specific language IDEs since I am not always using the same language.

What Do You Use?

Seriously. If you are doing Python development regularly what are you using and why? What features have you found to be amazing and which ones are overrated marketing dribble? Am I the only guy who continues to go back to vim in this day and age?

Introducing Flask-Track-Usage

A little while ago one of the guys on a project I work one was asking about how many people were using the projects public web service. My first thought was to go grepping through logs. After all, the requests are right there and pretty consumable with a bit of Unix command line magic. But after a little discussion it became clear that would get old after a while. What about a week from now? How about a month or year? Few people want to go run commands and then manually correlate them. This lead to us looking around for some common solutions. The most obvious one was Google Analytics. To be honest I don’t much care about those systems. While that one may not (or may be) intrusive on users I just don’t feel all that comfortable forcing people to be subjected to a third party of a third party unless there is no other good choice. Luckily, being that the metrics are service related, the javascript/cookie/pixel based transaction wouldn’t have worked very well anyway.

So it was off to look at what others have made with a heavy eye towards Flask based solutions so it matched the same framework we were already using. Flask-Analytics came up in a search. The simple design was something I liked but the extension was more so aimed at using cookies to track users through an application while we want to track overall usage. I figured it was time to roll something ourselves and provide it back out to the community if they could use it as well.

Here it is in all it’s simplistic glory: Flask-Track-Usage. It doesn’t use cookies nor javascript and can store the results into any system which you provide a callable or Storage object. There is also FreeGeoIP integration for those what want to track where users are coming from. The code comes with a MongoDB Storage object for those who want to store the content back into their MongoDB. Want to know a bit more of the technical details? Check out the README or the project page. Patches welcome!