flask-track-usage 2.0.0 Tutorial

Check out John Dupuy’s video on getting up and running with flask-track-usage 2.0.0!

Advertisements

flask-track-usage 2.0.0

flask-track-usage 2.0.0 has been released! Thanks to all who helped provide patches and test. Note: 2.0.0 is the recommended upgrade version from 1.1.0. 1.1.1 was released for those who are unable to make the needed changes to move to 2.x. You can check out the latest docs over at readthedocs.

The changes include:

  • MANIFEST.in: Add alembic to dist
  • CONTRIBUTORS: Add John Dupuy
  • py3: Fix import issue with summarization
  • .travis: Change mysql driver
  • test: Fix summerize tests for py3
  • travis: Add 3.6
  • docs: Quick fixes
  • README.md: Update docs to rtd
  • Use parens for multilines
  • Update versions to 2.0.0
  • sql: Increase ip_info from 128 to 1024
  • alembic: Upgrade ip_info from 128 to 1024
  • alembic: Support for upgrading SQL schema
  • sql: Create table if it is not present
  • couchdb: Add track_var and username
  • redis: Add track_var and username
  • Adding user_defined variable field to storage
  • Hooks: add new hooks system
  • test: Skip mongoengine if connection can not be made
  • storage: Rename to PrinterWriter
  • output: Add OutputWriter
  • storage: Create base class and Writer
  • requirements: Added six
  • Copyright now a range
  • Add CONTRIBUTORS
  • doc: Add note about py2 and 3
  • py3: Fix most obvious offenders
  • Move mongoengine ref in Travis CI config
  • Update Travis CI config to include mongoengine lib
  • pep8 fixes
  • MongoEngineStorage: updated docs; added get_usage
  • added testing
  • moved MongoEngineStorage to mongo.py
  • doc: Minor updates for a future release
  • Initial support for multiple storage backends
  • Update versions to denote moving towards 2.0.0
  • Added MongoEngineStorage code; adding test next.
  • docs: Update version to 1.1.1
  • release: v1.1.1
  • Updates for freegeoip function
  • test: Update sqlalchemy test for updated flask
  • test: Update mongo test for updated flask
  • test: test_data works with current Flask results
  • travis: Force pymongo version for travis
  • storage: Minor doc and structure updates for new backends.
  • Redis support
  • Added CouchDB integration. (#30)

From Gevent to CherryPy

I’ve been working on a project for the last few months on GitHub called Commissaire along with some other smart folks. Without getting to deep into what the software is supposed to do, just know it’s a REST service which needs to handle some asynchronous tasks. When prototyping the web service I started utilizing gevent for it’s WSGI server and coroutines but, as it turns out, it didn’t end up being the best fit. This is not a post about gevent sucking because it doesn’t suck. gevent is pretty awesome but it’s not for every use case.

The Problem

One of the asynchronous tasks we do in Commissaire utilizes Ansible. We use the Ansible python API to handle part of host bootstrapping of a new host. Under the covers Ansible uses the multiprocessing module when executing it’s work. Specifically, this occurs when the TaskQueueManager starts its run. Under normal circumstances this is no problem but when gevent is in use it’s monkey patching ends up causing some problems. As noted in the post using monkey.patch_all(thread=False, socket=False) can be a solution. What this ends up doing is patching everything except thread and socket. But even this wasn’t enough for us to get past problems we were facing between multiprocessing, gevent, and Ansible. The closest patch we found was to also disable os, subprocess and a few other things making most of gevents great features unavailable. At this point it seemed pretty obvious gevent was not going to be a good fit.

Looking Elsewhere

There are no lack of options when looking for a Python web application server. Here are the requirements that I figured we would need:

Requirements

  • Importable as a library
  • Supports WSGI
  • Supports TLS
  • Active user base
  • Active development
  • Does not require a reverse proxy
  • Does not require greenlets
  • Supports Python 2 and 3

Based on the name of this post you already know we chose CherryPy. It hit all the requirements and came with a few added benefits. The plugin system which allows for calls to be published over an internal bus let’s us decouple our data saving internals (though couples us with CherryPy as it is doing the abstraction). The server is also already available in many Linux distributions at new enough versions. That’s a big boon hoping to have software easily installed via traditional means.

The runner up was Waitress. Unlike CherryPy which assumes you are developing within the CherryPy web framework, Waitress assumes WSGI. Unfortunately, Waitress requires a reverse proxy for TLS. If it had first class support for TLS we would have probably have picked it.

Going back to a more traditional threading server is definitely not as sexy as utilizing greenlets/coroutines but it has provided a consistent result when paired with a multiprocessing worker process and that is what matters.

Porting Time

Porting to a different library can be an annoying task and can feel like busy work. Porting can be even worse when you liked the library in use in the first place as I did (and still do!) with gevent.

Initial porting of main functionality from gevent to CherryPy took roughly four hours. After that, porting it took about another 6 hours to iron out some rough edges followed by updating unit tests. Really, the unit testing updates ended up being more work, in terms of time, than the actual functionality. A lot of that was our fault in how we use mock, but I digress. That’s really not much time!

So What

So far I’m happy with the results. The application functionality works as expected, the request/response speeds are more than acceptable, and CherryPy as a server has been fun to work with. Assuming no crazy corner cases don’t crop up I don’t see use moving off CherryPy anytime soon.

Flask-Track-Usage 1.1.0 Released

A few years ago the initial Flask-Track-Usage release was announced via my blog. At the time I thought I’d probably be the one user. I’m glad to say I was wrong! Today I’m happy to announce the release of Flask-Track-Usage 1.1.0 which sports a number enhancements and bug fixes.

Unfortunately, some changes are not backwards compatible. However, I believe the backwards incompatible changes make the overall experience better. If you would like to stick with the previous version of Flask-Track-Usage make sure to version pin in your requirements file/section:

flask_track_usage==1.0.1

Version 1.1.0 has made changes requested by the community as well as a few bug fixes. These include:

  • Addition of the X-Forwarded-For header as xforwardedfor in storage. Requested by jamylak.
  • Configurable GeoIP endpoint support. Requested by jamylak.
  • Migration from pymongo.Connection to pymongo.MongoClient.
  • Better SQLStorage metadata handling. Requested by gouthambs.
  • SQLStorage implementation redesign. Requested and implemented by gouthambs.
  • Updated documentation for 1.1.0.
  • Better unittesting.

I’d like to thank Gouthaman Balaraman who has been a huge help authoring the SQLStorage based on the SQLAlchemy ORM and providing feedback and support on Flask-Track-Usage design.

As always, please report bugs and feature requests on the GitHub Issues Page.