Matt Warren

Author: Matt

  • Deploying Flask With Ansible

    Over the last couple weeks I have been getting acquainted with using Ansible to provision and deploy servers. One nice thing with using Ansible over a tool like fabric for deployment is that everything from provisioning, initializing, updating, management, and application deployment can be done with a minimal amount of code.

    There are tips I learned as I got started with Ansible:

    1. install ansible with pip into a virtualenv.  This seemed to fix issues I ran into when having it installed globally on my computer.
    2. create a new directory to work with – it will have all the templates for nginx, supervisord, roles, playbooks, and server lists in it
    3. create an inventory file that lists all the servers you will manage
    4. split things into multiple playbooks – I have a webserver, database server and deploy playbook. having the deploy playbook separate keeps it simple and it will run quickly.
    5. the roles found on ansible-galaxy are almost never exactly what you want, even if they are you’ll likely have to read through the templates to find out what variables can be used to customize them.  In almost all cases it’s good to clone the role into your working directory’s roles subfolder so they are easily accessible and editable.
    6. the idempotent property of ansible playbooks make it easy to procedurally build up your playbook.
    7. it’s often easier to have an open ssh connection to manually see what’s changing on the server, make sure templates are rendering correctly etc.

    my working directory looks like this for a simple single server Flask webapp.

    digital_ocean

    For my deploy playbook things couldn’t be simplier:

    ---
- hosts: webservers
  vars:
    app_name: awesomeapp
    repo_url: https://github.com/mfwarren/secretproject.git
    repo_remote: origin
    repo_version: master
    webapps_dir: /srv/webapps
    virtualenv_root: /srv/webapps/pyenv/versions/latest_v3
  tasks:
    - name: git pull project
      git: repo={{repo_url}} dest={{webapps_dir}}/{{app_name}} version=master
      notify:
        - restart app
    - pip: requirements={{webapps_dir}}/{{app_name}}/requirements.txt virtualenv=latest_v3 executable={{virtualenv_root}}/bin/pip3
    - shell: "{{virtualenv_root}}/bin/python3 manage.py deploy"
      args:
        chdir: "{{webapps_dir}}/{{app_name}}"
  handlers:
    - name: restart app
      supervisorctl: name={{app_name}} state=restarted

    the tasks are very easy to follow, git pull the latest changes, install requirements that might have changed, run the ‘deploy’ command on the app which does DB migrations, and finally kick supervisord to restart the app.

    On the command-line my process for executing this deploy looks like:

    ansible-playbook deploy.yml -i ansible_hosts

    Ansible is a tool for power users. It would be quite difficult to use without experience setting up the servers manually. Unlike Heroku or another PaaS, it will take several hours to learn and configure your first server through Ansible. However that onetime cost should pay for itself with the additional control you have, and server cost savings.

    I certainly would love to see tools created to analyze your project and create a set of playbooks and templates which would create a production quality server setup, similar to how Heroku’s Buildpacks can figure out how to get a Django/Rails/Node.js app running.

  • Another side project

    One of the most influential pieces of code I’ve written (that I use myself every day) is something I’ve talked about before.  It sends me an email alert when I haven’t blogged or written code as often as I want to, and keeps sending alerts until I do the action.

    When I broadened the software out to handle reading books it got me thinking about taking this public.

    I started building a web service over the last couple of weeks to make this a more general productivity tool for anyone to use.  In the coming months my partner Colum and I will launch the web site, and work on a mobile app to go with it.

    A well designed mobile app might be able to attract a few loyal users.

  • Getting Started With Ansible

    Everyone who does web development will probably find themselves doing devOps tasks at some point.  I keep an eye on servers hosting several different projects at different cloud providers and some private servers.  Until recently management of those servers wasn’t difficult enough to warrant using tools to deal with things.

    Ansible is a commandline tool that allows you to manage your servers.  Lets say like me you want to introduce it into an environment with servers already in place.  Here’s how you can get started.

    Install the Ansible commandline:

    brew install ansible

    Create a hosts file (file that lists all the servers you want to manage):

    ansible_hosts:

    [ec2]
ec2-.compute-1.amazonaws.com  ansible_ssh_private_key_file=/PATH/TO/key.pem  ansible_ssh_user=ec2-user
ec2-.compute-1.amazonaws.com  ansible_ssh_private_key_file=/PATH/TO/key.pem  ansible_ssh_user=ec2-user

    Ping all the servers to see if they respond to ssh:

    ansible ec2 -i ansible_hosts -m ping

    Ansible comes with a large library of low level ‘modules’ for doing interesting things on your servers. You can call simple things right from the command line as in the previous example (uses the ping module). If you want to upgrade openssl package on all your (Ubuntu) servers:

    ansible ec2 -i ansible_hosts -m apt -a "name=openssl update_cache=yes" --sudo

    or upgrade all updates to a CentOS server:

    ansible ec2 -i ansible_hosts -m yum -a "name=* state=latest" --sudo

    There are over 200 modules built into the core of Ansible to make interacting with lots of different programs easy – git, shell commands, postgres databases, supervisorctl, django, gem packages. Doing a git checkout of your application across a pool of web servers can be done with a single command line.

    If that was all you could do with Ansible I’d be happy. But that’s really just scratching the surface. Building on these low level modules is what Ansible calls ‘roles’. Roles encompass a series of commands to do a bigger task – like install and configure nginx. You can tie multiple roles together into a ‘playbook’ that can define how to build out your entire infrastructure.

    In my next post I’ll cover how to go from zero to fully deployed web application with Ansible.

  • The Quantified Programmer

    Expanding on the thoughts in my last blog post about measuring opposing indicators I wanted to explore things from a different perspective.

    There is a growing community of people that measure and record as much as they possibly can about themselves.  This quantified Self movement is predominantly focused on health related aspects exemplified by the growing market for activity trackers like FitBit.  The more information you have to track yourself the more you are able to identify longer term trends that would otherwise be twisted by your own fallible perception.

    Programmers strangely do not have much going on for measuring themselves.  Even though it would be quite easy to glean information from the coding output from any programmer.

    I would be quite interested to see a collection of stats about myself.  A daily digest of basic productivity, and charts of longer term trends would be motivating and insightful.  Would lines of code per day increase as I got better at a new language/framework?  Would there be plateaus in my productivity? Could I identify the impact of distractions? How much more productive am I working from home vs. at the office?  There are so many questions that would have valuable answers for me personally in my quest to improve.

    While no measurement is going to be perfect, by measuring myself against myself, for personal insight provides some incentive not to cheat and keeps the relative performance as an informative metric.

    GitHub provides a decent API which could be used to dive in and gather this information.  It would be interesting to launch a service that could mine through a user’s github account to generate a detailed productivity report.

    Unfortunately another project I don’t have the time to build myself.  If someone else builds it though. I would surely be a paying customer.

  • Taking a Machine Learning Course

    I started a Stanford course yesterday on Machine Learning.  It’ll be interesting to update my knowledge since things have changed a great deal in that area since I did an AI class at Waterloo 10 years ago.

    Machine Learning has really taken off in the last couple years as the number of problems and the technical capabilities of AI algorithms has evolved.  In particular services like voice and text recognition are like magic.  Applications like self-driving cars, flying drones, game bots, chat services, marketing segmentation, bio data analysis and astrophysics have exploded with new use cases for this type of approach to software.

    A good machine learning algorithm offers tremendous leverage, since it has the chance to solve problems that people have a hard time defining how to solve them.  A solid self-driving car system could disrupt a trillion dollar industry.  Self piloted drones could create a new trillion dollar industry.  These are algorithms that could impact everyone on the planet.

    So I’m kind of excited to refresh my knowledge and learn a few new things.

  • Measure Opposing Indicators

    All software developers know that measuring a developer’s productivity by counting lines of code written is not an effective way to measure their output.  The correlation between the technical difficulty of a problem and the number of lines of code is not always 1:1.  Which means that one developer can write 10 lines in a day and be very accomplished, while another could copy/paste a solution of 1000+ lines in a day which was trivial.

    On the factory floor managers always should be measuring opposing indicators.  If you’re measuring the volume of output – you also need to measure the quality of output in order to prevent creating an unbounded method for taking advantage.

    For example.  If a manager of Intel processor factory production line focused solely on the number of chips produced each day, and cancelled all quality control checks then production output would likely skyrocket.  In short order, however, the defects would be found by customers, the business would suffer.  On the other extreme, if the manager focused solely on quality and checked every aspect of each chip produced then costs would skyrocket and output would crash.

    The problem with measuring a software developer by the number of lines of code they produce is that there isn’t an opposing measurement of the quality/difficulty of those lines.  Gauging the quality or difficulty of the produced code would likely require peer review, and for that peer to grade the code.  Nobody wants to be the ass that gives their co-workers D- grades on their code so it becomes difficult to get honest scores.

    In my opinion, counting anything is better than counting nothing.  If Bill suddenly goes from committing 100+ lines of code per day down to 10 lines per day it may indicate something that a manager should investigate.  If you weren’t measuring output then it would be impossible to know about or fix anything that Bill was having trouble.

    Authors measure themselves by pages written per day.  Writing 10 pages of crap is better than ending the day without having written anything. At least once things are on paper they can be communicated and others can help edit things to find the gems inside.

  • Make Reading Inevitable

    One of my big goals this year is to read one book every 10 days.  That’s 36 books to read before Christmas.

    Inevitable thinking is about asking one question: “What can I do to make the outcome I want INEVITABLE?”  The phrasing of that question prompts you to think about things differently.  Instead of just stating the goal – I want to read 1 book every 10 days – you are forced to delve deeper to take actions that make sure it is guaranteed to happen.

    In order to keep pace with my reading goal I have leveraged the same tool that has kept me blogging every week for over 5 years and kept my GitHub streak going strong (currently at 84 days in a row!)

    I’m keeping my reading history in a spreadsheet on Google Docs:

    Books_-_Google_Sheets

    Then I have a small script scheduled to run every hour.  It reads this spreadsheet, finds the most recent date I finished a book and if it was more than 10 days ago I get a rather offensive email hounding me to read a book.  I’ll continue to get one email every hour until I finish a book and update the spreadsheet.

    This simple nag has proven effective at changing my behaviour over the long term.  With this system in place I’m confident I’ll hit my reading goal this year.

  • The Most Often Overlooked Programmer Skill

    Programmers often take something for granted that would make them significantly better at their job.  No it’s not communication skills, or management.  The biggest differentiator between the best and average programmer is how often they have to refer to google to solve a problem.

    Any programmer who knows the basics of programming in a C like language feels like they’re competent enough to write Javascript – even if they only have to program it occasionally.  What ends up happening is they stall on every other line of code to crawl stack overflow for solutions.  Despite their lack of skills they can still write complex code and get things done.

    No doubt that Google and Stack Overflow are awesome resources that make it easier to write things you don’t really have the knowledge to do off the top of your head.  Each detour to the search engines eats time.  It takes 10x more time to find a good piece of code on the internet than if the programmer knows how to write it already.

    The skill that most programmers overlook is memorizing the APIs and languages they use.

    When you take away all the crutches we use to help write code by writing with pen and paper, without the help of reference material gaps in your knowledge become brutally apparent.  Try this from time to time.

    Most programmers I know seem to think that picking up a new language or framework can be done in just a few weeks.  My experience is that unless they take deliberate time to read and learn and build expertise the typical result is getting to a passible skill level and then plateauing.

    Do not underestimate the value of depth of knowledge.

  • Don’t Write Code Unless You Use These 11 Tools

    ToolboxDo you want to write code faster, with fewer bugs, and feel confident that what you write is good?

    Over the years, programming has evolved, and many new platforms and tools have been created. Most of the tools aren’t very useful, but a few are.

    Here are the 11 tools that have helped make my programming a bit easier and more successful:

    Tool #1: Linting

    Every language these days has a selection of Linting tools.  These are programs that parse your code to detect and flag best practices.  Linting tools enforce consistency of code style among all those in your team, can detect complexity, unused variables, naming conventions, or a long list of other infractions that help you to keep your code clean.

    A side benefit from Linting tools is that they can hint at things which may just be general improvements.  Enforcing Javascript semi-colons, suggesting modernization of Ruby code or helping keep Python code PEP8 compliant can often trigger you to think twice and maybe re-factor to improve your code structure.

    If you’re code environment doesn’t have linting built in, get a better development environment, or install a plugin. Linting is a great tool for improving your programming skills.

    Tool #2: Dash

    Google and Stack Overflow are great resources for solving quick questions you have about something you need to do but having a local copy of documentation to reference can speed you up tremendously.  Dash is a Mac App that lets you search through documentation for various languages and frameworks quickly and easily.  It copies them locally so it works when you’re offline too.

    Everytime you search online for an answer to a question there are two things that can go wrong:

    1. You get distracted reading articles, browsing things and loose time
    2. You get the answer you want without enough knowledge to understand it, copy/paste and promptly forget how it works.

    Reading real documentation often forces you to learn something new.  Well written documentation will explain things enough to give you a firmer foundation to help you remember it the next time you encounter similar questions.

    Tool #3: A Solid Text Editor/IDE

    I have been using Atom for the last couple months as a coding editor.  I’ve always personally felt that IDEs can often do too much which makes me uncomfortable.  Which ever side of the fence you’re on, one thing is clear.  You need to get yourself a great editor.

    This is the primary tool you use when coding – don’t be afraid to spend a little bit of money to make your life better.  Take the time to truly master this tool.  Write your own plugins, customize the shortcuts, tweak the theme, read a book!

    Tool #4: Vim

    Yes, vim is a text editor, but regardless of your editor of choice you should also know how to use vim.  Especially if you work on servers.  Vim is installed by default on just about every Linux machine, so it’s the workhorse that is very nearly always available for you to use.

    Tool #5: Git

    There are many methods for managing source code versions. Git is the one you should be using.  CVS and SVN are long dead, Mercurial and Baazar lost out to git.  Even if you just are coding locally, without any other developers git is a gift that will someday save your ass.  There is really no excuse not to have your code under source control.

    If you want a GUI for interacting with git – checkout SourceTree.

    Tool #6: Commandline

    A master of the commandline can accomplish a lot of things with relative ease.  One of the nice things about a commandline interface is that if you know how to do something on the command line (as opposed to in a GUI) you know how to automate it or turn it into a script without any additional thought.  With a GUI you know how to do something by moving and clicking the mouse – and it’s unlikely you’ll every investigate how to do that task any faster even if you have to perform it 1000 times.

    Tool #7: GitHub

    You’re using git right?  GitHub makes git better in several ways.  By adding Issues, Wikis, Pages, you can plan your Todos and publish your documentation.  You can easily give access to repos to new people when they join the team, or make things public so anyone can fork your project.  Use Github webhooks to plug in and automate your own workflows.

    Have a public face to yourself and your company. Github is the first place I look when checking out a potential new hire.  Your github profile is the modern extension of your resume.

    Tool #8: Newsletters

    K, this isn’t really a tool, but I think it’s important.  Reading the various Google+ Pages, Reddit subreddits, Twitter, and blogs for the languages, tools and frameworks you use, is simply too much noise to reasonably parse.  I suggest signing up for a weekly newsletter instead.  A much more reasonable weekly digest of the most important new things will keep you up to date, without sucking up all your time.

    Tool #9: Books

    Books are often underrated by software developers.  Unlike blog posts which are often written quickly and published in the first draft.  Books are often written by multiple authors, reviewed by a panel of experts and edited by professionals.  These extra filters helps to lift up the quality of content in a book much higher than you find laying around on the web.

    If knowledge a $20 book gives you saves you 10 minutes then it’s paid for itself.  It’s much more likely that a good book will save you days.  A great book could help you double your efficiency.

    Tool #10: Continuous Integration/Unittests

    I have found that different languages and tools necessitate different levels of unit testing.  Python is relatively explicit and APIs are generally consistent – python code usually behaves the way you expect it to.  Ruby syntax on the other hand is prone to ambiguity, has multiple syntaxes for the same things, is poorly documented and often implicit in how things work.  Even after over a year of full time Ruby development under my belt I trust that any untested code is broken.

    Continuous Integration runs your tests after every commit, or every change you make locally.  By running tests often you can catch regressions quickly before they become an embarrassing bug.

    I recommend using a web service like Travis-CI.  Setting up your own CI Server can be a bit of a pain.

    Tool #11: Practice

    If you hired a coach to teach you how to play tennis, you expect to run drills, learn how to grip, swing, stand and step. You’d hit thousands of balls until the mechanics of how to hold the racket, swing and step become so instinctual that they can be performed without thought.  You would practice several times per week, for 1-2 hours each time just performing basic moves.  It would be months before you actually are ready to play a game.

    Nobody practices anymore.

    I believe that deliberate practice is what differentiates an average developer from a 10x developer.  Practice with the languages, frameworks and libraries you use and the tools you write with.  Every API that you memorize will add up to hours of saved time searching Stack Overflow.

    Conclusion

    Writing code doesn’t have to be hard. There are a ton of free tools out there that can help you write better code faster.

    The 11 tools I mentioned above are the main ones I use on my projects. Give them a shot. They’ll change your coding experience.

    What other tools do you use when coding?

  • Practice, Homeostasis, Resolutions and the Path to Mastery

    It’s that time of year again when everyone is making ambitious new years resolutions.

    I started by re-reading one of my favourite books:

    Mastery is about the journey – an endless one. Taking comfort in the practice, ritual and repetition of the fundamentals while endlessly pushing the edge of your skills and knowledge.

    Something that’s been on my mind has been how to get even better at software development. Several months ago I did a massive purge on my blog subscriptions in order to make room for reading more books. Split with reading time, for the last two months I have made an effort to freecode everyday (my github account is on a 66 day streak).

    Making real changes involves a disruption to your homeostasis. You can expect to encounter forces to bring you back to your existing patterns. Knowing to be on the lookout for this is half the battle. The other is to put in place new checks and balances to create a new homeostasis. My most successful long term changes started with something other than just a personal commitment. Blogging and Github commits have been made permanent by having scripts that email me when I get out of balance. Training for a marathon worked only when I had found a social group that expected me to show up.

    One thing that I have been trying to think more about lately is how to apply practice to software development. If you were taking tennis lessons you might hit thousands of backhands for hours multiple times per week before moving on to a different stroke. If you were learning piano you’d probably spend hours playing scales. If you were learning karate it might take a decade of repeating the same motions to reach black belt. Practice is key.

    I don’t know any software developers that practice their craft in the same way. It’s hard to find any information about other developers taking a similar approach to learning.