Programming

Sorry if this post is overly technical.

I haven’t had a chance to find a useful project to code up yet, but I took some time yesterday to start getting my head wrapped around the development and deployment processes. In particular I’m looking at getting web applications deployed to cloud computing services like Amazon’s EC2 service.

If you’re not familiar with cloud computing is fairly simple. Basically you can request a computer be started for you somewhere in a remote server farm and then be billed by the minute for usage. The nice thing that this allows you do to is to dynamically scale the number of computers you have available for you to use. So for instance, for a one time test you can just start a machine, log on, run some code, collect the results, shut down the machine and discard it. The whole process may only cost you a few pennies to do. Where as purchasing a fresh machine from Dell could take weeks to be delivered, days to set up, space in the server room, and cost several thousand dollars.

At work right now we have quite a large server room (considering the size of the business). Many of the machines however sit idle for much of the day. So many of the CPU cycles get wasted away. I think that scaling the servers horizontally with distributed algorithms would allow us to avoid having to purchase new $30,000 machines every 18 months and instead we could just add a few more blades to the virtual farm.

Anyway back to my project.

What I have been looking at is a number of different solutions for:

1. Organizing code for multiple projects
2. version control of the code
3. managing the required 3rd party library code
4. testing any applications that I write – local and remote testing
5. deploying to a fresh bare computer
6. debugging web applications

The tools that I want to use are:

1. Server side development with Python and Django framework
2. Try out some of the latest HTML5 and CSS3 functionality
3. Client side scripting with Javascript and jQuery
4. Linux servers on Amazon EC2
5. Back to basics – programming with a text editor and command line again

What I have found is that there are a number of clever tools available to manage this kind of project.

1. Virtualenv is a python project that allows you to quickly separate environments for different projects. This allows you to install required libraries, have environment variables set, and versions of the language quickly changed with a single command line call. This allows me to manage 10 projects where one may require version 1.3 of library x while the other requires version 1.4 of library x. It allows me to manage projects that are targeting different versions of python without installing them globally on my computer.
2. distribute and pip are very handy tools for finding and install libraries I need. Using public online repositories I can simply use a command line like `pip install Django` and it will automatically find, download and install it. If there happens to be any compilation steps it will do that for me. The really nice thing about this is that when built into the deploy script I can take a totally bare computer with nothing installed and have it automatically download and install everything I need. If I had to setup 100 identical machines it becomes trivial to do.
3. Fabric is the tool that does this deployment scripting. It connects to the deploy target machine using SSH and will do whatever I need to get it up and running. That could include installing libraries, copying files, installing libraries, setting up apache web servers, installing and configuring databases and running tests.
4. Firebug is probably the coolest tool for debugging the client side of web applications I have ever seen. It allows you to pull up any webpage and dynamically edit the CSS, HTML and Javascript and see the results live right in Firefox. I used it quickly the other day to tweak the theme to my blog.
5. Git/Mercurial I’m kind of torn about which route to take on this. Git seems far more popular and is referenced everywhere I look, but I have already started using Mercurial and really enjoy it. These tools allow me to easily version control my code without having to set up svn servers etc. distributed source code management (DSCM) is the future.
6. Textmate is more than a simple text editor. While I still like using VIM, textmate is already setup and ready to go with lots of nice shortcuts for programming. (I’m actually writing this blog post from textmate)

Yesterday I took some time to find some programming info on iTunes U. If you have never heard of it, it’s basically a category of podcasts where universities have recorded classes and published the videos/audios. You can find decent course material for just about any subject from some of the top universities in the world. Between the content in iTunes and what I could find on YouTube there’s enough how to guides out there to learn how to do everything.

Now I just need a few sample projects to run through to totally wrap my brain around all these tools. Learn by doing, practice and repeat.