tjtestharness - a language-agnostic DVCS unit-test/continuous integration tool

I’ve been wanting to have a way to visualize which unit tests(or sets of them) passed for a given commit, if for no other reason than the sense of accomplishment from watching boxes turn yellow, then green, as they pass tests. The trouble is, I write code in a lot of different languages for a lot of different projects. I also don’t want to bother with running unit tests individually–I want them to run as I perform commits.

My main project right now is working on a fast quadratic programming(QP) solver for series of related QPs, and in turn using that QP solver as the basis for a sequential quadratic programming method with a penalty globalization. It’s working okay, but there’s a lot of levels to the project, and it’s easy to make a change in one place in the code that has far-reaching effects on other parts. It’s also got a very complicated structure for testing, because of the nature of nonlinear programming. Testing runs something like

• First, need to check the most basic version of the QP solver itself, and how robustly it solves the most basic QPs.
• Next we start adding in bells and whistles: We use a couple different types of numerical algorithm approaches.
• There are also two functionally identical but yet distinct approaches to actually formulating the QP.
• A layer up from all of the QP solver, we’re simultaneously building onto a nice penalty SQP method–which itself has many options and tweakable parameters.
• A layer up from that, we’re testing the penalty SQP method on many different nonlinear programs. They might require different settings in terms of tolerance parameters(standard tolerance might be difficult to reach in a given problem, for example.)
• A layer up from that, we’re also testing the penalty SQP method on mixed integer nonlinear programs. This involves solving many, many NLPs.

So there’s a lot of levels and layers and levers and settings. What was happening is that I would change some parameter and eventually it’d get checked into the repository, and that wouldn’t play nice with other codes.

I realized that I wanted some unit testing to automate this testing process, and document how it should be run, and soforth. But I also wanted the video game-style insta-rewards vibe to it, along the lines of the [albeit less professional.](http://www.panic.com/blog/2010/03/the-panic-status-board/ Panic Status Board) I also didn’t want to spend a ton of time on it. I planned out the interface and databases over some beers with my roommate one night and wrote the entire thing in Django the next evening. I wanted it to run automatically on every repository commit. I also wanted it to try to make some sense out of Python, MATLAB, and C++, since I use all three extensively. It should also actually be able to build a project from scratch, and every build should run from nothing–this enforces the rule that the repository is self as self-contained as possible. Finally, I also wanted to be able to look back at the actual run log of that unit test and see how or why it failed(and also have instructions on which repo/commit to checkout to see the bug in action).

My basic idea was that I wanted some program that kept a database of projects I’m working on, displayed some commits from those projects, showed the status of any test_*.{exe,mat,py} commands it could run. Okay, so I need a table of projects, one of commits, and another of tests. Originally I built it that way, but I wasn’t sure how to get columns uniformly figured out. So I decided to split the ‘test’ table into ‘Tests’ and ‘TestResults’, which worked out nicely.

The other tricky part was figuring out how I should add commits to the database, and how tests should get run. I ended up writing a fairly nice script for the commits to put in .git/hooks/post-commit, though it requires the kludgy ‘git push origin master’ in the post-commit as well. This is causing a bit of trouble with modifying commits(ie, it breaks the other script and causes extra merges/commits.) The not-so-interesting side was the one that peeks into the database, grabs the ‘test’ (which is basically just a shell-script to run), and runs it. It also pushes the data back to the database when it’s done.

Django is a really amazing framework. The entire coding part of the project went together in about 8 hours. I’m still not happy with it. I’ve got a laundry list of stuff to add here:

• check for executable test_. (this is basically done, along with subdirectory test checking)
• it’d be nice to move the actual unit testing to a server instead of my laptop. But maybe it’s safer to not have the interface EVER web-pointing.
• Big one: time/datestamp in the commit rows, to tell when you did it.
• button to undisplay old commits, or a limit of how many commits show on the main page. This could get out of hand REALLY fast.
• collapse commit sha1 (did this, should add: re-expand commit sha1’s)
• button to rerun tests(did this).

If you’re interested, check it out at code.traviscj.com’s tjtestharness.