Shipping ordinary software on time is damned hard. Shipping great software in any time frame is extraordinary. Shipping great software on time is the rarest of earthly delights.
– Jim McCarthy, Dynamics of Software Development

I received my copy of Dynamics of Software Development by Jim McCarthy in the mail today, and this quote was just sitting there on the first page.

I’ve shipped great software on time and I’ve shipped ordinary software late. This inconsistency to a logic-driven mind is maddening. I think I’m fairly decent at writing code, but getting the whole process to a point of repeatable success from a business standpoint is what I’m in the dark on.

Though my computer science education was fantastic in teaching me about logic and theory, the nuts and bolts of software engineering is something I am lacking. There was a software engineering course offered but I opted for the programming language design course, instead. I don’t regret that decision–I love programming languages–but  in the world of waiting customers and deadlines that come much too quickly, I’m having to play catch up.

So for now, it’s time to hit the books.

Yahoo! Graded Browser Support

The Graded Brower Support, or GBS, is the categorization of browsers into different levels that Yahoo! has agreed to support. An A-grade ranking means that the browser has a large enough user base, and that those browsers will receive Yahoo!’s highest level of QA and testing support, while the other levels receive little to none. This doesn’t mean that Yahoo! sites (or sites that use the YUI tools) will be broken in non A-grade browsers, it’s just that they won’t be QAed as thoroughly. Usually these fringe browsers like Chrome and Opera are more compliant to standards, and work automatically with valid HTML, CSS, and JavaScript. They don’t get the A-grade yet, mainly due to their small user base.

IE6 is still an A-grade browser, while they have plans to move browsers like Firefox 3.0 to C-grade X-grade status by the end of the year. This is amazing. For comparison, Firefox 3.0 came out about a year ago (ie, June 17, 2008), while IE6 came out in August 2001. When IE6 was released, the WTC towers were still standing, no one had seen any of Lord of the Rings or Harry Potter movies, George Harrison was still alive, and it remained to be seen what kind of president George W. Bush was going to be. If IE6 was a human, it would be starting the 3rd grade soon. To put it in web terms, this was 4 years before the first video was uploaded to YouTube, 2.5 years before anyone Facebooked anyone, and even 3 years before Google went public. The browser’s been out awhile.

2001: An Ancient Web Browser

The fact that Yahoo! has still shown a commitment to ensure that IE6 users get the best experience that Yahoo! can give them means two things — there are a still a lot of IE6 users and the company still makes money off of them. This study done 5 years ago (though still 3 years after IE6 came out) reported that IE users were 4 times more likely to click an online ad than Firefox users. Since Yahoo! is still courting to IE6 users, this must mean that those users are still clicking those ads.

At some point, Yahoo! and the web as a whole will have to drop their A-grade support for IE6. HTML5 and CSS3 are now ready to be consumed with Firefox 3.5, Safari 4, and Chrome 2.0 all available for download. This doesn’t mean that developers and designers have to completely break the web for IE6 users, but maybe these users don’t get the full experience that the new browsers can deliver. We can do what we can for the IE6 users, but also allow the newer browsers to take advantage of the new, exciting things.

I like the analogy that YUI has on their browser support page comparing the different browsers to televisions. You have TVs that handle the incoming signal differently, starting with hand-cranked survival radios that only pick up the audio, to black-and-white TVs, all the way to the 1080p Hi-Defs with millions of colors and surround sound. The broadcast stations send out the signal and your TV handles it as best as it can. No one is surprised when a black-and-white TV doesn’t have color and IE6 users shouldn’t be surprised that their favorite website doesn’t look the same as when seen in Firefox. As stated on the GBS page, “[s]upport does not mean that everybody gets the same thing.” This is a hard fact to digest for companies with a web presence and designers that believe in one true version of their art.

Hi-Def, 60's style.

So, why don’t IE6 users upgrade their browsers? Simple — they don’t know what a browser is. This video asks random people the simple question “what is a browser?”. The people they ask get the answer (sometimes hilariously) wrong, but these people are not stupid. It’s just that what browser they use is not important to them. All they care about are the sites they can access with it. And you know what, they’re 100% right. It is just about the web. The reason that web developers and designers are so passionate about browsers is that we are trying our damnedest to make the most forward-thinking, interactive, immersive sites for these people, but we can’t. IE6 — and its large market share — won’t let us.

Good news though is that I think this will change soon and IE6 will be nudged to the door by its own creator, Microsoft. The reason that people still use IE6 is because it was already installed on their Windows machines, so the browser changes when Microsoft says it does. Microsoft will soon see what Google saw when Google decided to build Chrome. Google owns the web, and Microsoft owns the browser. Google wanted a piece of that browser pie, as it meant that more people would be enjoying the web in its full capacity, so they created a browser. Microsoft saw this plan and realized if they wanted a slice of the web market they were going to need to put a real contender  into the search market (sorry, MSN), and thus Bing was born.

If Bing takes off, and I honestly think it will, the incentive will be there for Microsoft to really encourage their users to take advantage of the newest version of IE, and finally put IE6 out to pasture.

* Few small edits for clarity.

I found this printed out on Hacker News, and I think there is a lot of truth in it about the perceptions of software and its development.

The problem with IT, generally, is lack of respect. Because it’s something that looks easy and looks like something anyone could do, people don’t respect experience and talent.

For some reason this just doesn’t work in other fields. While everyone feels they can put a sticking plaster on a cut, they go to a doctor for more serious stuff. Any fool can build a garden wall, but if they want a tower block one hires an architect and a construction company and lets them do decisions.

Whereas, it’s quite common in IT for micromanagers to overrule the professionals.

It doesn’t happen in building. Managers simply aren’t allowed to say things like “Oh, I don’t think we should RSJs in those load bearing walls”. If the architect says they’re needed, then they go in. We also don’t hire doctors by picking the tools they’ll use. When you want a doctor to take out an appendix, you want a qualified general surgeon. You don’t ask them whether they prefer a #3 or a #4 handle on their scalpels.

And yet it’s pretty common to see projects which have picked a technology first (“Oh we’re going to use J2EE for this”) and THEN hire the team around that (“Wanted, tech arch, 5 years exp with J2EE, C , C++, JAVA, Perforce, Apache, Perl, Python.”) and only at that point produce the spec. And then if the job isn’t one the technology is suited for, that ends up being the developers fault and they’re the ones who work overtime to put that square peg in the round hole.

It’s scarcely surprising that a) almost nothing works properly, b) there’s constant chaos and that therefore c) almost no-one finds their work rewarding in ways other than money.

I freely admit that a lot of my time, I feel like some sort of thief. I sit in projects which are doomed. They’re more doomed than a plane that’s lost both wings and is on fire. I see the people around the project running about setting more of it on fire as it hurtles towards the ocean and removing more control surfaces and making it worse. And I watch developers switch in and out of seats as we trail wreckage and smoke down. And it’s always been like this. The project is a free-fall disaster before I join and after I leave and there’s just nothing, nothing, nothing I can do to rescue it at that point in time. And yet I’m being paid to be there.

And I feel bad about that. I feel that I ought to be being paid to achieve something. But usually I’m just being paid to sit in a seat in a vehicle performing a ballistic trajectory straight into its crash site. And even if I quit then the next thing will be the same and whoever takes my place in this seat will be in the same situation.

And this is not how I wanted to work. I wanted to build things that people wanted. Not turn up at an aircrash and fill a seat in it for a while. And while it’s financial fulfilling, it’s not very emotionally fulfilling.

Most bridges don’t fall down. Most patients don’t die. Most IT projects are a failure in one way or another. It’s like being a surgeon back in the days before anaesthetics or antibiotics. Most of our patients die… and that’s got to be bad for morale.

And I can’t help but think that this really has its roots in the fact that people think software is simple enough that they can exercise control over it at a level they ought not to be and that they also don’t want to pay what it actually costs. Architects generally don’t compete on price and you don’t get to tell architects that their idea of how strong structural steel is is an underestimate you can feel you can ignore in this project. But people time and time and time again pick the cheapest option for software, design it like they’d design a tin shack and then act surprised when the end results turns out to be flimsy tin shack instead of a tower block.

Originally from http://discuss.joelonsoftware.com/default.asp?joel.3.762361.7, it appears.

First, the question:

In a certain survey of the work of chemical research workers, it was found, on the basis of extensive data, that on average each man required no fume cupboard for 60 per cent of his time, one cupboard for 30 per cent and two cupboards for 10 per cent; three or more were never required. If a group of four chemists worked independently  of one another, how many fume cupboards should be availabe in order to provode adequate facilities for at least 95 per cent of the time?

My line of thinking to solve this was to find every combination of the 4 chemists needing 0, 1, or 2 cupboards, the probability of each of those combinations happening, and finally summing up the probability of all the hoods needed.

For example, out of the 81 different possible combinations of cupboards required (3 * 3 * 3* 3, with the 3 coming from 0, 1, or 2 hoods needed), there is only one way where 0 hoods are needed in total and this is where all 4 chemists need no cupboards. Following this, there are 4 ways to have 1 hood total be required, with each of the chemists exclusively requiring a cupboard and the other three needing none (1, 0, 0, 0 & 0, 1, 0, 0 & 0, 0, 1, 0 & 0, 0, 0, 1).

So having one cupboard covers the probability of needing no cupbards amongst the 4 PLUS the probability of needing 1 cupboard amongst the 4.

I first did this problem long-handed, figuring out the probability of 0, 1, 2, and so on cupboards until I got to a sum that had a probability > 0.95. I was making a simple arithmetic error (as usual) and my answer was not matching up with what was in the back of the book, so I thought I would write a simple program to calculate the answer since I was confident in what I was trying to do, but was just having trouble multiplying and adding.

Here’s the Python script I wrote (note: you need Python >= 2.6 as I use itertools.product to recreate all the combinations of cupboards needed).

The output is the summing of each of the probability of needing 0 cupboards + 1 cupboard + 2 cupboards and so on. The line with the probablity greater than 0.95 is the answer. In this the case, the answer was 4, which would cover the chemists’ needs 95.85 percent of the time.

from collections import defaultdict
import itertools

probs = {
    '0': 0.6,
    '1': 0.3,
    '2': 0.1
}

trials = itertools.product('012', repeat=4)

totals = defaultdict(float)

for trial in trials:
    #how many hoods needed in this trial
    trial_sum = sum(map(lambda x: int(x), trial))

    #figure probability of exact trial occurring
    total_prob = 1.0
    for item in trial:
        total_prob *= probs[item]

    #add probability of trial to total prob for this number of hoods needed
    totals[trial_sum] += total_prob

#print out all probabilities
keys = totals.keys()
keys.sort()

running_prob = 0.0

for i in keys:
    running_prob += totals[i]
    print i, running_prob * 100

I wrote my first Twitter widget. It is the ‘Latest Tweet’ widget on the far right column of the page that uses Twitter’s public JSONP API to pull in my last Twitter update.

The only thing even mildly interesting is that it has a couple of regexes that finds any ‘@’ names and links them up and (naively) hooks up any hyperlinks as well. These weren’t difficult, but they always take a little bit of tinkering to get right.

More excitingly, I think, is that I decided to push it out to github under a BSD license. This is technically my first open source software, minor as it is.

Since SICP runs close to 600 printed pages and approximately 40 HTML files , I’d rather not read it in my browser and printing it on the home printer is not really an option. I decided that using Lulu might be a workable solution.

Lulu takes PDFs so step one was to convert the SICP website to one big PDF.

First, I used wget to mirror the site. Now that I had all the files, I wanted to clean them up a little. Every single page had the previous and next links at the bottom and this was obviously not needed when the pages are in physical form. I ran the following sed command to remove these lines:

sed -i "/\[Go to/d" *html

The next step was to convert the HTML to PDF. I used htmldoc for this particular task.  First, I put all the names of the HTML files in one text file, on one line, and in the correct order. I called this file “all_files.txt”. The htmldoc command I used to convert to PDF is the following:

htmldoc -f sicp.pdf --webpage --left .75in --right .75in `cat all_files.txt`

I then uploaded this file up to Lulu and designed my (very) simple cover. I made it clear on the back  cover text that I was printing this book under the rights granted by the aforementioned license and would receive no profit from this book with a link back to the original source. I’m not a lawyer so I hope that covers all bases.

Lulu has a convenient feature that will let you do a private printing. I could probably make  this book public, setting my profit to zero, and even though that would be covered under the license, it still feels strange to do.

I am very curious how this book will turn out. The Lulu process was actually fun and if this turns out well, I could see myself using the service again. Once I get the book, I’ll post my reviews of the service and possibly some pictures of the final product. Nonetheless, I’m excited to get a print version of this book for a much-reduced price.

Edited to add:
Here’s the download for the PDF: sicp.pdf

I decided to fix this for myself using a ridiculously short Greasemonkey script, using the window object’s “onbeforeunload” function to make sure I actually meant to close the tab/window.

Here it is for consumption. This is obviously not a work of genius or of any considerable effort, just a fix for my own absentmindedness.

Updated:
I added a little more functionality to the script. There is now a button that will show up in the top right corner that will let you turn off and on the close prompt so that you can browse the last.fm without getting the ‘Are you sure?’ prompt on every page load.

For my day job, I’m a PHP and Javascript developer, enjoying the latter more the former, and for personal scripts and side projects, I’ve been a Python guy for years.

I wanted something that was a departure from these more imperative languages which obviously lends itself to the functional languages. At the same time, I wanted something that wasn’t purely academic, but I could build real-world software with it that other people could easily use.

I’ve been fascinated with Lisp and its S-expressions and the idea of code being data and vice versa. At the same time the speedy, static-typed languages like OCaml and Haskell were very intriguing.

After playing with OCaml for a week or two and also working through the first few chapters of Real World Haskell and watching the entertaining Simon Peyton-Jones,  I enjoyed them, but I wasn’t infatuated with them. No magic spark, I guess you could say. I think it comes down to me being more inclined to the dynamically-typed languages which seems to cut down one more barrier between me and the implementation of my code.

So ruling out the OCaml and Haskell brand of typed, functional languages, I found myself working my way back to Lisp. Surveying the Lisp scene, I found myself confronted with quite a few choices of Lisp implementations. I decided to go with Clojure, as from my readings, it seems to be a nice Lisp, even coming from the older Lisp guys and you get the almost endless number of  libraries of Java, as well as  the speedy JVM that runs on all the major platforms. I’m sold.

I’ve played with SBCL going through bits and pieces of Practical Common Lisp and I really enjoy the expressiveness and the speed of going from the idea of what you want to do to actually seeing it run. Stuart Holloway (of nearby Chapel Hill) has written a series of blog posts porting pieces of the PCL code to Clojure which should serve as a nice introductions as well as the screencasts.

I’ll post updates on new discoveries I make, and any software I decide to work on.

I’d eventually like to start contributing code, but I’d like to get used to the community first and see how things are done. With such an intentionally lead codebase, contributing actual code might be a challenge, so helping people out in the channel might be the best way to help.

Next up, I’d also like to help out with django by doing the same. With a more extensive codebase where size is not as big of as concern as it is in jQuery, getting a patch in or two would not be as difficult I would imagine.

I opted for the Ubuntu 7.10 256 MB slice since that was what the sites were priorly hosted on. I decided that I wanted to do things by the book so I set up a lighttpd server that served my media straight up and funneled all Django requests through Apache/mod_python. I couldn’t find an exact way to this previously published so I thought I would add a few code snippets to help others looking to do the same.

First off, I got the Apache/mod_python setup working. This was pretty much a cut and paste from the deployment docs found in the Django book. Nothing tricky there. By default, Apache runs on port 80, but that will be changed later on.

The next step was to get lighttpd working properly. The relevant snippet is below:

$HTTP["host"] =~ "^(www.)?example.com" {
    $HTTP["url"] !~ "^/(public|media)/" {
        proxy.server = ( "" =>
            ( (
                "host" => "127.0.0.1",
                "port" => 81
            ) )
        )
    }
}

Broken down, this says if we get a request for example.com and the URL does not contain one of our media directories (read: a Django request), proxy it to port 81. This way lighttpd will serve up all the static files and then redirect the Django stuff to our Apache instance that will be listening on port 81.

After that, change the ports.conf file for your Apache instance to listen on port 81, and the “server.port” variable in your lighttpd.conf to listen on port 80, and then restart both servers. If everything went correctly, you should now have a 2 servers doing what they do best and have a happy machine to boot.

I did add a couple of lines to my Apache conf to get better performance.

First off, I turned off KeepAlive as suggested by Jacob Kaplan-Moss. KeepAlive is useful if you are using Apache to serve up several files over one TCP connection, like multiple images on a page load. Since on every page load we are making just one request to Apache (for the HTML itself) and lighttpd is handling all of the static serving (which it is very good at it), KeepAlive helps us none and actually hurts us as Apache will quickly become RAM hungry holding on to your full Django code when there is no need for it.

Second tuning measure was to bump MaxRequestsPerChild up. I set mine at 500 and saw a big RAM usage drop. This way since these are fairly RAM-heavy Apache processes, they’ll get cleaned up once they reach a certain number.

Overall, I have been very pleased with slicehost and the Django book was very helpful in getting everything up and running.