Monthly Archives: January 2012

So, Yahoo messed up today. They've messed up other days, too, but this was an especially red-letter day amongst other red-letter days, and this is one that has me ticked off.

For reasons I don't know, Yahoo laid off the highest level of Flickr's customer support, the people that end up filing bugs against the developers and helping the trickier cases get solved for the members. Those guys getting shown the door is as bad as it sounds.

When sites get larger, both in members and staff, the gap tends to grow between the people that build the site and the people that use it. Sometimes this happens with product decisions, but it almost always seems to happen with developers. Our job is to write and ship code to the best of our abilities, though, through no acts of spite or laziness on our parts, our code is not perfect. It's a fundamental nature of current software. We're human, we're imperfect, and we write bugs.

After we write and ship code that probably contains a bug or two (or three), our job is to then write more code, which will also contain bugs. It's a bad cycle.

This means that someone has to be in the middle, as the face of Flickr, acknowledging these mistakes and going to great lengths to fix things. This is often a thankless job, as users just want their problems to go away and developers (usually) don't like to be told they messed up. But they do it for the good, and for the love, of the site. Every bug that gets filed and every support case that gets carefully answered makes the site that much better.

After being a liaison between these two worlds long enough, you end up knowing more than anyone else on the team. When you have millions and millions of users that hit every button and link in combinations you would never dream of, then reporting the “interesting” outcomes of their explorations, these support agents become walking encyclopedias of the ins-and-outs of the site and with Flickr, there are odd edge cases waiting on every page. Having people on your team aware of everything the site does is huge. You literally can't buy that or replace it or outsource it, though it appears that Yahoo thinks it can.

With big sites, not only do you have bugs, but you have outages. These same agents that can recite all the guestpass-viewing conditions and know offhand whether a photo should be visible in Germany, also get to sit on the front lines and explain to users with emotions ranging from impatient to pissed-off that some section of the site will be back as soon as possible. This is not a position to be envied but one they always handled with grace and aplomb.

To be constantly deluged by the requests and demands from stressed users and keep showing up in high spirits day after day demands a special kind of character. Not only do you have the patience of a saint (imagine getting asked the same 3 questions, 50 times a day, every day) but also the tact to work with developers and product folks whose priorities are different from the users, as those things tend to go.

And that's probably the biggest thing that hurts: the users of Flickr lost their major advocates today. At product meetings and developer meetings, it would be these support folks constantly asking, “But what about the users?”

---

On a personal note, Flickr lost several good people today. If you had me name the top 10 Flickr employees that loved the site the most, half of them got handed pink slips today. Working with that entire team was absolutely one of the highlights of my time at Flickr and any other company that has a need for calm, intelligent, and resourceful customer support folks would do well to contact me or any other person that has ties to Flickr to get you an introduction.

To the support folk that are now ex-Flickr, you've got a stupidly strong alumni organization and we know how good you guys are — we're here for you.

---

I don't really know the real purpose of me writing this. I'm always hesitant to write anything good, bad, or otherwise about my past employers, but this one deserves to get called out. Yahoo made a major mistake today and there's no other way to interpret it. I'm mad and this is my soapbox.

Flickr-the-site will be fine but Flickr-the-culture took a huge hit today and those suits in Sunnyvale balancing some column or doing their thousandth “re-org” are completely to blame. I bet they don't even know what they've done and that's probably the worst part of the whole thing.

This is lifted from the README to the github repository.

I decided to take all the blog posts, Twitter messages, and Flickr images I made this year, combine them, typeset them, and then get it printed
in a hard-bound book. I wrote a bit about the reasoning here.

There was a lot of poking and pawing at the scripts I used to create the final product so I thought I’d share them in case someone else could get some use out of them.

Big warning: these are mostly worthless until you change them to fit your project. While all the code here works, and it ended up giving me a decent-looking book, you’ll need to modify it, which is mostly the point. This is your retrospective and thus shouldn’t be a cookie-cutter running of the code I wrote (if that would even work).

I’ll now explain a bit about the pieces:

The Blog Posts

All my blog posts are just flat HTML (via jekyll) so getting my blog onto my PC was already done. You’ll probably need to run some magic incantation of wget or curl to get yours if they’re hosted somewhere else.

TeX, specifically pdflatex, was the workhorse on typesetting it so I needed to get these HTML files into tex format. I ran a find . -name "*html" | xargs -I{} python texify.py {} in my jekyll’s site directory which then ran each of the files through pandoc. Pandoc is a super magic text transformation library that will slurp in most text format and then spit out a transformed version. In this case, I was reading HTML and spitting out .tex files. You can see the command in texify.py.

After I had all these converted tex files, I actually loaded all my files up in vim, made a macro that cleaned out things like header and footer, and then just ran the macro across all the open files. I forgot this magic spell almost as soon as I did it, but bufdo sounds familiar. I’d google something like “vim macro across all open buffers” or something.

Now that I have a directory full of tex files, one file per blog post, you need a master tex file that actually describes the full document, as well as the pointers to all the various tex files to include. This is the book.tex file in this repository. This is mine lifted as-is, so this is what the finished result looks like and should give you a good idea of how to put yours together.

TeX is a frustratingly arcane markup language, but it is extremely powerful and can create beautiful documents. It’s worth it, trust me.

I’ve also included a sample blog post tex file. This post includes a couple of images by includegraphics to give you a heads start on that.

Twitter

To format your Twitter posts, you first need the actual Twitter messages. This is actually hard, if not impossible, if you’re especially prolific.

Twitter famously only allows you to fetch your last 3200 messages. This limit is enforced but on the official website and by the API.

I’ve been running tweetnest on my server for a year or so, mainly because I think it’s pretty, but it turned out to do a whizbang job of archiving as well. Surprise, surprise: this was the source of Twitter messages for my book. I just dumped the table to a text file (via mysqldump) and used that as my source file.

Inside of twitter/tweet_transform.php, you’ll see the reading of this file and then spitting out the tex file, separating the messages by month and then by the day.

There are some positively Nolan-specific things in here. All the dates in Tweetnest (and probably Twitter’s API) return a timestamp for each Tweet using seconds since the epoch. If I only tweeted from San Francisco in all of 2011, getting nice dates would have been easy: just set the timezone at the top of the script and then call it a day. But as it turned out, I climbed on and off airplanes at various locations and at different times. You’ll see a block of code that dynamically sets the timezone according to when I was boarding and de-boarding airplanes.

Another sort of fuzzy, human thing I added to this that you may want to be aware of is that I fudged the edges of what constituted a “day”. Instead of a day being midnight to midnight, I grouped tweets on a 4am boundary. Best I could tell, I never tweeted before 4am after waking up, and never tweeted past 4am by staying up from the night before. This way a day is defined as waking up to going asleep (or passing out, some nights).

This script also runs follows some common URL shorteners so you won’t see any bit.ly or goo.gl links in your permanent archive.

The hard part of getting the Twitter section together is actually getting the tweets together, but once you do that, it’s a breeze.

Flickr

I uploaded about 600 pictures to Flickr this year. I really wanted to display every single picture for the sake of completeness but figuring out a way to that visually was difficult.

I ended up going something like Google’s image search. Stephen Woods was also a major source of inspiration for the layout. This layout lets you plop a lot of images on a page and letting them use their natural dimensions to shoulder out more space as needed.

Instead of forcing tex to layout individual images, or individual rows, I figured it would be easier to create an image that represented the full page and then put that on the page, not unlike the old days of people adding <area> tags to full-page images in the early days of the web.

The flickr/justified.php file is what creates these image files and then the flickr.tex file that includes them all.

I used Aaron Cope’s parallel-flickr as the source of the images. This project conveniently creates an easy-to-query database so I could do something like “give me all the images from Jan 1, 2011, to Dec 31, 2011 ordered by date_taken ascending”. I used the output of this query to select the appropriate images in the correct order and rsynced them to my book’s Flickr directory.

There are a few fuzzy parameters that lets you set things like a maximum row height, and how wide your rows are. Feel free to twiddle these knobs as you see fit.

Conclusion

Nothing about this is drop-in-and-run but there are a lot of gotchas that I came across that might help someone else if they ever decide to tackle a project like this.

The Code

Feel free to browse the code at my github repository.

My mom called to let me know that they had to put Dipstick down today due to the kind of things that happen to cats when they get old. She actually ended up being 14 years old, which was older than I thought she was.

It's a cruel fate that takes away these constant companions, especially ones that lived as quietly as she did, not asking for anymore than a small laundry room, a bowl full of food, and a few pats on the head as we walked by.

So, with that, I'll be hugging Oliver and Lola a bit harder tonight.

I was on a wiki walk when I came across this fascinating, but long gone, summertime event that used to happen inside Yosemite.

So, they used to pour still-burning embers off the top of Glacier Point at nightfall every day during the summer to make, what they called, a “firefall.”

This image is from Flickr user Cliff Stone (his real name!) taken in the summer of 1962 before the event was banned.

The firefall began when people down in the valley would see the embers from the nightly bonfire from the Glacier Point Mountain House kicked off the cliff’s face and then people would start specifically asking for it to happen, as seeing a river of fire coming down the face of cliff was quite the spectacle.

There were various attempts to stop the firefall between its inception in 1872 until its demise in 1968, mainly citing the crushing traffic of the sightseers and the fact that it was a man-made event in a setting that the rangers wanted to be celebrated for its natural beauty.

Even President John F. Kennedy saw it one night on a visit to the park while in office, but as he had to finish a phone call, they delayed it from its usual time of 9pm to 9:30pm.

The ritual usually kicked off at 9pm with a call-and-response, with someone in the valley yelling, “Let the fire fall!” with the response from the top of the mountain with, “The fire falls!”

There are so many things I enjoy about this. Don’t get me wrong, if they tried to start this again, I’d be first in line to protest, but in retrospect of it actually happening, it’s fascinating.

I can almost see the kitschy 1950s postcard saying “Come see the great Yosemite Firefall!” and the hordes of middle America in their station wagons parked on the road at nightfall. And it’s so audacious to think that that they used to allow a hotel to dump burning embers off the top of a mountain, especially when park rangers will definitely ticket you in most national parks for having even small campfires.

Now, people come to see a natural version of this, when the late winter February sun strikes Horsetail Falls just right, and sets the flowing water seemingly aflame. I’d like to see this one day and, in a small way, I’m sad I’ll never get to hear the people yelling and then pouring fire off the high peak, purely for amusement.

As a Christmas gift, Meghan gave me a new razor, one of the “old-fashioned” safety razors, complete with a shaving brush and soap that smells vaguely of tobacco.

I had actually been eyeing this setup for a few months as I’ve been using one of the modern razors with 3-5 blades for years, and I was always disappointed with the quality of the shave, as well as how quickly my coarse facial hair would wear down the blade.

I watched a few YouTube videos on optimal lathering technique with the brush and soap and the proper angle and direction to pull the razor. (I’m sure our grandfathers learned the same way…) And just like any ritual, there are as many ways to do it as there are are people performing it.

So wanting a possibly better shave was a major reason for wanting (and receiving) this shaving setup, but I’d be lying if I said that was the only reason.

Sure, there’s that slightly Mad-Menish flair of dragging a sharp blade across your throat with a chromed-out razor, but for me, it appealed to my love of little rituals.

Up to a year or so ago, the pieces of my day were mostly the same, many of them spent in front of my computer. Like I’d imagine a lot of people that “live” on the Internet, I divided my chunks of time into roughly 15 second slices: read 1 email, switch to Twitter and read the top few most recent messages, switch to the feed reader and skim a dozen headlines, oh look a new Twitter message, ooh more email, and so on and so on. I knew I needed a reprieve from getting that constant stream of endorphins from making all the numbers go down.

The first little ritual I introduced was brewing a cup of coffee and making oatmeal on the stove every morning. Such a little thing became a real meditation. I could only do one thing at a time and I had to pay attention. This was a complete flip from the non-stop information gluttony I usually participated in and this was good.

So this new shaving process in the morning is a similar thing. (And as a side benefit, I’m getting a better shave!)

Another thought, that might be related:

Over the holidays, Meghan and I went and stayed with her parents in a beach house on the Florida Panhandle. Her brother would turn the TV own after our afternoon walks on the beach and around 6pm, Meghan’s mother would ask for him to to change the channel to the news.

Ten years ago, this would have seemed commonplace, but today, the idea of sitting down with the frame of mind of “now, I will consume the news for 30 minutes” is noteworthy. With an always-on, usually-tuned-in Internet connection, there’s no official news time. It’s all the time, whether you like it or not.

Instead of watching the talking heads, I tried to decide which mode is better. With the Internet, I can know of any world event within seconds of it happening. With the TV, I get a daily condensed version of the highlights.

I think I decided that getting the news, in whatever form, once a day in a solid chunk might be the best way. Beyond living in ignorance for a few hours in the day, not knowing normal news events until later usually has very little direct impact on my life.

It’s also another ritual, a devoted time set aside for one purpose. Do I need to be constantly awash in world news? I’d say no. Getting that dose of information in one chunk probably also lets you digest it better and it also lets others (definitely for better or worse) filter out a lot of the noise.

I also realized that a big reason that I would constantly refresh cnn.com or nytimes.com was that I was bored. Why was I bored? Because there was nothing new on those pages. I realized that’s a pretty harsh cycle to be caught up in.

So that’s why I’m taking solace in what little rituals like shaving with a safety razor, or sitting down to watch the evening news, or grinding and pressing a pot of coffee has on my always plugged-in attitude.

And I’m learning that the old saying is true: sometimes you need to stop and smell roses, or, in this case, the tobacco-scented shaving soap.