Three Levels of Developer Productivity

If you work on developer productivity, you sit in an unique spot. You are often given a broad mandate — “make the engineers more productive”-and typically little guidance on what the word ‘productive’ means. (Spoiler: There’s no one right answer here.)

To start to give some shape to the job, in my experience, there are two important categories of problems you are solving. There are other categories, of course, but these two feature prominently and will drive much of the work.

The first problem is one of engineering morale. The tools engineers use for their work factor greatly into the satisfaction they get from their jobs. If you place too high of a hurdle between an engineer making a change and them seeing the impact of that change, or if you make the risk of making those changes too great, you will sap people’s satisfaction. People do quit over slow build times and people do quit over dangerous deploys.

There’s also a breadth of research that show that teams that have fast feedback loops are higher performing. While this research may speak to many, what I’ve found to be universally resonant is the ethos of “don’t frustrate engineers with bad tools and processes.” Nearly everyone can get behind that.

The second problem, in coarse terms, is engineering return on investment. There is a good chance that one of your company’s biggest expenditures is engineering headcount. And every new feature shipped and every bug fixed is generally gated by an engineer doing something. Therefore, engineering cycles are one of the most debated, negotiated and protected resource in any tech company and the bulk of processes have their roots growing out from the perceived scarcity of those engineering cycles. There will always be a natural pressure, from all angles of the business, to get more from less, to speed engineering up, to help engineering do “more” and whether said directly or not, part of your job is to understand this and make improvements here.

If you’ve been around larger organizations, you’ll quickly see that some of these productivity problems can be solved with engineering elbow grease, but it’s not hard to see that engineering as a whole is often slowed down by the sheer coordination overhead that comes naturally to complex organizations running complex technical systems.

The challenge is that improving build times will make a small dent if any on coordination costs (unless you are starting on one extreme and moving it another extreme), leaving the “engineering is the bottleneck” problem largely unaddressed. But, when you start to focus on the second problem, the work isn’t as easily understandable to others.

To make the work more understandable, a framework I’ve had some success with is to explicitly put developer productivity efforts under 1 of 3 different banners: individual productivity, team productivity and team-of-teams productivity.

Individual productivity: In my experience, this category of work is what is typically seen as the core duty of developer productivity teams. This work includes efforts like speeding up build times, reducing test flakiness, increasing deploy cadence, making debugging easier and so on. The primary focus is on the individual developer and moving the code they author through the pipeline faster and more reliably.

There is tremendous value in this category. If you can take a routine process that every engineer is involved in multiple times per day and speed it up by 10%, that’s a huge number of valuable engineering cycles that are freed up, making a scarce resource that much less scarce, speeding up valuable feedback loops and making engineers enjoy their jobs a bit more.

And, at some point, some of these processes hit a phase change where because something has become so cheap to do, it encourages a whole lot more of that thing to happen. Think of the difference between a full build taking a day and that same build taking 10 minutes, or a weekly deploy becoming on-demand. The entire economics of your engineering processes change and it has knock-on impact across the business.

Team productivity: This category focuses on improving production by focusing on the flow of work between engineers, and usually engineers within the same team.

This is the realm of team processes: how does code get reviewed, how many projects can one team undertake, how does a backlog get managed and so on. This is where words like “agile” and “scrum” get mentioned.

In my experience, individual teams often have a lot of leeway in how they manage themselves. Where bad tales arise is often from a top-down mandate to “become agile.” There is also value here but it is riskier and efforts here will falter unless the most senior leadership is actively involved, you have a network of coaches throughout the org and you are willing to take a short term (eg, 3 to 6 months) dip in productivity while teams learn to operate differently and tweak their processes. Lining up and sustaining these preconditions is tough and there is no one weird trick to pursue here.

One approach to this level of productivity is to focus on the handful of teams that are recognized as a shared resource (eg, SRE, centralized tools teams, platform teams) and are on the critical path for some key projects and then help improve how they take in, prioritize and execute their work. You’ll likely get a large percentage of the value of “becoming agile” mandate while not actually having to spend the time and effort to make every team implement Scrum, as an example. Your org likely has a few critical teams that are the bottlenecks or dependencies on many efforts and there is high leverage in improving those specific teams’ processes.

Team-of-teams productivity: Problems here possess the delightfully confounding combination of being harder to see, requiring a delicate touch, are usually seen as “not your job” and are notoriously hard to measure. Yet, in my opinion and experience, making headway on these problems is where real leverage lies, not just for the engineers but for the whole business.

This level is the realm of quarterly/yearly planning processes, org design, platform migrations and other processes that are expensive, hard to change and slow. Simply because these things are expensive, hard to change and slow make any improvements in efficiency here incredibly valuable.

One example: Migrations are a fact of life of larger and older orgs. At any point, there are likely many migrations any individual team could be involved in, or that the organization would like to be doing. This presents a tradeoff: is it better to focus on one migration at a time, or to spread your efforts across many migrations? In general, migrations have little value mid-migration — you have the overhead of living in two worlds simultaneously plus the overhead of the migration itself. So reducing in-progress time should be a priority. This leads naturally to focusing teams on a single migration, which reduces overhead, reduces technical complexity, reduces status checks and reduces context switches, which are all good things.

So in that example, a team-of-teams productivity improvement would be to cap the number of in-flight migrations occurring at any one time.

The ways to make improvements at this level is by having relationships with pivotal engineers and managers, getting involved at the right time (eg, quarterly or yearly planning) and making a case for a productivity-improvement constraint when stakes are fairly high and involve influential people with differing goals and motivations. Nudges and tweaks to existing processes typically work much better than large campaigns.

When the “CI/CD person” shows up to suggest improvements to quarterly planning processes, using the framing of improving team-of-teams productivity, with a focus on overall program and organizational effectiveness to help the whole organization deliver more value sooner, helps explain the motivation behind the suggestions.

– — — –

Working on developer productivity can be hugely rewarding and there is a lot of value that comes from working on it at different levels, even outside the typical domains of automation and CI/CD.

Rules over Safety

Last year, MUNI removed a few seats from the front of its standard electric buses by permanently locking them upright. They did this due to safety reasons: “these seats do not have a barrier in front of them to protect a person from falling in the event of a sudden stop or collision.” (link)

It’s easy to read through the lines to glean that the bus manafacturers and likely the San Francisco Municipal Transportation Agency have done this to protect themselves from litigation in the case of an accident, and not in the true spirit of making the bus safer. The bus makers have said: “Here’s the way you should use our buses. If you color outside these lines, it’s on you.”

What happens in reality? For those who have an even passing knowledge of SF transit knows that the buses get incredibly crowded — if there’s a place to stand where a seat isn’t, someone will stand there. Now, instead of someone sitting on a seat where they might be thrown forward, that person is now standing in a spot where they may be thrown forward, and due to centers-of-gravity and torque and all those other fun Newtonian physics concepts, it is more dangerous. It is a common sight to see someone half-sitting, half-leaning on the raised seat, poking at their phone. A sudden stop would send this person over top a row of sitting people versus hitting the single person in front of them.

This is a good example of a policy enacted by fear of litigation over safety but still publicized as being done “for your safety.”

Hello, Slack

Two weeks ago, I started at Tiny Speck as their engineering manager, working on Slack. Slack is getting bigger in almost every way that matters and I'm excited about getting to take part in it.

So what does taking this particular role at this particular company mean to me? It means I'm working for the same people that built the company that hired me that moved me from the east coast to San Francisco. It means I'm managing at a company that shaped most of my thoughts about software development and how to build products that people love. It also means I get to take those experiences and principles and help build frameworks where we keep doing those good things but at a different scale than we are all used to.

So, is this a little scary? You betcha. Am I excited? Oh yeah.

A Tahoe Trip

Meghan and I got back from Lake Tahoe this afternoon. We shared a house in Incline Village, Nevada with her parents and three brothers who had flown out from Tennessee. It was a relaxing week of eating, drinking, skiing, reading, and catching up with the in-laws.



A Train Adventure in Italy

A couple of months ago, my wife and I went on a two week vacation to Europe for our 5th wedding anniversary, splitting our time between Paris, Rome, and Cinque Terre. The first leg of our trip was in Paris where we spent a few days exploring the city. From there the plan was to take the TGV to Milan so we could see the Alps on our way to Vernazza.

Our destination after Paris was Milan–specifically the Porta Garibaldi station. This station is the main hub for the high-speed, long-distance trains like TGV. To get to Vernazza, once in Milan we had to transfer to another station, Milano Centrale. These two stations are less than a mile apart but with our luggage and general unfamiliarity with all things Italy, we gave ourselves over an hour to get from one station to the other. In the worst case, we could grab a cab. (Below is a map of Milan with P. Garibaldi on the west and Centrale under the marker.)

View Larger Map

In my research of planning this train ride, I discovered that you could buy full trips in advance, so I did. I had tickets that would take us from Paris to Milan, Milan to Sestri Levante, and from there onto Vernazza. I was confident, prepared, and had no idea I had already messed up.

The morning of the trip, we walked the mile from our apartment in the Bastille to Gare de Lyon. Gare de Lyon is a huge transit switching station, serving TGV, RER, as well as the Paris metro. If you are a train in Paris, this is your Champs-Élysées.

Gare de Lyon

Our TGV train left on time and we were soon zooming through the French countryside. We sat in front of an older Australian couple, who both seemed to have generic digestive issues, belching loudly and passing gas, much to the dismay of the coutured French woman across the aisle from us, who would occasionally spritz her Chanel No. 5 in their direction which they, of course, didn’t notice.

The trip proceeded without event until we reached a small town at the foot of the mountains. Most of the stops we made were less than five minutes. This one dragged on for about ten minutes and there was a sense in the air that something wasn’t right. After about 15 minutes, a young, dark-haired girl walked into our car, crying and being escorted by one of the train attendants. They were speaking French but I got the sense she was looking for someone or something was wrong with her ticket. She eventually made her way out to the train platform and we left.

We trudged slowly up, and often through, the Alps. Eventually, one of the tunnels we went through popped us out into Italy. Upon exit of the tunnel, the train stopped. And we stayed stopped.

After about 20 minutes sitting less than a half a mile inside the Italian border, the conductor finally broke silence and announced that there was a medical emergency with one of our fellow travelers and we were waiting for an ambulanza. We sat for another 10-15 minutes while the medical staff attended to the sick person. After that was dealt with, the train started moving. The conductor said, through the PA that “we were delayed 16 minutes.” That wasn’t too bad. We still had about an hour to switch train stations.

Turns out “sixteen” and “sixty” sound approximately the same when said through a train’s PA system, spoken by a French train conductor whose second language was Italian and was giving a valiant attempt at English. I realized my mishearing after our arrival time came–and went–and we were in the middle of a giant field with no Milan in sight.

Italian countryside_46
Not Milan. Photo courtesy of prof50000 on Flickr.

We arrived at Milano Porta Garibaldi with about 20 minutes to spare before our Milano Centrale train left. I switched to Optimistic Mode (aka: Denial Mode) and laced up my shoes, imbued a sense of urgency to my wife, and then sprinted through a foreign station in a foreign land to a foreign taxi stand. If stars aligned, and Lady Travel Luck smiled on us, we’d be resting comfortably in a Trenitalia train cruising towards our coastal town apartment in just a few minutes.

But, there were no cabs. There were signs outside the door pointing towards where taxis normally should be but now pointed to an empty stretch of asphalt. We waited for a few more minutes but I knew our window had closed and it was time to figure out Plan B.

My concern now switched to not just finding a new set of tickets into Vernazza but to find a way to get there that day. Vernazza, being a small town off the main line, didn’t have regular train service after 8pm and it stopped earlier than most stations. In my original booking, I knew there were only two or three trains after ours. Time was ticking.

My first priority was to buy tickets from where we were to where we wanted to be as quickly as possible and then see about getting a refund later on. I went to one of the self-serve kiosks, and after assistance from a 7 year-old girl and a college-aged art student who knew a little English, I gave another 80 euros to the Italian train system.

Now that I had tickets that ensured that we wouldn’t be sleeping in Milan, I queued up to speak to one of the station’s service agents about a refund for the missed train tickets. The line moved slowly as there were two agents, one of which seemed to know half-a-dozen languages and enjoyed to talk and the other that spoke what I would call “Gruff Train Agent Italian.” I managed to get the latter. I gave him both our original tickets and the ones I had just purchased, in hopes that it was evident what went down. He focused primarily on the new ones and indicated that I had plenty of time to catch that train. I kept signaling through the glass towards the original tickets but to no avail. I looked longingly at the multilingual agent and then shuffled back to my frazzled wife, beaten by the Italian train system.

Our new tickets left from that station and took us to yet another Milan train station. At this station, we found our platform with plenty of time to spare. We just had to stand there and our train would arrive. Or so I thought.

Literally two minutes before our train was to arrive, an announcement–in Italian–was made and everyone surrounding us on the platform quickly went to the stairs, went under the tracks, and went to the platform over. Our assumption was that our train had been switched to another platform. So we followed the crowd.

This was a mistake.

At the new platform, I looked at the sign that indicated the train number. The new sign didn’t match what was on our ticket but the old one still did. The train at the new platform pulled up and everyone around us got on. I overrode deep instinct and decided to not follow the herd of fellow Homo Sapiens. We sprinted back, burdened by our luggage, to the original platform.

A train arrived, its number matching the number on the sign matching the number on the ticket and we boarded. I was 90% sure we were on the right train.

On this particular train, the cars were split up into cabins that had 3 seats across from 3 seats. We made our way to our cabin and found our seats. The only issue was that there were two business men in expensive suits already situated comfortably in our seats. I was now 35% sure we were on the right train.

I had been generally confused about transit things for hours, was drenched in sweat from lugging around a large suitcase in the heat, and assumed that the comfortable-looking business people were in the right and we were on a train to somewhere that was not Vernazza. In desperation, I showed the two men our tickets. They sighed heavily and then an intricate social dance ensued. One man stood up and leaned over to the woman across the aisle from him that from my vantage point had nothing to do with our seats or our situation. She packed her belongings, and she and the other man left the cabin. The first man then took the seat of the woman, leaving Meghan and me two now-empty seats.

Since this leg of the trip was about 3 hours, I had plenty of time to contemplate what had transpired. In Italy, you can buy two kinds of tickets. The first is what we originally had: specific seats on a specific train on a specific date. The second is what everyone else in the country seemed to have: permission to ride a specific route within the next 3 months. With the latter, you grabbed a seat and if a reserved ticketholder came by, you moved to the next empty seat. My guess is that the man that took the woman’s seat had a reserved seat, had let her sit there, and once we interjected our American confusion into the whole thing, he sat in his assigned seat, bumping her to the next cabin.

Now riding calmly, I read my Kindle and stared out the window at the gorgeous seaside towns, even striking up a conversation with a young woman who was an economics major in Milan who gave us advice of things to do in Vernazza. Things were going well.

At one point, with my Kindle in my lap, I leaned over to say something to Meghan. My Kindle slid off my lap and into the crack between the seats and fell under my seat, out of reach. The 5 other people in the cabin–my wife included–wanted to see what the silly tourist was going to do next. I surprised them all: I gathered myself and then did absolutely nothing.

About an hour later, after a couple of people from our cabin had disembarked in Genoa, I had room to attempt to rescue my Kindle. I crouched down in this cramped cabin and began to fight with the seat. By everyone’s intense interest, I could tell even the regular Trenitalia travelers had no idea how the seats worked. After fiddling for a couple of minutes, I discovered the seats moved in a very non-intuitive way, giving me just enough room to slide my sweaty arm under the seat to a point where I could reach my Kindle. I pulled it out and held it up with a dramatic, “Ta-da!” Everyone was very impressed, I imagined.

We reached Sestri Levante after dark. We disembarked and I sprinted to the ticket punch machine to validate our tickets to Vernazza. (These were the second kind of tickets I mentioned earlier, where you don’t have a reserved seat. Before you ride though, you use a self-serve ticket punch machine to mark the ticket on the date of travel.) I punched the ticket, went to the video screen to figure out which platform we needed to be on. Vernazza wasn’t on the screen, and our train was supposed to arrive in less than 5 minutes. “Oh no,” I thought, “Not again.”

I ran back towards Meghan, prepared to just hop on whatever train came next and let come what may. I found her talking to a group of people that were obviously tourists, sporting sunburns and speaking English. Turns out that they not only spoke English, they were American. And not only that, but they were from the same small city in North Carolina that we lived in before moving to San Francisco. Best of all, though: they knew which platform the train to Vernazza would come into. We followed them to the platform, chatted a bit more, and once our train arrived, we boarded and collapsed into our seats.

Waiting for the Train at Vernazza

After all that transit excitement, we spent three wonderful days exploring Cinque Terre and didn’t even bother looking into buying tickets for Rome until the morning we left.


What I learned from all this is that Italian trains are much closer to having schedules like buses and subways than airplanes. The system is built around things happening that change your plans. I learned not to expect to be in any station at any given time and instead buy my tickets for the next leg on the spot.

So when in Italy do as the Italians do: Relax and make your away across the country, one station at a time.

La Spezia Centrale

Tiny Tiny RSS, a Google Reader Replacement

Google Reader is being discontinued on June 30th. This is probably the web application I’ve used the most and for the longest so I was a little bummed when they made the announcement. I immediately started looking for its replacement.

I wanted the replacement to be something that had a liberal open-source license (so I could poke at the code and share patches) and something I could host myself to avoid having to worry about the whims of some big corporation’s product roadmap.

After some exploration, I found one called Tiny Tiny RSS. It used just PHP and MySQL, which meant I didn’t need to install or maintain any new things on my server. The code looked relatively sane and clean so I downloaded and installed it. Within probably half an hour, I had a very usable and surprisingly feature-rich RSS reader that even included an API. It’s also multi-tenant right out of the box.

The only caveat I had was that it didn’t have a usable mobile version. Today, I figured out that some kind developer had written a plugin that exposed a fever-compatible API which Reeder for iPhone supports. (Hint: After installing the plugin you have to enable API access for your account to make it work. In retrospect that seems obvious, but I had to printf-debug my way through the app to figure why it wouldn’t accept my username and password.)

Another your-URL-goes-here app

The end result of all this is that I have an RSS reader that will stay running as long as I’d like with no danger of it and/or my data being sold to the highest bidder along with a nice-looking iPhone app to read on the go.

This whole thing is also proof that the web can be resilient to vendor lock-in as long open standards (like OPML, RSS, HTTP, etc.) and the spirit that encourages them (like Reeder’s your-URL-goes-here screen) sticks around.

How Secure is my Dropbox?

The answer to this is: Secure as any other “private” content uploaded to the Internet, which is “not very.”

Dropbox is a widely-used service that lets you keep a specific directory in sync across many computers. Copy a file into the Dropbox folder on your work computer and it is nearly instantly available on your phone and your home computer. No more emailing files to yourself or scrambling to find a site that’ll let you upload a big file. They even give you a slick, web-based interface to browse your files.

This all sounds really easy and convenient which, unfortunately, usually means it’s not secure. The problem is that Dropbox stores your files on their servers encrypted in a way that they can read them.

First, what does one mean by “secure”? My definition of secure is that no one else in the world could possibly see the contents of something unless I let them. If it’s a text file, no one except me can read it; if it’s a photo, no one except me can see it.

For most of the files people want to share or sync, the security level of Dropbox is adequate. Disregarding that bug a couple of years ago where anyone could log in with any password, Dropbox is password protected and they recently introduced two-factor authentication, where you have to type in both your password and a short-lived, always-changing set of numbers from your phone. All connections between you and Dropbox go over SSL. This means no one can snoop on files you send to Dropbox and no evil person can trick your computer into thinking they are Dropbox.

From my limited knowledge of Dropbox’s internals, gleaned from reading a few security analysis reports, they do encrypt your files before uploading them. The downside is that they own the key to decrypt them “to ensure everyone has the ability to view and share files on the web painlessly.” Translated, this means that people-who-are-not-you can read your files.

I’ll assume that Dropbox, the company, follows industry standards for security. Only certain engineers get access to certain machines. Only certain support people get access to your files as necessary. Only code that’s been properly vetted for security bugs is deployed.

The weak spot in all of Dropbox’s efforts are the people. This isn’t a knock on Dropbox at all; people are the weakest spot in ANY security system.

Servers are constantly barraged by people trying to break in, and they often succeed. Support people sometimes stray and snoop at files they they shouldn’t. Developers write bugs that let random people on the internet get access to things they shouldn’t. It happens, despite best efforts in engineering or culture. People make mistakes.

If you’re wanting to keep your music in sync between computers, or want to quickly send a photo to a friend, Dropbox is great. It’s incredibly convenient. If it’s a file that you wouldn’t want another person to have, like your password file or financial documents, you don’t give it to Dropbox.

There is one (and only one) workaround to this though. If you encrypt a file on your computer before giving it Dropbox, they won’t be able to read it. 1Password, the popular password manager, takes this approach. They store your passwords in a file they then encrypt on your computer using high-grade encryption software. They then place this encrypted file into your Dropbox. Even if this file was to leak somehow, no one else but you could open it. Dropbox is purely the syncing service, which is still a handy thing to have.

The sad part of encryption is all the tools are terribly hard to use. TrueCrypt is probably the easiest of the bunch but there’s still a bit of learning curve to the terminology (partitions, volumes, and encryption algorithms?). I use GnuPG to encrypt my files, but that involves using a command-line interface, something most people aren’t (and probably shouldn’t have to be) comfortable using. OpenSSL is the Swiss Army knife of all things encryption but using it properly is like knowing some secret wizard’s spell.

Dropbox can read anything you give to it, in the state you give it to them. Give them something that only you can read and you get the joy of having this file everywhere while still being the only one who can open and read it.

I’m not saying to not use Dropbox. It’s just a fact that any file uploaded to Dropbox, given enough time, stands a good chance of being seen by someone you don’t know, so adjust your file syncing accordingly.

This post was inspired by a Twitter message from Kellan yesterday.

Afternoon Chats with the Navy

This morning, I read Claire Vaye Watkins’ essay The Ivy League Was Another Planet about the college application process for an American rural high schooler. Her story was nearly identical to my experience.

My high school had one overworked guidance counselor that also doubled as a college counselor. She was a very nice woman that seemed concerned about our futures, but there were only so many hours in the day for her. I vaguely remember my one or two sessions with her consisting of showing me the SAT testing calendar and pointing me towards the federal aid forms. Through some of confusion of mine, I didn’t think I needed to fill out these aid forms. These turned out to have been a prerequisite for many merit-based scholarships as well, which would have been useful information.

Somehow I escaped these sessions with only applying to one school, the main state school in North Carolina. I chose this school because a couple of my good friends were going there and I knew it was a “good school.” No one let me know that applying to just one university was a bad idea. I did really well in high school and not going to college if I had missed on this one application would have been a disaster.

Even the only application I filled out was a disaster-in-waiting. Like most college applications, it required an essay. I don’t remember getting anyone to proofread mine. I can barely write my name without a grammar mistake so I’m surprised they even let me on campus. Who knows, maybe they saw me as a great fixer-upper.

Watkins’ memories of taking the SAT were reminiscent of my own. The test was paid for out of pocket, and it didn’t seem to be guided by any internal force in school. It was just one of those things that we knew we needed to look into and apply to take. Afterwards, I learned from people that went to other high schools that the SAT was something that was taken multiple times and that taking preparation classes was commonplace. I took the sample test in the SAT packet and felt like I was being extra studious by even doing that. I had college friends that had studied hard for the math portion, rested during the verbal sections, and then did the inverse on another taking since you’re allowed to combine your best scores from your sittings. We could have probably figured out this strategy for ourselves, but the thing is we shouldn’t have had to. Even so, taking a $50 test multiple times would definitely have been out of reach for most of my classmates.

Our entire class also took the ASVAB. This test was administered in the school cafeteria and we got out of an afternoon’s worth of classes to take it. I did well on this test and within a week I had military recruiters calling me. The Navy must have called dibs on me as every afternoon after school, a Navy recruiter would call and we would chat. He was an excellent salesmen. He nearly convinced me that living in a submarine for months at a stretch was a completely normal life choice. The military’s rigid environment appealed to me at the time and I was probably a 60/40 split between going to college or joining the Navy. I realized this morning that if I had enlisted, I would have probably been at basic training during the September 11th attacks.

My high school ended up sending roughly half of my graduating class to college, half of those to the local community college and the other half going to a four-year school. Half of the four-year students went to Appalachian State University, which is a great school up in the mountains about an hour’s drive from the high school. I imagine the high application rate to this school was the same phenomenon that Watkins mentioned in that it was the closest four-year college to our high school and there were a lot of alumni around.

I know people who went to elite private schools that applying to college was a multi-year process with the school helping you research colleges that matched your academic needs, keeping track of application dates, paying for tests, and generally herding you through the often-confusing and always-expensive process of something that is still one of the best ways to improve one lot’s in life. Colleges in the US, I believe, are welcoming and available to everyone with the wide range of diversity and financial aid scholarships they offer but, like too many things, some are already starting a few steps closer to the finish line.

FlickrTrickle

This is a little tool that I wrote a couple of years ago that I think a handful of people use occasionally. To keep up with my philosophy of “if it can be open sourced, it should be open sourced,” combined with asking for people's read-and-write privileges to their Flickr accounts without showing the code, I finally got around to taking out the secret config bits, writing a README and opening it up. Below is a copy of the README.

What?

FlickrTrickle lets you slowly introduce photos you upload to Flickr into your contact's streams so they'll get seen by working around Flickr's interface that only shows the last five photos show from a contact, regardless of how many they just uploaded.

This is the code behind FlickrTrickle.

Why?

So let's say you go on a trip, like hiking the Appalachians, touring federal prisons, or visiting Disneyland, and get shutter-happy and take more than 5 photos. With the rise of digital cameras, taking multiple pictures in one day is not unheard of, and is practically encouraged in some circles. Now, you want to upload your more-than-a-handful number of pictures to Flickr. You're quite proud of these photos, either due to their composition, their dynamics, or the way the light plays softly against edges of your latte art next to the kitten, and want your friends to see them. But Flickr (rightfully) wants to keep your photos-from-your-friend's stream from being dominated by one person if they were to upload 150 photos so they only show the last five uploaded from each your contacts. This is very egalitarian of them.

Through anecdotal experience, anything beyond that 5-photo barrier gets next to no views. Maybe my storytelling abilities don't drive people to want to see those next photos but the interface doesn't help me out either.

You want more. You want your friends to see your photos. The best way to make this happen for me is to only upload a maximum of five at a time. Remembering to do this and keeping track of what you've already made public is an exercise in bookkeeping, something that computers are quite good at and my wife will attest to that I'm terrible at.

How?

The way around this is to use three different Flickr features: tags, privacy settings, and the date-posted-at attribute.

Flickr sorts your photos in the global stream by the date posted to Flickr. Like most things on the site, this is adjustable by the API, meaning you can lie to the computers and say “yep, I uploaded this one a second ago even though you saw me upload it two weeks ago.” Computers are gullible that way. We can use that superpower and upload photos to Flickr whenever we'd like but set them private just to get them up there. Sometimes you want a photo to be a private without being visible to this trickling-interface (trickle-face?) so the code only pulls photos you've tagged with “flickrtrickle.”

Then through a magical web interface (aka, this code), you select the ones you want to be visible to your friends and the date stuff automatically work itself out so it looks like you uploaded them at that moment.

You get to upload all your photos to Flickr at one time, and then “trickle” them in a few at a time.

Until a few months ago, my Flickr account was still using the same 6-letter password that Yahoo automatically assigned to me in the late 1990s. Most of my other accounts were using a variation of this password, with a dumb algorithm that used pieces of the site's URL with some numbers attached. None of this was good.

After Mat Honan's terrible hacking fiasco this past summer, I realized that, yeah, this could happen to me too and I should just take care of locking everything down the best I can. Here's the current state of how I'm staying safe online.

The first major improvement I made is that I started using a password manager, going with LastPass. I've used a Linux laptop at home forever and use Macs at work and have an iPhone so I needed something that worked on these different systems. LastPass has an iPhone app, a Chrome plugin, and a nifty website so this worked for me. Quite a few of my friends use 1Password, which works on the same principle of storing passwords in an strongly encrypted file. The main difference is that LastPass stores this encrypted file for you whereas 1Password requires you to take care of transporting, protecting, and syncing this file.

After using a password manager for a few months, there's no way I could go back to remembering a hundred or so slightly different passwords and often having to reset them due to forgetfulness.

I then started making the effort that every time I hit a site that hadn't been added to my LastPass account, I would change the password to a new, machine-generated password. I also started storing fake answers to security questions alongside each password in a special notes field the manager provides. Now every site I log into has a long, hard-to-crack, and unique password with security questions that no one could answer.

My current goal is to know only the password to my password manager with all the others locked away. The downside to this is all my passwords are protected by just one password which is a weak spot as passwords can be logged, guessed, or brute forced. Luckily, LastPass offers two-factor authentication which means that people would need both the password and my phone that runs the Google Authenticator app.

The tricky thing

Using two-factor authentication is incredibly secure but the tricky part to consider is what happens when I can't access my phone, like in the case of the battery being dead, or stolen or lost in the worst case. Most services that offer two-factor authentication also give you the option to set up ahead of time a short list of one-time passwords. These should be generated and stored where you always have access to them. I store these in two spots: in my wallet and a symmetrically-encrypted file on my personal server for redundancy.


Security and convenience sit on opposite ends of a spectrum. While it takes a little effort to set up, it makes your accounts much more secure from data leaks and hacking attempts and not having to remember multiple, insecure passwords is a burden one should be happy to be relieved of.