I woke up this morning around 7:30 with my laptop (a Lenovo R61i running ArchLinux) not feeling quite as snappy as usual. After viewing top, I saw where my nightly backup to tarsnap was still running, which kicks off via cron. This script usually takes 5 minutes, tops.
My first feeling was that it was probably the couple of gigabytes of email that I backed up from Gmail yesterday, though even with the extra bits, my outbound bandwidth should have been able to allow that go through much faster than 4+ hours.
So I attached to the running tarsnap process with strace, just to see what it was doing. I saw lots of selects, receives, and sends, so it appeared to be sending things across the network, which was good.
Seeing the network activity, even though it was behaving properly, set off alarms. A bunch of new files to backup wasn't the only thing I changed yesterday: I had also switched out my Linksys WRT54GL's router firmware with Tomato, mainly due to its extensive Quality-of-Service settings.
We've got a few streaming media devices around the house and I wanted to give them priority (based on MAC, static IP, network interface, etc).
Another nice thing about Tomato is that comes with graphs. I love graphs, and Tomato gives you bandwidth graphs, broken down by timerange and NIC.
Pulling up the graph verified that I had a block of outbound traffic start at 3am, but it was never rising above 30kbps. It was being capped.
So I went back to the QoS interface and sure enough there was a class of traffic labeled “lowest” for “Bulk Traffic” (which matched any TCP/UDP packets heading out to a port 1024 or above) and it was being capped at 30kbps. While fiddling with the inbound limits in the QoS admin, I completely glazed over the outbound limit defaults. (Also, this is why I'm not a network operator.)
Since this is the only outbound traffic of any significance through this router, and I run it at night to not interfere, I just disabled all outbound limits.
Here's the 24-hour bandwidth graph after turning the outbound limits off:
The green is outbound traffic, being capped for several hours, and then spiking and then quickly finishing.
So Tomato is awesome but is evil in that way that only computers can be, which is by doing exactly what you tell them to do.