Late Night Thinking

by Thomas L. Kula

Thoughts about On-Call

This month there have been a couple of interesting discussions about on-call rotations in the tech industry. The first was started by Charity Majors, who sparked a thread on Twitter: All this heated talk about on call is certainly revealing the particular pathologies of where those engineers work. Listen: 1) engineering is about building *and maintaining* services 2) on call should not be life-impacting 3) services are *better* when feedback loops are short

Disabling Yubikey 4 OTP

Since I can never remember this: I don't make use of the Yubikey OTP mode, so I don't want what a former co-worker called "yubidroppings" when I accidentially brush my key. Short answer: get ykpersonalize and run ./ykpersonalize -m 5, since I only want U2F and CCID modes enabled. Tell it yes twice.

Restic Systems Backup Setup, Part 4.5 - Why not just rclone

This is Part 4.5 of my series on building a restic-based system backup series. The rest of the articles can be found here. .@thomaskula nice article! Did you consider just running rclone in a loop? — restic (@resticbackup) January 15, 2018 After I posted part 4 of my restic backup series, @resticbackup asked the above question, and I thought trying to answer it would be a good intermediate article.

Restic Systems Backup Setup, Part 4 - Replication and Runsvdir

This is Part 4 of my series on building a restic-based system backup series. The rest of the articles can be found here. Replication A goal from the start of this project has been replicating backup date to multiple locations. A long personal and professional history of dealing with backups leads me to the mantra that it isn't backed up until it's backed up to three different locations. Restic has several features which make this easy: backend storage (to a first approximation) is treated as append only — a blob, one stored, is never touched although may be deleted as part of expiring snapshots.

Updates and Engagement

The standard end-of-the-year party and eating season conspired to keep me from much creative work here, but I've been off work this past week and managed to wrap up a new issue of Late Night Thinking and do some work on my restic systems backup setup. Both will appear here shortly. Also, if you're one of the small number of people who haven't found this out from any number of places, on 1 November 2016A I got engaged to E, my boyfriend of two years.

Restic Systems Backup Setup, Part 3 - Setting up a client

This is Part 3 of my series on building a restic-based system backup series. The rest of the articles can be found here. We've got enough things setup that we can start backing up a client system. We'll do this in two sections: setting up the server side, and setting up the client side. Setting up the backup server side Using 'new-restic-server' to set up the server You can find new-restic-server in the git repo.

Restic Systems Backup Setup, Part 2.5 - dealing with 'Unable to backup/restore files/dirs with same name'

This is Part '2.5' of my series on building a restic-based system backup series. The rest of the articles can be found here. You should be reading Part 3 here, but in the development of that, I ran into this restic bug: Unable to backup/restore files/dirs with same name. Unfortunately, for historic reasons (buried in some of the oldest code in restic), only the last component of a path being backed up in a restic repository is reflected in the repo.

Restic Systems Backup Setup, Part 2 - Running minio under runit under systemd

Part 2 of my series on building a restic-based system backup setup. Part 1 can be found found here. As described in Part 1, my general strategy is to have a centralized backup server at a particular location, running an instance of minio for each server being backed up. In essence, I'm going to want to be running N minio server --config-dir=/... instances, and I want a simple way to add and start instances, and keep them running.

Current PGP Practices: GPG 2.1 and a Yubikey 4

I might write this up as a full tutorial someday, but there's already a few of those out there. That said, here's a short outline of my current usage of PGP, aided by modern GPG and the OpenPGP smartcard functionality of a Yubikey 4. Use GnuPG 2.1. Private keys are stored in a .d directory, can act as an ssh key agent, and you can forward your local gpg-agent to a remote server.

Restic Systems Backup Setup, Part 1

This is the first in what will undoubtedly be a series of posts on the new restic-based system backup setup. As I detailed earlier this week, I've started playing around with using restic for backups. Traditionally, I've used a variant of the venerable rsync snapshots method to backup systems, wrapped in some python and make, of all things. Some slightly younger scripts slurp everything down to a machine at home so I've got at least another copy of everything.