Sun, 18 Feb 2018
Thoughts about On-Call
This month there have been a couple of interesting discussions about on-call rotations in the tech industry. The first was started by Charity Majors, who sparked a thread on Twitter:
All this heated talk about on call is certainly revealing the particular pathologies of where those engineers work. Listen:— Charity Majors (@mipsytipsy) February 10, 2018
1) engineering is about building *and maintaining* services
2) on call should not be life-impacting
3) services are *better* when feedback loops are short
A couple days later John Barton followed up with an article that I really enjoyed, and pretty much whole-heartedly endorse. I had a few thoughts from both of these, and wanted to talk about them here.
"But that's just an incentive for engineers to weasel extra pay by building broken systems": I think this falls apart in several ways. First, that extra pay doesn't just appear with no additional consequences — the engineer on-call still has to actually fix the problem, wake up at odd hours, be bothered when they'd much rather be bowling or watching a movie or reading a book or sleeping, etc. Second, if this actually works at your company, your management is broken. Period. That's the whole point of it, to put an explicit material cost to this additional duty. If your management tolerates abuse of this pay, they either explictly consider this part of the cost of doing business, or they're not paying close enough attention, and both of those cases are entirely on them.
To everyone that argues that an engineer's pay covers this, I'd counter by asking "Okay, how much of that pay represents the on-call expectation?" I'm guessing many places wouldn't be able to do that. And unlike many things an employer pays for that are fuzzy, hard-to-define criteria, this one is easy, all it takes is a stop-watch and a calculator to count up how many minutes are spent responding to incidents. Is what you're being paid for it worth it? As John points out, many other industries with highly trained professionals pay on-call differentials, and tech shouldn't be any different.
I'd also add a guideline to John's list: if someone gets a page, the next day someone covers for them for 24 hours. While this isn't official policy where I work, it's my own unofficial policy to offer to cover for my co-workers when they have a particularly bad on-call day. Someone who is woken at 3am, even if they can go back to sleep ten minutes later, doesn't get as good as rest and isn't as effective the next day. Having that followed by another interrupted sleep the next night both makes the problem worse and also makes it so that the the most critical person on your team, the one responding to an emergency, is in less than peak condition. Don't let people shrug this off with an "I'm fine" — there's a large body of sleep research that disagrees with them.
Like many things in tech that I think are bad, it's only going to change if expectations start changing, and expectations aren't going to change unless we start prodding them in the right direction. I think these kinds of questions, asking for the kinds of policies John advocates, needs to be something more standard industry-wide. If my situation warrants, I plan on making this part of the questions I ask any potential employer, and if your situation warrants, I'd ask you to do the same.
Sun, 21 Jan 2018
Disabling Yubikey 4 OTP
Since I can never remember this:
I don't make use of the Yubikey OTP mode, so I don't want what a former co-worker called "yubidroppings" when I accidentially brush my key.
Short answer: get
./ykpersonalize -m 5, since I only want U2F and CCID modes enabled. Tell it
Sat, 20 Jan 2018
Restic Systems Backup Setup, Part 4.5 - Why not just rclone
.@thomaskula nice article! Did you consider just running rclone in a loop?— restic (@resticbackup) January 15, 2018
After I posted part 4 of my restic backup series, @resticbackup asked the above question, and I thought trying to answer it would be a good intermediate article.
As a brief background, rclone describes itself as "rsync for cloud storage". It can talk to a rather comprehensive number of storage providers as well as local storage, and can perform operations as simple mkdir, cp and ls and ones as complicated as syncing between two different storage providers. It, like rsync, is a useful tool in your kit.
So why not just run rclone in a loop? Actually, that might not be a bad idea, and it's certainly a simple one. Pick a loop sleep that matches your replications needs, put some error checking in, and fire away. If I were going to do this, I'd likely use rclone copy rather than rclone sync. copy will copy files from the source to the destination, but will not delete any files on the destination which do not exist on the source. sync on the other hand, will make the source and destinations look exactly the same.
My preference for copy over clone is two fold. First, I like the idea of having different retention policies at different repositories. For example, some of the storage provider options are so inexpensive, for my scale of storage needs, that I basically treat them as "just shove things in there forever, I don't care", or, at least, care about once a year. On the other hand, local fast storage is much more expensive, that perhaps I can only afford to keep, say, a week or two of backups around for all of my systems. By treating the repositories as distinct, and with the append-only nature of restic, I can do that, keeping what I'm likely to need for most restore operations at hand, and keeping longer term, much much less likely to be needed restored data off some place where it's harder to access but cheaper to keep.
The second reason for treating the repositories separate is it helps guard against "oh shit!" moments: if you are cloning every five minutes and you accidentially delete some data you need, you've got a narrow window to realize that and stop the clone. At some point in your life, you will do this — I remarked once that "[s]ome people think I'm smart, but that's not it. I just remember exactly what each of those bite marks on my ass means."
That all said, I'm going to keep using the mechanism I outlined in the last article, of firing off a new sync job every time a new snapshot appears. There's a few reasons for this. First, it's there, and its working. Baring some overriding need to change this setup, I don't plan on exerting energy to change it — for now.
Second, there is some amount of overhead cost here every time I do a sync. My goal is that data starts being synced within a couple minutes of a new snapshot being created. I'm still, however, mostly doing the one-backup-a-day-late-at-night model (at least for now). With that, I'll actually have work to do less than one-tenth of one percent of the time, which just feels off. I'll admit, of course, that's just a gut feeling. In addition, even if I'm not copying data, building up a list of what I have locally and, more importantly, what's at the remote repository, has some cost. All of the storage providers charge something for operations like LIST, etc. That said, honestly, I haven't ran the math on it and the charge here is almost certainly one or two epsilons within nothing, so perhaps this isn't much of a reason to care.
The two important bits on conclusion: first, I have something working, so I'm going to keep using it until it hurts to do so, which, honestly, is a good chunk of the reason I do many things. We'll be fancy and call it being "pragmatic". Second, your needs and costs and criteria are certainly different from mine, and what's best for you requires a solid understanding of those things — one size certainly doesn't fit all.
Mon, 15 Jan 2018
Restic Systems Backup Setup, Part 4 - Replication and Runsvdir
A goal from the start of this project has been replicating backup date to multiple locations. A long personal and professional history of dealing with backups leads me to the mantra that it isn't backed up until it's backed up to three different locations. Restic has several features which make this easy: backend storage (to a first approximation) is treated as append only — a blob, one stored, is never touched although may be deleted as part of expiring snapshots. Second, everything is encrypted, so you can feel as safe spreading your data to any number of cost-effective storage providers as you trust restic's encryption setup (which I generally trust).
In general, I want the client systems to know only about one service, the server we're backing up to. Everything else, the replication to other storage, should happen on the backup server. Also, we want new snapshots to get replicated relatively soon after they are created. If I decide to make an arbitrary snapshot for whatever reason, I don't want to have to remember to go replicate it, or wait until "the daily replication job"
These criteria lend themselves to something which watches for new snapshots
on the backup server. Restic makes this easy, as one of the very last things
it does after a sucessful snapshot is make a new
There's one directory to watch, and when a new object appears there, replicate.
How to do that, though?
Minio does contain a notification system, and I strongly considered that for a while (to the point of submitting a patch to some incorrect documentation around that). But that offered two complications. First, setting up notification involves both changing the minio configuration file and also submitting some commands to tell it what you want notifications for, which complicates setup. Second, I quickly fell down a rabbit hole of building a RESTful notification service. This isn't impossible to overcome, but it was blocking the real work I wanted to do (more on that later).
My next consideration was using the Linux kernel
to watch for events in the snapshot directory, but that also fell under
roughly the same problems as the previous solution, and also added some Linuxisms
that I didn't want to add at this point. Of course, that said, I do freely
bash scripts, with some bashisms in them, instead of a strictly
POSIX-compliant shell, but, frankly, I'm not all that interested in running
this on AIX. So, take this all with an appropriate grain of salt.
The solution I finally set on is
backup-syncd, the not as elegant
but still useful setup. This simply runs in a loop, sleeping (by default for a
minute) and then looking at the files in the
If the contents have changed, fire off a script to do whatever syncing you want
to do. There's some extra stuff to log and be robust and pass off to the
sync script some idea of what's changed in case it wants to use that, but otherwise
it's pretty simple.
A decent part of systems engineering is fitting the solution you make to the problem you actually need to solve. I'm not expecting to back up thousands of systems to one backup server, so the overhead of a watcher script for each client waking up every minute to typically go back to sleep isn't really a consideration. And yes, depending on timing it could be almost two minutes before a system starts replicating, but that's close enough that I don't care. And while I do want to eventually build that RESTful syncing service to work with Minio's notification system, that's a desire to understand building those services robustly, and shouldn't get in the way of the fact that right now, I just want backups to work.
That said, another decent part of systems engineering is the ability to make
that solution not fuck you over in the future. You have to be able to recognize
that what fits now may not fit in the future, and while what you're doing now
may not scale to that next level, it at least won't be a huge barrier to moving
to that next level. In this case, its easy enough to swap out
with something more sophisticated, should it be necessary. You could go
another way, as well — for a low-priority client you could certainly
backup-syncd to only wake up every few hours, or even
forgo it completely in lieu of a classic cron-every-night solution, should
the situation warrant.
Now that we have more than one service running for each client, I've updated
the setup to use a per-client
runsvdir, which manages all the
services a particular client needs to do backups. Here we have a top-level
runsvdir, called by the systemd unit file, which is responsible
for running the services for all clients. In turn, that top-level
runsvdir for each client, which in turn runs minio and
backup-syncd for that client. The idea here being that I want to treat each
client as a single unit, and be able to turn it on and off at will.
There's a small issue with the way
runsv manages services. To
runsvdir and everything its running, you want to send
SIGHUP. The way we start a client
runsvdir is to
make an appropriate symlink, which does what we expect. But when we remove
that symlink, the supervising
runsvdir sends the client
SIGTERM signal, which makes the client
runsvdir go away without touching the child
it started. You can customize what happens to the client
process, however, and I'll be doing that in a future phase of this project.
I'll end here by outlining some future ideas and wants for this setup:
- Monitoring and sanity checking: I want some sort of audit of every storage backend for a client, to make sure that the snapshots I want are where I want them
- Restoration checking: A wise person once said that nobody wants a backup system, everybody wants a restoration system. Something which restores some set of files and does some sanity checking would be good
- Metamanagement: Instead of making symlinks and poking around manually, I want scripts where I can enable and disable a particular client, get the status of a particular client's backups, etc.
Updates and Engagement
The standard end-of-the-year party and eating season conspired to keep me from much creative work here, but I've been off work this past week and managed to wrap up a new issue of Late Night Thinking and do some work on my restic systems backup setup. Both will appear here shortly.
Also, if you're one of the small number of people who haven't found this out from any number of places, on 1 November 2016A I got engaged to E, my boyfriend of two years. Wedding is this coming November.
Thu, 02 Nov 2017
Restic Systems Backup Setup, Part 3 - Setting up a client
Setting up the backup server side
Using 'new-restic-server' to set up the server
You can find
new-restic-server in the git repo.
/backups/bin/new-restic-server -H aclient.example.com -p 9002
will set up all of the per-client setup on the backup-server: making a
config and path for storage, setting up a
runsv directory to run the
minio server, and creating access key and secret for the
You will have to make sure the port you picked (in this example,
distinct between all clients backing up to this service.
Activing the minio server
minio server is a distinct step, but an easy one with our
ln -s /backups/systems/aserver.example.com/conf/runsvs/minio /backups/systems/conf/runit/aserver.example.com-minio
A few seconds later,
runsvdir will detect the new symlink and start the minio process.
Setting up the client side
Installing the binaries
I install these all in
/usr/local/bin, you'll need to get a recent copy
of restic, as well as the
restic-wrapper scripts from the
directory of the git repo (handily linked at the end of this article).
First, make an
/etc/restic configuration directory:
sudo install -o root -g root -m 700 -d /etc/restic
Create the environ file
/etc/restic/environ contains a series of environment variables that the
client will use to identify the repo to backup to, as well as the access keys for it. It looks like
export AWS_ACCESS_KEY_ID=key goes here export AWS_SECRET_ACCESS_KEY=secret key goes here export RESTIC_REPOSITORY=s3:https://backup-server.example.com:9002/backups export RESTIC_PASSWORD_FILE=/etc/restic/repo-password
Most of these are self-explanitory. The
RESTIC_REPOSITORY is marked as
because that's what
minio looks like to it. It ends in
you have to put things in a "bucket"
to read from that file, instead of prompting for a password.
Create include and exclude files
Now the hardest part, deciding what to backup and exclude. Everything will be backed up from
/, use full paths in the include an exclude files, which go in
Configure repo password
sudo /bin/sh -c 'pwgen 32 1 > /etc/restic/repo-password'
Here, we're using the
pwgen command to generate a single, 32 character long
password. YOU MUST NOT LOSE THIS. This is the encryption key used to encrypt
everything in the repo, and without it, you won't be able to recover anything. I store mine
in a GnuPG encrypted git repo that I backup outside of my restic setup.
Initialize the repo
sudo /usr/local/bin/restic-wrapper init
will initialize the repo. It will spit out something like:
created restic backend bcae9b3f97 at s3:https://backup-server.example.com:9002/backups Please note that knowledge of your password is required to access the repository. Losing your password means that your data is irrecoverably lost.
Set up a cron job to do daily backups
contains a useful
cron.d snippet to perform daily backups, modify to your taste.
We now have a client system which backs up daily to a backup server storing
minio. Future articles will talk about automated replication
to additional repositories for redundancy.
Sat, 30 Sep 2017
Restic Systems Backup Setup, Part 2.5 - dealing with 'Unable to backup/restore files/dirs with same name'
You should be reading Part 3 here, but in the development of that, I ran into this restic bug: Unable to backup/restore files/dirs with same name.
Unfortunately, for historic reasons (buried in some of the oldest code in restic),
only the last component of a path being backed up in a restic repository is
reflected in the repo. For example, in my case, when I wanted to back up both
/usr/local they would show up as
at the top of the repo, a very confusing state. Later versions of restic would rename
things so there was a
local-0 and a
local-1, etc, but it's
still very confusing.
The primary restic developer is working on resolving this, as many other people
have ran into it, and it is expected to be fixed in restic 0.8. Until then, the
suggestion is to tell restic simply to back up
/, and exclude out
everything you don't want to back up. A workable enough solution, but I still want
something where I can think in terms of backing up what I want, and something else
figures out how to do the exclusion. That way, I can just add or remove things from
my list and I don't have to re-figure what to exclude. Or, things can come and go
that I don't care about, and they won't accidentially get backed up.
A few hours of work and experimentation, and I had restic-unroll, which does just that. In the same directory in that git repo is an example bash script you might wrap things in to do a daily system backup.
As a reminder, you can find the canonical repository of all my utility scripts in this series here
Sat, 16 Sep 2017
Restic Systems Backup Setup, Part 2 - Running minio under runit under systemd
As described in Part 1, my general strategy is to have a centralized backup
server at a particular location, running an instance of minio
for each server being backed up. In essence, I'm going to want to be running N
minio server --config-dir=/... instances, and I want a simple
way to add and start instances, and keep them running. In essence, I want
a simple init service.
Fortunately, if you're looking for a simple init service, you need look no
further than runit. It's an incredibly
tiny init-like system, composed of some simple tools:
run a service, keep it up and optionally log stdout output somewhere;
sv to control that service by simply talking to a socket; and
runsvdir to keep a collection of
going. Defining a service is simple, in a directory there is a
file, which is used by
runsv to start the service. If you want
to log, create a
log subdirectory, with it's own
file — that file is executed and given the stdout of the main process
as its input (the included
svlogd command is a simple process
for handling logs). To run a bunch of
runsv instances, put
them (or symlinks to them) all in a single directory, and point
at it. As a bonus,
runsvdir monitors that directory, and if a
runsv directory is created or goes away,
does the right thing.
It's an incredibly useful set of commands, and allows you to manage processes
fairly easily. In this case, every time I add a machine to this backup
scheme, I make an appropriate
runsv dir with the correct
minio incantation in the
run file, and just
symlink it into the
runsvdir directory. We've been using
runit at work for quite a while now in containers, and it's
an awsome tool.
My newly-minted backup server is running Debian Stretch, which uses
systemd as its init system. Creating systemd unit files is still
something I have to think about hard whenever I do it, so here's the
one I use for
[Unit] Description=Backup Service Minio Master runsvdir [Service] ExecStart=/usr/bin/runsvdir -P /backups/systems/conf/runit/ Restart=always KillMode=process KillSignal=SIGHUP SuccessExitStatus=111 WorkingDirectory=/backups/systems User=backups Group=backups UMask=002 [Install] WantedBy=multi-user.target
Here, systemd starts
runsvdir, pointing it at my
top-level directory of
runsv directories. It runs
it as the
backups user and group, and makes it
something that starts up once the system reaches "multi-user mode".
Part 3 is coming, where I'll document backing up my first system.
Current PGP Practices: GPG 2.1 and a Yubikey 4
I might write this up as a full tutorial someday, but there's already a few of those out there. That said, here's a short outline of my current usage of PGP, aided by modern GPG and the OpenPGP smartcard functionality of a Yubikey 4.
- Use GnuPG 2.1. Private keys are stored in a
.ddirectory, can act as an ssh key agent, and you can forward your local
gpg-agentto a remote server. Oh, and it supports OpenGPG smartcards.
- Use OpenSSH > 6.7. Makes
gpg-agentforwarding much easier.
- Use contemporary GnuPG configuration settings Riseup used to have a good guide for this, but it's sadly vanished behind HSTS. But this YubiKey-Guide has the settings, if not the explanation. It's also a fairly comprehensive step-by-step set of instructions for the entire process.
- Keep backups of your master keypair and revocation certificates. Pretty straightforward, not only will you need this to, say, load a new Yubikey or change a subkey, you'll also need this to sign anyone else's keys. I keep three copies at all times, with one always in a bank safe deposit box.
- Generate your master key offline. A Raspberry Pi not plugged into any network is a great tool for this, although you'll most likely have to bang your hands on the keys quite a bit to generate enough entropy for key generation.
- Use an imense passphrase on your offline key. This is very easy, since you'll only need to actually use this to a) update any subkeys; b) sign anyone else's key; c) push your subkeys into your Yubikey. And the Yubikey will be protected by a) being a physical thing that; b) must be plugged in; c) unlocked with a six digit PIN; d) and touched to actually do anything. Speaking of....
- Use the yubitouch utility to require touch. You can find that here. I use the mode where you have to touch the Yubikey for all three subkey usage, and fix it so that the setting can't be changed without re-loading the key material. This can be slightly paranoid, and I do wish it had a mode to "require a touch if it hasn't required a touch in the last N seconds". But I do like knowing that every use of my Yubikey requries me to physically touch it.
- Make a 'transfer' copy of your GNUPGHOME to load subkeys onto your Yubikey The process of loading your subkeys into a Yubikey replaces the secret key material with a pointer that says "this subkey is in OpenPGP card with the serial number...", and traps it in the Yubikey (by design).
- Use git to track your offline keys This has saved me from at least one blunder, and it gives me a history of what I've been doing to the keys over time.
- Set your key expiration to a fixed date, and update every few months You can set a key to expire in, say, two years, and then three months later, move the expiration date forward three months, etc. This has got a couple useful side effects. One, if for some reason you lose control of your key, it at least will go away sometime. Two, it forces you to touch your master key at least semi-occasionally. In my setup, I touch all three copies of my master key once every three months, so I'll be able to recover if one of the USB thumb drives decides to give up the ghost. Much better than leaving a drive in a drawer and five years later learning that it's unreadable.
- CHANGE BOTH PINS ... but after you've done all the card setup. Many of the things above will require you to enter the unlock and/or admin PIN, and it's much easier to type '123456' or '12345678' for all of this. Make a good PIN, don't make it something easily guessable, etc. etc. In used three 2d10 rolls to make mine.
- Entering the PINs too many times doesn't brick the card. We had some confusion about this at work, and thought we'd bricked a card. It turns out that entering the regular PIN enough times just makes it so that it won't do anything other than allow you to use the admin PIN to reset the regular PIN. And if you enter the admin PIN wrong three times, it just wipes the key material from the key and resets it to factory defaults. In fact, I'm fairly certain that the script
- Other info from Yubico
- All of this owes a great deal of debt to Alex Cabal's Generating the Perfect GPG Keypair, which got me thinking all about this in the first place a few years ago.
Sat, 09 Sep 2017
Restic Systems Backup Setup, Part 1
This is the first in what will undoubtedly be a series of posts on the new restic-based system backup setup.
As I detailed earlier this week, I've started playing around with using restic for backups. Traditionally, I've used a variant of the venerable rsync snapshots method to backup systems, wrapped in some python and make, of all things. Some slightly younger scripts slurp everything down to a machine at home so I've got at least another copy of everything.
In my previous post, I discussed my initial attempt at restic, simply replicating that home backup destination into Backblaze B2. That works, but it feels a bit brute-force, and there have been other things I've wanted to change about this for a while:
Replicating from colo to home takes an order of magnitude longer: Backing up the ten or so VMs I have on my colo machine takes about 10 minutes. Pulling that down to home takes 100 minutes or so. (I'll note here that the bulk of my 'large' data is in AFS; what I'm backing up on systems is primarily configuration files, logs, and some things that happen to live locally on a system).
Some of this is due to the fact that the replication traffic goes
from Michigan to New York, while the initial backups are all
happening within the same physical host. But the larger part,
I think, is due to the fact that in order to replicate my system
backups, I have to preserve hardlinks. A bit of background
here: the 'rsync snapshots' method works by using the
--link-dest option to rsync. As I backup a system,
if the file hasn't been changed, rsync makes a hardlink to the
corresponding file in the
--link-dest directory. This
doesn't use any additional space, and it's an easy way of keeping,
say, fourteen days worth of backups while only using more space
for the files that change from day-to-day. Most of my systems
keep that may days of backups around.
Since I want to replicate all of those backups (and not, for
example, only replicate the latest day's worth of backups),
but I want to keep the space savings that
gets me, I need to use the
-H argument to the replicating
rsync so it can scan all the files to be sent to find multiply hard-linked
files. This takes a long long time — so much so that the
sshd man page warns about it:
Note that -a does not preserve hardlinks, because finding multiply-linked files is expensive. You must separately specify -H.
The backing-up or replicating rsync must run as root: Of course the rsync on the machine being backed up must run as root, it needs to be able to read everything to be backed up. But the destination side also has to run as root, because I want to preserve permissions and ownership, and only root can do this. I've long wished for an rsync 'server' that spoke the rsync protocol out one side and simply stored everything in some sort of object storage. Unfortunately, the rsync protocol is less a protocol and more akin to C structs shoved over a network, as far as I understand. And the protocol isn't really defined except as "here's some C that makes it go".
Restoring files is done entirely on the backup server: Because of the previous issue, I didn't want root on the client servers to ssh in as root on the backup server — I felt it was much safer and easier to isolate backups by having the backup server reach out to do backups. There's no ssh key on the client to even be able to get into the backup server. It's not a big issue, but if I need to restore a handful of files spread out I've got kinda stage them somewhere and then get them over to the client system. And because the backup server has a command-restricted ssh key on the client server, it takes some convoluted paths to get stuff moved around.
Adding additional replicas adds even more suck: Adding another replica means another 100 minutes somewhere pulling stuff down. And it also means a full-blown server, someplace where I can run rsync as root, and it's got to be some place I trust. Also, most of the really cheap storage to be found is in object storage, not disks (real or virtual) — part of what attracted me to restic in the first place.
When I started playing with restic, I saw a tool that could solve a bunch of those problems. Today I've been playing around with it, and here's my ideas so far.
Distinct restic repositories: One of the benefits of restic
is the inherent deduplication it does within a repo. And if I were backing
up a large number of systems, I might save something by only having one copy of,
/etc/resolv.conf. But really, most of what I'm backing up
is either small configuration files, or log files. And these days, the few
tens of gigabytes of backups I have there isn't really worth deduplicating.
In addition, the largest consumer of backup space for me — stupidly
unrotated log files that get a little bit appended to them every day —
would benefit from the deduplication, even if it's only deduplicating on a
More important than that, however, is that I want isolation between my systems. For example, the backups of my kerberos kdc are way more important than, say, web server logs. And I really don't want something that would run on a public-facing system be able to see backups for an internal system. So, distinct repositories.
Use minio as the backend: My first thought when I was going to experiment was to use the sftp backend to restic. But to isolate things fully, I'd have to make a distinct user on the backup server to hold backups for each client, and that sounds like too damn much work.
Unrelated, I've been playing around with minio. Essentially, it's about the simplest thing you can get that exposes the 90% of S3 that you want. "Here's an ID and a KEY, list blobs, store blobs, get blobs, delete blobs". Because it's very simple, it doesn't offer multi-tenancy, so I will have to run a distinct minio for each client. That said, I think that should be easy enough, especially if I use something like runit to manage all of them.
Benefit from the combination of minio and restic for replication:
Minio is very simplistic in how it stores objects:
stored as the file
/top/of/minio/storage/some/key/name. This has two
benefits: first, because the minio storage directory is also a restic
repository, I can just point a restic client at that directory, and as long
as I have a repository password, I can see stuff there. Second, every file in
the restic repository other than the top level 'config' file is named after
the sha256 hash of the file as it exists on disk, and all files in a repository
are immutable. This makes it trivial to copy a restic repository elsewhere.
While I'll likely start by simply using the
b2 command line tool to
sync things into B2, I think you can do it even faster. I haven't looked deeply,
but my gut feeling is that the
b2 sync command looks at the sha1
hash of the source file to decide if it needs to re-upload a file that exists
already in B2. We don't need to do that at all; repository files are named after
their sha256 hash, so if the files have the same name, they have the same
contents . So moving stuff around
is incredibly trivial.
Future niceties. I've got a bunch of other ideas floating around
in the back of my head for restic. One is a repository auditing tool: since nearly
everything in restic is named for the sha256 hash of the file content, I'd like a
tool I could run every day that would pull down, say, 1/30th of the files in the
repository and run
sha256 on them, to make sure there's no damage.
The second is some way of keeping a local cache of the restic metadata so operations what have to read all that are much faster. Third, and related, a smarter tool for syncing repositories. For example, I'd love to, say, keep three days of backups in my local repository, and be able to shove new things to an S3 repository but keeping seven days there, and shove things in B2 and keep there until my monthly bill finally makes me care.
Anyways, this has been a few hour brain dump of a few hours of experimentation, so I'll end this part here.
- : Well, until sha256 is broken....