Sat, 14 Jul 2018

Wrapping Consul Lock

I've recently installed a Consul cluster at home, mostly to act as an HA backing store for Vault. If you've been following along, I've also been moving to Restic for my system backups so, of course, I want snapshots of Consul to end up there.

But this isn't a post about that — when I've got it running well and cleaned up, I'll post it and talk about it. What I want to talk about is the way that I'm wrapping the consul lock command.

The gist of the script is:

  1. Run consul snapshot save... and save the output to a local file
  2. Archive that file away

I want that running on all three Consul servers, but I only ever want the snapshot to happen on a single server at a time. That way, if I ever have to take a Consul server down for maintenance, snapshots will still happen within the cluster. This is a common practice, and it's really easy with consul lock. This is also useful where you want a cronjob or task to run, and it can run on more than one server but you only want it to run on a single server at any time. Rather than only running it on a single server, and having to remember to switch it elsewhere in the event that server goes down, or having to install some sort of distributed queueing system, you can use this.

If you do consul lock only one instance of will run at a time, which is the basis of the technique I'm using. But I don't want to put the consul lock logic in my crontab, I want it embedded in the script. Here's a simple way of doing that: 

#!/bin/bash

[...]

function inner {
    #do stuff inside the lock
}

#parse arguments here, getopt, etc

if [ "$1" == "inner" ]; then
    shift
    inner $@
elif [ "$1" == ... ]; then
    #other subcommands
else
    consul lock $KVPREFIX $0 $ARGUMENTS $GO $HERE inner
fi

If you call your script normally, it will use consul lock to call itself with the argument inner, which does the stuff which should only be done on a single host at a time.

Posted at: 23:03 | category: /computers/consul | Link