After I finished the primary work on my Internal Certificate Authority I started down another rabbit hole which has led me even further from the original reason I created the internal CA to begin with. My tooling takes a snapshot of the Vault data, and I wanted to keep some sort of audit log. I have a normal backup process for the data, but I want to make checksums of the data and configuration, and then publish that to some sort of public record so the next time I need to use the CA, I can verify the state. There’s more details, but that’s what started the journey.
At the same time, I was also wondering about using age encryption, maybe with a YubiKey, to store the Vault seal keys. This led me to looking at the web page to see what the latest status on that was, and I coincidentally came across this page
That page describes how age uses sigsum to sign releases, which you can not only
verify but which are also logged in a public append-only log.
Which sounds exactly like what I want to do.
I went through the sigsum getting started page,
and was successful, but I was trying to understand fully what is actually going on, and what everything in
the “proof” file sigsum-submit gives you means. As is often the case, the best way for me to do this is
to study the code of sigsum-submit and related tools, and then write my own code to independently verify
the signature on a file &emdash; if I can do that, I understand things properly.
Here’s the process:
Create a file called hello.py
cat << EOF > hello.py
#!/usr/bin/env python3
print("Hello, world!")
EOF
The file hello.py should have exactly 47 characters with a sha256sum of
7a3b2aef6d971667623e7a679c76bfefaa52db592a1fd7df7876b7c3bdb99724.
Create a signing key pair
Run sigsum-key generate -o test-20261014-0.key. This generates an
Ed25519 key pair, formatted to look like an SSH
key. The file test-20261014-0.key is our public key, and looks like
this in my testing:
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIC5ofJ1wYqe4VqePTgf657A6q0S3dVGMcKRn8rJcqY2y sigsum key
Get the key hash
Run sigsum-key to-hash -k test-20261014-0.key.pub. In my case, you get back
97725c0c5fe58778e60ab1957b86596751c1c527cca1c318d9649aea06ddfd89
Sign and submit the checksum
sigsum-submit -P sigsum-test1-2025 -k test-20261014-0.key hello.py
This will take some seconds to minutes, depending on how long the log
server you are using from the sigsum-test1-2025 policy takes to
get a quorum of cosignatures. You will get back a hello.py.proof
file, which starts like this in my test case:
version=2
log=4e89cc51651f0d95f3c6127c15e1a42e3ddf7046c5b17b752689c402e773bb4d
leaf=97725c0c5fe58778e60ab1957b86596751c1c527cca1c318d9649aea06ddfd89 b77835100d48ba6039cbe1fe19d7fdb662c893180c465668f57f2e87d4d012a8b05b3a4055f29e827b2ca87d1ed34f61ea9edd40e2078d81aba0f169f1b0f10f
This is known as a leaf,
and unsurprisingly, the leaf line is what you use to verify the signature on
hello.py. The remainder of the file is the signatures of the log which you
submitted the checksum to, along with the cosignatures of “witnesses”, a
quorum of which is required and defined by the policy file, plus a Merkle-tree
has of every previous log entry. (We’ll cover those in the future).
So what does this line mean?
leaf=97725c0c5fe58778e60ab1957b86596751c1c527cca1c318d9649aea06ddfd89 b77835100d48ba6039cbe1fe19d7fdb662c893180c465668f57f2e87d4d012a8b05b3a4055f29e827b2ca87d1ed34f61ea9edd40e2078d81aba0f169f1b0f10f
Well, notice that the first part, starting with 97725c0, is the key
hash of the public key. This turns out to be a 32 octet sha256 hash
of the raw public key, which is not the same as the public key file
itself.
Remember, sigsum keys are stored as SSH-encoded private and public keys.
So AAAAC3NzaC1lZDI1NTE5AAAAIHexbRjvFqdqHjDPPquVGD7lYVVCaONJC1jT3l6VOLfT
is a base64 representation of the following things:
uint32(len('ssh-ed25519')) || 'ssh-ed25519' || uint32(len(key)) || key
And what sigsum uses as the public key, and what that public key hash is a hash of, is just the last part, the key.
What’s the second part, the part starting with b7783510? It’s 128
characters long, and in hex, which means it encodes 64 octets, which
just so happens to be the length of an Ed25519 signature. Somewhere
in here has to be a hash of hello.py, but a sha256 hash, which
sigsum uses exclusively, is 32 octets long.
But, if you think about it, if you are using this proof file to verify the hash of a file, you already have the file, and you can hash the file. In fact, you’d need to, right, to verify the file? So that’s not included in the proof. It is stored in the log. Well, a derivative of it.
What do we mean by that? Again, if you look at the specification for a leaf you’ll see that a leaf is defined as:
struct tree_leaf {
u8 checksum[32];
u8 signature[64];
u8 key_hash[32];
}
But what you send to add a leaf is:
- message: The message used to create the
checksumabove. It must be exactly 64 hex characters, encoding 32 octets. - signature: The Ed25519 signature, also hex encoded. It must be exactly 128 hex characters, encoding 64 octets.
- public_key: The hex-encoded public key corresponding to the private key which created signature
The log design document goes into detail on the Objective and Threat Model, and a key part of that is that sigsum uses exclusively sha256 hashes and Ed25519 signatures. The sha256 hashes are useful for several reasons:
- The log size is very well defined: a valid log leaf is exactly 128 octets long, a log submission is 256 characters, encoding 128 octets. That stores everything necessary to verify some data, but only that which is necessary to verify the data.
- It is difficult to store sensitive, malicious, or illegal data in 32 octets: Logs, to be useful, have to be durable. They have to stick around for some time to be a public record of signatures. And a complete log is a Merkle-tree hash, which means you cannot remove any arbitrary data from it, because if you do, every log entry following it is now unverifiable. A log operator who allow submission of sensitive, malicious, or illegal data, and who gets an ethical or legal obligation to remove that, faces two choices: ignore that obligation, or remove the log entirely. By limiting the size of the data in the log, sigsum drastically reduces the chances of that.
Sigsum recommends that the message you send be sha256(data to sign). The checksum stored in the leaf
is sha256(message), which means ultimately checksum := sha256(sha256(data to sign)).
The signature is defined as an Ed25519 signature over the following data:
'sigsum.org/v1/tree-leaf' || 0x00 || checksum
That sigsum.org/v1/tree-leaf is what is known as a namespace. In the description of
the cryptography sigsum uses
they explain that the namespace string is prefixed to all messages signed in sigsum to provide
domain separation, and to tie the signature specifically to sigsum. You have to deliberately
choose to sign data and mean it for sigsum.
With all of that together, how do we verify hello.py with our proof?
- Compute
sha256(hello.py). In our case that is7a3b2aef6d971667623e7a679c76bfefaa52db592a1fd7df7876b7c3bdb99724 - Create the checksum as
sha256(sha256(hello.py)``. In our case, that is7138d30e618745ee20d3e09b81ca45cfbf3a7db0eb526557a04d34e2ef2a5039. **NOTE**, the outersha256is a checksum of the *binary*sha256(hello.py), not of the string713b2ae….` - Build the string
'sigsum.org/v1/tree-leaf' || 0x00 || checksum. Remember, again, thatchecksumis the binary prepresentation of our checksum. - Create a Ed25519 signature of that with the private key. That signature just so happens to be:
b77835100d48ba6039cbe1fe19d7fdb662c893180c465668f57f2e87d4d012a8b05b3a4055f29e827b2ca87d1ed34f61ea9edd40e2078d81aba0f169f1b0f10f, the second part of ourleafline above.
Coming up eventually I want to answer the following questions:
- What does the rest of the proof file mean?
- The sigsum verification is offline, but how I can find the leaf entry in the log?
- Can I use the log entry to also determine when a leaf entry was signed?
Code I used to figure all of this out can be found here.