logd
Started around December 2023. logd is a log daemon: services fire log lines at it over UDP, and you tail or query them live from anywhere. It ran in production, where it did more for our observability than anything else we had, and it is still the most fun I have had building something.
I had been building protocols for fun since I was young. This is the one that ended up in production — security flaws and all, as it turned out. UDP is most of why it exists. TCP hands you a stream and hides the machinery. UDP hands you a datagram and a socket and leaves the rest to you — who is connected, what arrived, what to do about the packet that didn’t, all of it becomes a decision you have to make on purpose. I wanted to make those decisions.
The tail #
The part I still like best: one command, every service, live.
Log lines are written under a path — /prod/my-app/http, /prod/my-app/udp, /debug. A query carries a key prefix rather than a key, so the store walks every ring whose name starts with what you asked for:
for key, r := range s.rings {
if strings.HasPrefix(key, keyPrefix) {
matchedPrefix = true
for d := range r.Read(offset, limit-count) {
out <- d
// ...
}
}
}
Which means tailing /prod merges every service beneath it into a single stream. Watching a whole deployment scroll past in one terminal, interleaved in real time, is a genuinely different way of seeing a system than opening one log file per box. That was the feature people actually used.
Ring buffers all the way down #
The store is a map[string]*ring.Ring — one fixed-size ring per key, sized in config, plus a fallback ring for anything unrecognised. A write is an index bump and nothing else:
func (b *Ring) Write(data []byte) {
head := b.head.Load()
b.values[head] = data
b.head.Store((head + 1) % b.size)
}
Memory is decided at startup and never moves. There is no rotation, no compaction, no disk, no vacuum job at 3am. Old lines fall off the back. For what is happening right now — the only question I ever asked it — that is the entire feature set.
The other half of that decision is UDP itself. A logger that blocks the thing it is logging is worse than no logger, and a write here is a sendto that cannot apply backpressure to the application even if the daemon is gone.
Nobody is connected #
UDP has no connections, so who is currently tailing? is a question the socket cannot answer. The server keeps a map of tails, each client sends a PING every two seconds, and missing three drops you. A client that dies stops pinging and is cleaned up by absence.
I like this more than I expected to. A TCP server learns about a dead client from an error on a write; a UDP one just stops hearing from it. Same outcome, less machinery.
The hard part #
Every packet has to authenticate itself, alone. There is no session to establish trust once and then ride on.
The wire format has been the same since v0 — a 32-byte SHA-256 sum, 15 bytes of marshalled time, then the Protobuf payload — and the sum is taken over the secret, the time and the payload together. The time is what bounds the damage: a packet more than 200 ms old is rejected before anything else happens, so a captured packet is only worth replaying for 200 ms.
Which leaves the actual question: have I seen this packet before?
Attempt one: remember every hash #
A ring of the last few thousand sums, compared one at a time under a mutex.
g.history.Do(func(v interface{}) {
b, ok := v.([]byte)
if !ok {
return
}
if bytes.Equal(b, sum) {
found = true
}
})
This gets worse exactly where it needs to get better. Every packet costs a full scan of the ring, so making the history longer makes each packet more expensive — and because the ring holds a fixed number of packets rather than a fixed duration, the span of time it covers shrinks as traffic rises. The busier the server, the less it remembers. It was backwards in both directions at once.
Attempt two: a cuckoo filter #
Constant time, comfortably over 200,000 packets a second, and about four lines of code:
func (g *Guard) replay(sum []byte) bool {
return !g.filter.InsertUnique(sum)
}
Two things wrong with it, and I only saw one at the time.
The one I saw: a filter has no notion of when, so it cannot expire anything. The only way to stop it filling is to reset the whole thing on a timer — ten seconds, in the shipped config — and at every reset all memory is dropped at once, so a replay from just before the tick sails straight through. There is a comment in main.go where I worked out the fix and never wrote it:
FilterCap: 16000000, // balance for memory, larger size reduces false positives
// now I know, we need 2 smaller filters, A & B.
// Each must pass, and each is reset off-kilter to prevent
// replay immediately after filter reset.
The one I didn’t see: InsertUnique returns false when the item is already present and when the filter is too full to accept it. So replay() reports a replay in both cases. Fill the filter before its timer comes round and the server rejects every packet until it does.
And the sizing tells the story on its own. Packets are valid for 200 ms. The filter holds ten seconds — fifty times the window — in 16 MB of fingerprints, to answer a question about the last fifth of a second.
The answer #
I found it in the WireGuard spec, which is worth reading whether or not you are ever going to write a protocol. A sequence counter, incremented per packet and included in each one. The server tracks the highest it has seen: anything far behind is rejected outright, without a lookup, and anything close is checked against a small bitmap of which of the recent numbers have already arrived.
The reason it works is that a counter is ordered and a set is not. Ordering makes forgetting free — once the counter has moved on, everything behind it is rejected by arithmetic, with nothing to reset, clear or time out. My filters had to remember far more than they needed to, because the only alternative on offer was remembering nothing at all.
And it wasn’t HMAC #
I described this, in the README and in the first version of this post, as hash-based message authentication. It isn’t, quite. HMAC is a specific construction — two nested hashes with padded keys — and it exists precisely because the obvious thing, H(secret || message), is not safe with SHA-256.
SHA-256 is a Merkle–Damgård hash, and its output is its internal state. Given H(secret || time || payload) and a guess at the length of the secret, you can resume the hash from that state and compute H(secret || time || payload || padding || anything) without ever knowing the secret. My verifier splits the packet at a fixed offset and recomputes over secret || time || payload, so an attacker who appends to the payload and supplies the extended sum passes the check.
What saves this particular deployment is narrow and accidental. The appended bytes have to include the hash’s own padding, and Protobuf will not parse the result — so the forgery is thrown out by the parser, one step after the check it defeated. And the 200 ms validity window means you have 200 ms to capture a packet, extend it and land it.
Neither of those is a defence. hmac.New(sha256.New, secret) is the same amount of code as the thing I wrote instead.
Looking back #
2024-05. This was an enjoyable project to rediscover writing software. It’s amazing to return to something after a change of mind. A new mind has more fun, and is more productive. The same thing happened when I re-imagined speedflying after my spinal-cord injury. Some years after my recovery, I rediscovered a lost joy with a new mind.
2024-12. I find it amusing that I struggled so long with replays. The sequence counter is simpler than everything I tried before it, and once you have seen it, a membership filter looks like the wrong shape for the job.
I would build it differently now — for most of what this did, a process reading the systemd journal is the honest answer, and the logger abstraction is a layer I would not reach for again. But the daemon did its job for a real team on real traffic, and the parts of it I got wrong are the parts I now understand. You don’t learn why HMAC exists by importing it.