Joey Innes

Designing the pressure, not the shape

Thu, 30 Jul 2026 00:00:00 +0000

The first post in this series was about getting lift out of a computer, and the second was about getting drag out of one. Between them they build an instrument: put a shape in, get numbers back.

What I did not expect is how little that helps. I could compute a section’s lift and drag to three figures and still have no idea whether the section was any good — because good turns out not to be a property a section has. It is a relationship between a section and a wing, and until you have pinned down the wing there is nothing to ask.

So this post is about the question rather than the machinery. There is no code in it, and nothing in it comes from my own solver — that one is still failing tests I wrote for it, and is in no position to advise anybody. Everything plotted here is published reference data.

This is also the last of the three posts that are about the field rather than about the thing I am building. If you only ever read one of them, it should probably be this one.

There is no such thing as a good aerofoil #

The first thing to pin down is not the shape. It is where you fly.

A wing in steady flight makes exactly as much lift as the aircraft weighs — no more, or you would be climbing, and no less, or you would be falling. Weight does not change with speed. So the lift coefficient is not something a designer picks: it is dictated, moment to moment, by how fast you are going.

Double your speed and the dynamic pressure quadruples, so the coefficient you need drops to a quarter of what it was. Fly slowly and you need a large one.

Each line is one aircraft, flown from its slowest to its fastest. Nobody occupies a point.

That single fact reshapes the problem. You are not designing for a condition, you are designing for a track — a band of lift coefficient, at a band of Reynolds number, traced out as the aircraft speeds up and slows down. A paraglider on my numbers runs from about Cl 0.97 hands-up-slow to 0.27 on full speedbar. It has to be good across all of that, and it has to behave when it leaves it at the top.

The deliverable is a curve #

Given a band rather than a point, the thing being designed is not a number. It is the whole relationship between lift and drag, plotted against itself: the polar.

One curve, four readings, and no two of them at the same place on it.

Four things get read off it, and the useful surprise is that they are all different points.

Least drag is the leftmost point of the curve. For this section it is Cd 0.0059, at Cl 0.62.

Best glide is not there. How far you travel per metre of height lost is lift divided by drag, and the largest ratio is where a line drawn from the origin just grazes the curve — which for this section is Cl 1.05, a long way above the least-drag point. Slowing down costs you drag and gains you more lift than it costs. That is why gliders have a speed to fly, and why it is not their minimum-drag speed.

The bucket is the flat-bottomed region where drag barely responds to lift at all. Aiming that region is most of what a designer actually does.

The top is where the curve turns over and the section stops working. Everything past it is stall, and it is the part of the polar that computational tools are worst at — mine has no useful opinion there whatsoever.

Minimum sink is another point again: you lose height slowest where Cl^1.5 / Cd is largest, because sink rate is drag times speed and the speed itself depends on Cl. It sits higher up the polar than best glide, which is to say at a slower airspeed. Thermalling and cruising want different speeds for this reason alone.

The budget #

Here is the piece of physics that everything else in this post is downstream of. It took me a while to see it, and once I did, most of the field stopped looking like a collection of tricks.

Air flowing over the top of a wing speeds up, and where it speeds up the pressure falls. That low pressure is the lift — nearly all of it, as the first post’s hero image shows: the suction on top does most of the work.

But the air has to leave. At the trailing edge the flow from the top and the flow from the bottom rejoin and depart together, so they must arrive at very nearly the same pressure — which is close to ambient, because the wing is not doing anything to the air a long way downstream. Whatever the pressure fell by at the front, it must climb back at the back.

And that climb is what a boundary layer cannot do indefinitely. The thin sheared film described in the last post is running on whatever momentum it has left near the wall. Falling pressure pushes it along; rising pressure holds it back. Ask for too much of a climb, and the air nearest the surface stops, reverses, and the flow lifts off.

Same section, same trailing edge. The angle of attack decides how big the hill is.

That is the budget, and it is the whole game:

Lift is how deep a hole you dig at the front. Drag and stall are the bill for climbing back out of it.

At 2° this section digs to Cp −0.98 and has to recover 1.2, spread comfortably over the back five-sixths of the chord. At 8° it digs to −3.0, right on the nose, and has to recover 3.1 — nearly three times as much — starting immediately. Nothing about the tail changed. The wing has simply written a cheque the boundary layer has to cash.

Every knob a designer turns is a decision about how to spend that, and there is no way to spend it twice.

Buying a laminar run, and what it costs #

The most valuable thing you can spend it on is keeping the boundary layer laminar.

A laminar layer has roughly a fifth of the skin friction of a turbulent one at a chord Reynolds number of a million, and the gap widens as you go faster. It is also fragile, which is the point: it stays laminar only while the pressure is still falling. Falling pressure damps the little disturbances that would otherwise grow; rising pressure feeds them. Move the point where the pressure stops falling further aft, and transition follows it.

So: hold the pressure falling to 60% of chord instead of 20%, and you have bought forty percent of the wing at a fifth the friction. But the same total climb now has to be done in 40% of the chord instead of 80%, at twice the steepness — and steeper means closer to separating.

That is not a subtlety in aerofoil design. It is aerofoil design. And it shows up in the polar as the thing that gives a bucket its shape.

The bottom panel is the cause and the top panel is the effect, and it is worth reading them in that order.

Over the flat part, this section holds laminar flow to roughly two thirds of chord and drag hardly moves. Ask for more lift — dig a deeper hole at the nose — and transition runs forward from 0.59 to 0.05 of chord within half a point of Cl, putting a turbulent layer over the whole wing. The drag more than doubles. The wall of a drag bucket is not a separate phenomenon; it is that collapse, plotted upwards.

This is the historically interesting bit. NACA 63-412 is not a shape description. The 4 is a design lift coefficient of 0.4 and the 12 is thickness, but the leading 6 names the series and the 3 after it says the minimum pressure sits at 30% of chord. The name is a pressure distribution. That is the 1940s, and it is the moment the field stopped drawing shapes and started specifying pressures — which is where this post is going.

Two sections, and neither of them is better #

Which brings me to the pair in the hero image. NACA 4412 and NACA 63-412 are both twelve percent thick. Both are cambered. Both are being run at the same Reynolds number, in the same solver, at the same turbulence level.

At Cl 0.35, the laminar-flow section has 19% less drag. At Cl 1.05, the older one has 37% less. They cross at about Cl 0.7.

There is no reading of that data where one of them is the better section. There is only a question about which side of Cl 0.7 your wing spends its life on — and, since the answer is usually “both, at different times”, how much you care about each end.

I would add one thing that took me longer to appreciate. The laminar section’s advantage is contingent in a way the other’s is not. It depends on transition staying aft, which depends on the surface being smooth and the air being quiet. Squashed insects, rain, a repair, a gusty day: all of them move transition forward, and the bucket washes out. The 4412 was never relying on that, so it has less to lose.

Preferring the design whose advantage survives being wrong is not a compromise. On anything you actually fly, it is the design decision.

The knobs #

With the budget in mind, the classic parameters stop being arbitrary and turn into ways of spending it.

Camber — how much the mean line arches, and where the arch is. Camber moves the whole low-drag region up in lift, which is nearly free performance if you fly at high lift coefficients. Real numbers, same family, same Reynolds number:

section	camber	least drag	at Cl	best L/D	Cm there
NACA 0012	0%	0.0054	0.00	76	−0.007
NACA 2412	2%	0.0055	0.35	101	−0.055
NACA 4412	4%	0.0059	0.62	129	−0.100

The bucket walks up the lift axis almost in step with the camber, and the best lift-to-drag ratio nearly doubles. The price is in the last column: the pitching moment. A cambered section wants to pitch nose-down, and something has to hold it. On a conventional aeroplane it is the tailplane, pushing down, which is lift that must itself be paid for in drag. On a soft wing there is no structure to react a moment at all — the lines and the internal pressure do it, and the wing changes shape while doing it. A Cm of −0.1 is a detail on a Cessna and a design constraint on a paraglider.

Thickness — volume for spars, and on an inflated wing, the ability to hold its own shape. It also blunts the nose, which as we are about to see decides how the thing stalls. Past roughly 15% the drag starts climbing quickly, because the flow has to be accelerated harder over the shoulder and therefore decelerated harder afterwards. Same budget.

Where the thickness peaks — move the maximum aft and the pressure keeps falling further back, which buys laminar run and costs recovery distance. This is the laminar-flow trade in geometric clothing.

The leading-edge radius — small, and the suction peak at high angles becomes a spike; large, and it stays rounded. Almost nothing else about the section matters as much for what happens at the top of the polar.

The trailing edge — a small amount of camber right at the back is worth a startling amount of lift. I found this out by accident, through a bug that built an unintended half-percent bulge at 95% of chord and moved the lift by several percent, because to the outer flow it reads as a small deflected flap. The last post makes the same point from the other direction: the boundary layer builds one of these on every wing, for free, whether you wanted it or not. Aft loading is also where the pitching moment is made or unmade, which is why reflexed sections — turned up at the tail — exist for tailless aircraft.

Stall is a design decision, not an accident #

The top of the polar deserves its own section, because it is the one place where being wrong is dangerous rather than merely inaccurate.

All three peak in about the same place. Only one of them tells you first.

There are three classical ways to run out of wing, and which one you get is largely decided by the nose.

Trailing-edge stall is what thick, blunt-nosed sections do. Separation starts at the tail — where the boundary layer has been climbing longest and has least left — and creeps forward as the angle increases. Lift rolls over gradually, and the separated flow shakes the aircraft on the way, which is a warning.

Leading-edge stall belongs to sharper, thinner sections. The suction peak at the nose gets severe enough that a small separation bubble sitting there bursts, and the entire upper surface lets go at once. The lift curve falls off a cliff, with nothing beforehand.

Thin-aerofoil stall happens on very thin sections and at low Reynolds numbers. A bubble forms near the nose early and simply grows longer as the angle increases, so lift breaks well below what the section might otherwise have reached and then mushes along.

The project instruction I wrote for myself before any of this existed is never optimise for maximum lift alone, and this figure is why: the number at the top of the curve tells you nothing whatsoever about what happens immediately after it. And where there is a real trade — there usually is, because the sharp nose that buys peak lift is the same one that lets go abruptly — a section reaching Cl 1.7 and then dropping without warning is, for anything with a person hanging underneath it, a worse section than one reaching 1.5 and rolling over. For a soft wing it is worse still: the pressure distribution is not only making the lift, it is holding the wing’s shape, so losing it abruptly does not merely reduce lift — it can fold the wing.

The same shape is a different design at a different size #

One more thing that has to be part of the brief, because it changes the answer rather than refining it.

Identical coordinates. The problem is not the same problem.

Identical coordinates, three Reynolds numbers, and the best lift-to-drag ratio falls from 129 to 107 to 78. The drag at the bottom of the bucket nearly doubles.

The mechanism is the one the last post ran into: as Reynolds number falls, the laminar boundary layer becomes less and less able to survive a pressure rise, until it starts separating before it has transitioned. It then goes turbulent in mid-air and comes back down, leaving a laminar separation bubble sitting on the surface — which is expensive, and which is the single hardest thing in this whole subject to predict well.

This is why model gliders use sections that would be poor choices full-size, and it is a live problem for real wings too. The tip of a tapered wing runs at maybe a half to a third of the root’s Reynolds number, and the tip is where stall behaviour gets decided. A section chosen on root numbers is not a section chosen for the wing.

Nobody designs the shape #

Which brings me to the thing I actually wanted to write down, and the reason for the title.

Read back over everything above. Not one of the trades is naturally expressed in terms of coordinates. They are all statements about the pressure: how deep, how far back before it turns, how steep the climb, how sharp the peak at high angles. The shape is not the design. The shape is the thing that happens to produce the design.

So serious aerofoil work inverts the problem. You specify the pressure distribution you want — a rooftop over the front to hold laminar flow, a recovery shaped to get back to the trailing edge without separating, whatever the brief needs — and you solve for the geometry that produces it. That is inverse design, and it is what Eppler’s code, Wortmann’s FX sections and Drela’s mixed-inverse mode in XFOIL all do.

I do not have an inverse mode. Most people learning this do not. The amateur version of the same discipline still works and is worth adopting immediately: change one thing, and look at the pressure distribution rather than the outline. Your eye cannot read an aerofoil — every section in the useful range looks like every other one — but it can read a Cp plot fluently after about a week. Where is the peak? How steep is the recovery? Did that change put the pressure rise somewhere the boundary layer can pay for?

The section is not the wing #

A closing note on proportion, because it took me embarrassingly long to work out and it changes how much any of this is worth.

A real wing is finite. It sheds vorticity off its tips, which tilts the whole force vector back, and that induced drag is set by span and lift distribution — not by section. At an aspect ratio of 5.5, which is paraglider territory, and Cl 0.6, the induced drag is about 0.0245. The section drag next to it is about 0.007. Add the lines and the pilot, and the section is a small minority of the total.

So the honest framing is this. Choosing between a section with 0.0060 of drag and one with 0.0070 buys you a couple of percent of total drag, and choosing a better planform, or fewer lines, buys you more. That is not an argument for indifference — a couple of percent decides competitions, and section choice also decides the stall, which decides whether the wing is any good to fly. But it is an argument against believing the answer lives here.

And for the wings I care about there is a further discount. A soft wing is not the section you drew. It inflates into shape, deforms under load, has its nose cut open for the intakes, and is pulled into scallops between the ribs. Rigid-section data is the first term of the answer and not the answer, which is exactly why the 2D solver has to be both fast and trustworthy — it is going to be called a great many times inside something larger.

Next #

Which is the point I have been circling for three posts. What I want out of Foilworks is not a number. It is a polar you can believe across a band of lift coefficient and a decade of Reynolds number, that knows where it stops being right and says so.

That is the brief, and the rest of this series is about trying to meet it. From here the posts stop being about aerodynamics and start being about the thing I built: how it is put together and why in Zig, how I tried to prove it was right and what that measurement caught me doing, and finally the rewrite that abandons the loop in the second post and solves the boundary layer and the outer flow as one system with a real wake attached.

They are all, in the end, about the same worry. It is very easy to build something that produces plausible aerofoil numbers, and nothing in the output tells you which kind you have got.

Every measured number here came from published XFOIL polars and the reference boundary layers committed to the project, checked while writing rather than remembered. The two schematic figures say so in their captions. If any of it is wrong, I would like to know.

The boundary layer, where drag and stall live

Tue, 28 Jul 2026 00:00:00 +0000

Back in the spring I wrote about the half of aerodynamics where air has no friction. That half is fast, it is accurate, and it gives you lift and pressure. It also cannot tell you how much drag a wing has, or when it will stop flying — which are, unfortunately, the two things a person choosing a wing actually wants to know.

This post is the other story. It is where drag lives and where stall lives, and it is where I have spent most of my time since. It is also much the harder of the two, for a reason that is worth stating at the outset: the outer flow and the film each need the other’s answer before either one can start.

The film #

Everywhere except very close to the skin, air behaves as if friction did not exist. Very close to the skin, friction is the only thing that matters, because air genuinely sticks to solid surfaces. The molecules touching the wing are, on average, going exactly the speed of the wing. A tiny distance away they are going full speed.

Everything friction does to a wing happens in that gap.

The vertical scale here is a lie by a factor of about two hundred. Horizontally this is a slice of wing; vertically it is a film you could not see.

The gap is thin. On the section in the hero image at Re 10⁶ it is about one percent of the chord over most of the wing. That thinness is the entire reason any of this is tractable, because it means the outer flow barely notices the layer exists, and the layer can be solved as if the outer flow were simply handed to it.

Nobody solves the profile #

The obvious approach is to work out the actual shape of that velocity curve everywhere, which means solving a partial differential equation over a two-dimensional region for every case. People do this. It is called a Navier-Stokes solve, it takes minutes to hours, and it is not what I want when the point of the project is to sweep a hundred candidate sections.

The trick, and it is a very old one, is to stop caring about the shape of the curve and keep only two numbers per station: how much air is missing, and how much push is missing.

Both areas are real quantities with units of length. Once you have them you can throw the profile away.

δ*, the displacement thickness, is the first. Add up the air that failed to arrive because of friction, and ask how far out you would have to move the wall to lose that much air from a flow with no friction at all. That distance is δ*. It is the number that matters most here, because it is the only thing about the boundary layer that the outer flow can see: as far as the frictionless solve is concerned, the wing is simply δ* fatter than it really is.

θ, the momentum thickness, is the second. Same idea, but weighting each bit of missing air by how fast it was going. This one is the drag, more or less directly — momentum that the air had at the front and does not have at the back went into the wing.

One number decides everything #

Their ratio, H = δ*/θ, turns out to be the interesting one. It says nothing about how thick the layer is and everything about its shape — specifically about how full the profile is near the wall, which is to say how much momentum the flow has left down there to keep pushing itself along.

The dashed line at each wall is the slope of the profile there, which is the skin friction. On the right it has gone to zero.

A turbulent layer runs at H ≈ 1.4. It is constantly dragging fast air down from the outside and slinging slow air up, which keeps the wall region energetic — at the cost of far more skin friction. A laminar layer sits around 2.6: much slicker, much less able to cope with anything difficult.

And when the flow is asked to climb into rising pressure — which is exactly what the whole rear half of an aerofoil asks of it — H climbs. Somewhere around 2.4 for a turbulent layer, and nearer 3.5 for a laminar one, the profile runs out of momentum at the wall, the flow stops, and it lifts off the surface. That is separation, and it is the mechanism behind stall.

So the entire drama of a wing section is: how long can you stay laminar (cheap, but fragile), and having gone turbulent (expensive, but tough), can you get to the trailing edge before H runs away?

Waiting for a wobble #

The switch between the two is called transition, and it is the single most consequential thing in the whole model, because it decides both the drag and the toughness.

It is not a threshold you cross. A laminar layer is unstable long before it is turbulent: a tiny disturbance at the right frequency grows, exponentially, as it travels downstream. The model I use — the e^N envelope, from Drela and Giles — integrates that growth along the surface and declares the flow turbulent once the most-amplified disturbance has grown by a factor of e^N.

Real data from the reference solution on this exact section. Nothing happens for the first three percent of chord; then it grows for the next forty.

The obvious cheaper alternative is Michel’s criterion, which is a local correlation: look up the state at this station, ask whether it has tripped yet, forget everything else. It is consistently wrong here, firing 0.1 to 0.2 of chord too early across Re 2×10⁵ to 10⁶ and costing about five points of mean drag error. The reason is the whole point of the previous paragraph. Amplification has memory. A lookup does not, and instability is a thing that accumulates.

Drag, at last #

With a laminar run, a transition point and a turbulent run, you can finally compute a drag that means something. The route is Squire and Young’s, from 1937, and it is a lovely piece of accounting: take the momentum thickness at the back of the wing, and extrapolate what it would become infinitely far downstream, where the wake has finished mixing out and the pressure is ambient again. Momentum that never comes back is drag, so that far-downstream θ is the drag, up to a factor of two.

What that buys is the ability to ask where a wing’s drag actually comes from, and the answer changes completely across the range you fly in.

Both bands are drag. Only one of them is friction, and it is the one that stops mattering.

At small angles a wing’s drag is almost entirely skin friction — the air rubbing along the surface, which is θ growing steadily along a well-behaved layer. That is the regime the whole integral method is comfortable in.

Push the angle up and skin friction barely changes, while pressure drag goes up by a factor of nineteen. Nothing is rubbing harder. What has happened is that the layer on top has thickened and begun to come away, so the outer flow can no longer close neatly around the back of the section, and the pressure on the rear no longer pushes forward as hard as the pressure on the front pushes back. That is the same d’Alembert cancellation from the last post, failing.

Two consequences worth carrying forward. The first is that a cambered section’s least-drag angle is not zero lift — it is somewhere near the lift it was drawn for, because that is where the pressure distribution is gentlest on both surfaces at once. The second is that Reynolds number moves all of this, and not always the way you would guess. Drop a NACA 0012 from Re 10⁶ to 5×10⁵ and its least drag rises from 0.0054 to 0.0062 — even though transition slides aft, from 0.69 to 0.79 of chord, so it is getting a longer run of the cheap kind. The friction itself is rising faster than the extra laminar run can save.

Go one step further down, to Re 2×10⁵, and something else appears entirely. Down there the laminar layer separates before it ever transitions, sits off the surface for a few percent of chord, goes turbulent in mid-air and comes back down. That is a laminar separation bubble, it is expensive, and it is the single hardest thing in this subject to predict. Everything in this post assumes the layer stays attached to the surface until it is good and ready to leave, and down there it simply does not.

The circular argument #

Now the awkward part. The boundary layer needs to be told the speed at the edge of the layer, which comes from the frictionless solve. And the frictionless solve needs to be told the shape of the body, which — because of δ* — depends on the boundary layer.

Neither solver can go first, so both of them go badly, several times, until they stop disagreeing.

So you guess, and iterate. Solve the clean shape, march the layer, thicken the shape by δ* along the surface normals, re-panel it, solve again. Repeat until the two stop disagreeing. On the sections I have tried that takes between four and twenty-two passes, and the whole thing still runs in a few milliseconds, because the expensive part — factorising the influence matrix — is a couple of hundred rows and only has to happen once per shape.

This shape of problem — two models, each needing the other’s output as its input — is worth recognising because it is not special to aerodynamics. What is specific here is that the loop has a hard edge to it, and the next two sections are about where that edge is.

The shape the air actually sees #

Here is what the coupling is doing, physically. At any real angle of attack the boundary layer on top of the section is much thicker than the one underneath, because the top is the surface being asked to climb the pressure hill. So the fattened body is not fattened symmetrically — it is fattened mostly on top, mostly at the back.

Nobody put a flap on this wing. The boundary layer did it, and it does more of it the harder the wing is working.

Which bends the effective camber line tail-up. Less camber means less lift, so the viscous answer sits below the frictionless one. That is viscous decambering, and it is the entire lift correction — the whole reason a real wing makes less lift at a given angle than the frictionless theory promises.

It also explains something that would otherwise look arbitrary. There is a fiddly detail in every scheme like this about what to do at the very back, where the displaced upper and lower surfaces have to meet at a single point again. It looks like bookkeeping. It is not: a bulge half a percent of chord deep over the last five percent of the wing reads to the outer flow as a small trailing-edge flap, and aft camber is disproportionately good at making lift. Get that closure wrong and you can move the lift by several percent in whichever direction your mistake happened to point.

Where the loop breaks #

That iteration works right up until the boundary layer starts to separate, and then it fails in a way that is worth understanding, because it is why the state of the art looks the way it does.

Marching the layer in direct mode means being handed the edge velocity and asked for the layer. As the flow approaches separation — as the shear at the wall approaches zero — that problem stops having a solution at all. Not a hard one: none. The equations become singular exactly at the condition you most want to resolve.

So a scheme like this has to be kept away from the edge, usually by clamping something on the way out of the march. The clamp is not physics, it is scaffolding, and it costs you exactly the region you would most like to be right about — because separation approaching the trailing edge is the onset of stall.

That is the honest limitation of the whole approach, and it is the reason the top of a lift curve is so much harder than the middle of one. The layer knows perfectly well that the flow is coming apart; the direct loop has no way of letting that news reach the lift.

What this cannot tell you #

It is worth being as blunt about the limits here as the last post was about the inviscid ones, because this method is the workhorse of practical aerofoil analysis and it has a shape of failure you should expect.

It is excellent at attached flow. Give it a section running somewhere in the middle of its range and it will hand you a drag figure good to a few percent, a transition location, and a lift corrected the right way by the right mechanism, in about a millisecond. That is a genuinely remarkable thing to get for the cost.

It is at its worst in exactly two places, and they are the two places that decide whether a wing is any good.

The top of the lift curve, for the reason in the section above. A method of this kind will tend to go on making lift past the angle at which the real section has given up. That is not a conservative error to have — a wing that never stalls on paper is very much the dangerous direction to be wrong in.

Low Reynolds number, where laminar separation bubbles appear and the assumption that the layer stays on the surface stops holding. Below about Re 5×10⁵ that is not a correction to apply, it is a different flow.

Both have the same root: the layer and the outer flow are being solved in turn rather than together. Fixing it properly means giving up the loop entirely and solving every boundary-layer unknown and every edge velocity as one system — which incidentally gives the wake behind the trailing edge somewhere to exist, and at the moment it has nowhere at all. That is a later post in this series.

Next #

Two posts in, there is now an instrument: shape and angle in, lift and drag and pressure out, fast enough to sweep. What there isn’t yet is any way of knowing whether the section it just measured is a good one.

That turns out to be a much less obvious question than it sounds, and it is the next post — what a wing section is actually for, why there is no such thing as a good aerofoil, and the one budget that every trade in the field is paid out of.

Every number in this post came out of the committed reference data or the solver while I was writing it. If any of it is wrong, I would like to know.

The question that broke Dave

Mon, 27 Jul 2026 00:00:00 +0000

I wrote about Dave here in May 2024 — a peer-to-peer network that disseminates packets of data, without accounts, without a token, and without anyone in charge. That article carries an edit note admitting it went out of date within months. This is what happened in those months, and why I’ve just thrown most of it away.

Underneath every version of it there’s one question, and it’s the reason I keep coming back to this instead of using something that exists:

Can you build a network that behaves like a shared pool of memory — anyone writes, anyone reads, the data stays around — with no payment system, no token, and no accounting of who is storing what? Just an incentive structure simple enough that nothing has to be coordinated at all?

The constraint is what makes it interesting. Filecoin, Arweave and Storj all answer the durability half with money and proofs of storage, and they work. Take money off the table and something else has to make a node want to keep your bytes, or the pool is only ever as good as the goodwill sitting in it.

Two years on, Dave has produced three different answers to that question. Two of them are in this post because I built them. The third I only recognised while writing it, and it had been sitting in the code since 2024.

The question in this article’s title is a much smaller one, and much more boring. It’s the one that broke it.

One thing to be clear about before any of the criticism, because it changes how you should read it. This was months of full-time work, and the decisions in it were deliberate — argued through, mostly written down, often revised more than once. What follows isn’t a list of things I hadn’t considered. It’s largely the opposite: mechanisms I designed carefully, which fail for reasons that only surface when you push a formula to its limit, or think like an attacker for five minutes, or run the thing at a scale I never reached. That’s a more useful kind of mistake to read about than carelessness, and a far more common one.

Where the code is now #

I’ve reset godave’s main branch back to commit 4bc87d5, from 19 November 2024, and tagged it pre-signed-dats. Everything after it — seventy-nine commits, three more weeks — lives on a branch called signed-dats. Nothing is lost, but the default branch is now the version I actually want people to read.

That version is 1,074 lines. It builds, and its tests pass. The version I abandoned in December is 3,573 lines across nine packages, and I could no longer hold it in my head.

The whole divergence is one commit, and you can see it in a single diff of the protobuf spec:

 enum Op {
-    DAT = 2;                    a thing that exists
+    PUT = 2;                    an operation on a named slot
 }
 message M {
-    bytes v = 3;   // value
-    bytes t = 4;   // time
-    bytes s = 5;   // salt
-    bytes w = 6;   // work
+    bytes datKey = 3;           addressable
+    bytes val = 4;
+    bytes time = 5;
+    bytes salt = 6;
+    bytes work = 7;
+    bytes pubKey = 8;           ownership
+    bytes sig = 9;              asymmetric crypto
 }

Addressability, ownership and signatures all arrive together. Before that commit a dat was a value, a time, a salt and a proof of work — content-addressed by its own hash, with no key and no owner. Nobody could update it, because there was no it to update; there was just the data, and the data was its own name.

The rename tells the story on its own. DAT is a thing that exists. PUT is an operation on a named slot. I changed one into the other in an afternoon and spent the next three weeks discovering what that cost.

What it did #

Each node keeps a table of peers and a table of dats. Every epoch it picks one dat at random and sends it to one peer at random. That’s the entire dissemination mechanism, and it’s the part I still like.

Storage was governed by something I called mass:

func Mass(work []byte, t time.Time) float64 {
	return float64(nzerobit(work)) * (1 / float64(time.Since(t).Milliseconds()))
}

Difficulty divided by age. The idea was that a dat with a harder proof of work should persist longer, and that everything should eventually decay unless somebody keeps re-publishing it. Trust worked the same way: when a peer sent you a dat you didn’t already have, its trust score went up by the mass of that dat, and trusted peers got picked more often for gossip.

That is the first answer to the question at the top: measure what a participant contributes, and pay them in the only currencies the network has — a share of its disk, and a share of its bandwidth. No token, no ledger, and a pleasing symmetry between the two halves.

The part that worked #

Before the things that didn’t, the part I’d write again unchanged — and which survives into the rewrite more or less intact.

Peer discovery and liveness are the same pair of messages. You send GETPEER; a well-behaved node replies PEER with a couple of addresses. If a peer hasn’t been heard from in DROP, it’s deleted. There’s no separate heartbeat, no separate bootstrap protocol, and no membership list — the packet that discovers peers is the packet that proves they’re alive.

That doubling-up turns out to do a third job as well, and it’s the one I’m most pleased with.

Nothing in the code mentions NAT. The traversal falls out of pinging everything you hear about, on a tick short enough that both sides do it at once.

Most nodes are behind a home router, and two of them can’t exchange UDP without help. The usual answer is a rendezvous server that tells both sides to transmit at the same moment, so each one’s outbound packet opens a return path through its own router. Dave has no such server and no code that mentions NAT at all. It gets there anyway, out of three unremarkable decisions:

// a packet from a stranger adds the stranger
_, ok := peers[pkfp]
if !ok { peers[pkfp] = &peer{pd: pkpd, fp: pkfp, added: time.Now(), seen: time.Now()} }

// gossiped addresses are added unconditionally
case dave.Op_PEER:
    for _, mpd := range pk.msg.Pds { if !ok { peers[mpdfp] = &peer{...} } }

// and the tick pings every peer in the table, verified or not
case <-pingTick.C:
    for pid, p := range peers {
        if !p.edge && time.Since(p.seen) > DROP { delete(peers, pid) }
        else if time.Since(p.peermsg) > PING { pktout <- &pkt{&dave.M{Op: dave.Op_GETPEER}, addr} }
    }

A newly gossiped peer has never sent a PEER message, so time.Since(p.peermsg) is enormous and it gets a GETPEER on the very next tick. Which means: once an edge node has told A about B and B about A, both sides start pinging each other within eight seconds, independently. Each one’s outbound GETPEER opens its own router for the other. Whichever packet arrives once both mappings exist gets through, and the receiver adds the sender on sight.

That’s a simultaneous open with nobody coordinating it. It works through full-cone and address- or port-restricted routers, which is most domestic kit; symmetric NAT and carrier-grade NAT still defeat it, because there the external port differs per destination and the address the edge saw is no use to anyone else. Good enough that I ran Dave across EC2 instances and Raspberry Pis at home without ever writing traversal code.

There’s a defence in there too, and it’s the detail I’d most like to have back if I’d lost it. Look at the two constants:

DROP        = 16 * time.Second // Time until silent peers are dropped from the peer table.
SHARE_DELAY = 20 * time.Second // Time until new peers are shared. Must be greater than DROP.

When you reply to a GETPEER, you only offer peers you’ve known for at least SHARE_DELAY. A peer that never answers is deleted at DROP. Sixteen is less than twenty, so an address that doesn’t respond is dropped before it ever becomes eligible to be passed on. You can inject a bogus address into one node’s table — anyone can, it’s a UDP packet — but it dies there, and no amount of injecting makes it spread. That’s peer-table poisoning closed off by an inequality between two constants rather than by any code that knows what poisoning is.

The same section has a rate limit on PEER messages: one accepted per PING interval, so a malicious peer can’t flood you with addresses to drown out the honest ones. I found that hole while writing the 2024 post and fixed it in the same afternoon, which is the only time writing an article has directly found me a bug.

Two honest limits on that, before I sound too pleased with myself. It stops addresses that are dead or invented; it does nothing about a live malicious node, which responds and therefore gets shared like anyone else. And discv5’s endpoint proof — the grown-up version of this — has exactly the same limitation, so at least the ceiling is in the right place.

I should be calibrated about the rest of it too. Combining discovery with liveness is what Kademlia’s FIND_NODE does. Refusing to gossip reputation is a call several deployed networks got wrong, but it isn’t new. Implicit hole punching happens incidentally in most DHTs — what’s unusual here is only that it was designed for rather than stumbled into. The delay inequality is the one I’d defend as properly neat, and even there the goal is standard; it’s getting it from an ordering between two timeouts rather than a round trip and some cryptography that I haven’t seen elsewhere.

So: not novel, mostly. I’ve stopped thinking that matters much. Landing independently on the same answers as the systems that got it right is the outcome you want from a first attempt at something, and it’s a better one than a genuinely new idea that doesn’t work.

None of this is what went wrong. The dissemination and storage side is where it comes apart, and it does so in three separate ways.

One packet is enough #

Look at what happens to mass as age approaches zero.

The cheapest legal dat is eight leading zero bits, about 256 hashes to find. At any age below 0.32 ms it is worth more than the entire trust cap.

Trust accumulates like this:

if novel {
    trust := peers[pkfp].trust
    if trust < MAXTRUST {
        trust += Mass(pk.msg.W, Btt(pk.msg.T))

MAXTRUST is 25. The cheapest dat the network accepts has eight leading zero bits, which costs about 256 hashes to find — call it free. Mass for that dat is 8 / age_in_ms, so it exceeds 25 at any age below 0.32 milliseconds.

An attacker mints a dat, hands it straight to the socket, and it arrives at a peer sub-millisecond on a LAN or from a colocated machine. It is novel by construction, because they invented it. One packet takes them from nothing to the trust ceiling.

And if the timestamp on the dat is the current millisecond, time.Since(t).Milliseconds() is zero, 1 / 0 in float64 is +Inf, and mass is infinite. The cap saves the arithmetic from doing any damage, but only by discarding whatever the number was supposed to mean.

The mistake underneath the arithmetic is the interesting one. I was rewarding peers for supplying novel data with valid proof of work, and I’d convinced myself that was evidence of useful contribution. It isn’t. It’s evidence of having spent CPU, and the attacker has CPU. A reputation metric only works if the thing it rewards is scarce to an adversary and abundant to an honest node, and “bytes nobody has seen before” is the opposite: trivially abundant to whoever is willing to make some up.

MAXTRUST exists precisely because I’d thought about farming — an uncapped score is obviously gameable, and the cap bounds what any peer can accumulate no matter how much they send. What defeats it is the interaction between the cap and the divergence, and that only shows up if you take the formula to its limit and ask what mass is worth at an age of zero. A cap you can reach in one packet isn’t a reputation system; it’s a single bit, and the bit says “has sent me something”, which every attacker can also say. The defence was real. It was aimed at the wrong axis.

Mass was recency wearing a costume #

Dats are stored in shards, and the shard is chosen by difficulty:

func keys(work []byte) (uint8, uint64) {
	return min(MAXWORK, nzerobit(work)) - MINWORK, xxhash.Sum64(work)
}

Every dat in shard k has exactly k + 8 leading zero bits. When the pruner keeps the most massive dats in a shard, the difficulty term in mass is the same constant for every dat it’s comparing — so the comparison collapses to one over age. Within a shard, mass is recency. The difficulty half of the formula does no work at all.

It isn’t quite doing nothing globally, and what it is doing is the best idea in the codebase — which I only worked out properly while writing this.

Difficulty picks your shard, and each shard has its own independent capacity. So a harder proof doesn’t buy you priority directly; it moves you into a less contended queue. And because every node prunes its own shards independently, the effect compounds across the network into a self-clearing price for permanence. Want a dat to survive? Do more work and land somewhere emptier. As more people do sixteen bits, that tier fills and the price drifts to twenty. Nobody sets the price, nobody verifies anything, no payment changes hands, and there is no ledger, no contract and no challenge-response anywhere in it. It falls out of local pruning decisions alone.

That’s worth sitting with, given where the project went. Every other way of buying durability in a decentralised store is heavy: Filecoin and Arweave use money and proofs of storage; Freenet uses popularity. The version of Dave I abandoned used storage challenges — the mechanism that dragged in replica tracking and reassignment and eventually ate the whole design. I already had a mechanism that gets a weaker version of the same property for nothing, and I walked past it to build the expensive one.

I can’t claim it’s unprecedented; I know the well-known systems rather than every paper. But I haven’t seen proof-of-work difficulty used as a storage tier with per-tier caps anywhere else, and I’d have expected to.

What defeats it in practice is mundane. Almost nobody does more work than the minimum, because nothing in the interface suggests a reason to, so everything piles into the first shard and fights over the same slots while fifty-five tiers above it sit empty. A market with one participant doesn’t clear. Combined with the mass formula collapsing to recency, the effective policy was “keep the most recent few thousand dats”, which is not what the documentation said. The interaction is subtle enough that I wrote both halves and still didn’t see it: partitioning by difficulty and then ranking by a formula containing difficulty looks like belt and braces, right up until you notice the second term is identical across everything it’s being asked to compare.

The idea deserves a better implementation than I gave it, and it’s separable from all of this — see below.

The anonymity claim was too strong #

From the README on that pinned commit:

Propagating dats in this way ensures that an adversary is unable to create a timing attack to discern the source of a dat, even with a broad view of network traffic.

The mechanism is sound and I still like it. Because every node emits one random dat to one random peer every epoch, at a constant rate, there’s no correlation between when a node received something and when it passes it on. A forwarder is genuinely indistinguishable from another forwarder.

But that’s forwarding. Origination is a different act, and the code does it differently:

case dave.Op_DAT:
	put(dats, &Dat{m.V, m.S, m.W, Btt(m.T)})
	for _, p := range rndpeers(peerList, trustSum, FANOUT, 0, 0) {
		pktout <- &pkt{m, addr}
	}

When an application writes a dat, it goes out to five peers immediately, and then repeats on a timer for however many rounds the caller asked for. The originator is the first node to transmit it and the most frequent. Random push hides everyone downstream of the source. It doesn’t hide the source, which is precisely what the sentence claims it does.

I don’t think I was being dishonest when I wrote that. I think I’d designed one mechanism carefully, and then let the claim it supported creep one step wider than the evidence.

The one I never solved #

Those three are things I got wrong. This one is different — it’s the problem I worked hardest on, never cracked, and still think is the most interesting thing in the project.

Every epoch a node sends one dat to one peer. Which dat?

v1’s answer is: pick one at random. That works beautifully at the scale I ran it — data moved around exactly as it should across a handful of machines. But the selection probability for any given dat is one in N, so as the store grows a newly-arrived dat waits proportionally longer for its turn. At a thousand dats it’s fine. At a million, a new dat is essentially never chosen, and propagation stops working at all.

The left panel is what v1 does. The right is what the problem actually wants — two mechanisms, because the two jobs have opposite requirements.

I tried a lot of things. The one still in the pinned code is a ring buffer: every novel dat is written to a thousand-entry ring, and each push epoch sends one dat from the ring as well as one from the store at large. That genuinely fixes the acute half — new data propagates immediately, however big the store gets.

What it doesn’t fix is the tail. A dat that falls out of the ring is back to one in N, so anything a node missed while offline, or anything that existed before it joined, only ever reaches it by luck. I also tried cuckoo filters: track what each peer has already seen, and don’t waste a push sending them something they’ve got. That work is on a branch called filter and it isn’t in the pinned version, which tells you how it went.

None of it worked because I was asking one mechanism to do two jobs with opposite requirements. Getting new data out fast wants a small working set, aggressive sending, and a short life. Repairing the long tail wants completeness, an unbounded working set, and can afford to be slow. A single random pick from a single pool is a compromise that is bad at both.

The literature has known this since 1987. Demers et al., Epidemic Algorithms for Replicated Database Maintenance, splits it exactly along that line: rumour mongering for new items — push them hard, and each time you push to someone who already has it, probabilistically stop, so the hot set limits itself and stays small — and anti-entropy in the background, periodically reconciling state with a random peer. You run both. My ring buffer is rumour mongering with its stopping rule missing; I had the instinct without the feedback that makes it terminate.

And the cuckoo filter was simply the wrong primitive, which is the part I’d most want to tell 2024 me. A membership filter answers “do you have this one?”, an item at a time. The question you actually need answered is “what are you missing?” — in one round trip, in bytes proportional to the size of the difference rather than the size of the set. That’s set reconciliation, and it’s a different tool entirely: Merkle trees over sorted ranges, as Cassandra and Dynamo use for repair; IBLTs, as Graphene uses for blocks; or PinSketch, which is what Bitcoin’s Erlay uses — having hit precisely this problem with transaction relay and solved it by pairing low-fanout flooding with periodic reconciliation.

The rewrite changes the shape of the question rather than answering it head-on. Flooding doesn’t select: every cell goes to every neighbour exactly once, deduplicated by the seen-set, so for new data there’s no choice to make and the problem doesn’t arise. What’s left is the tail — a node that was offline, or has just joined — and that is now a separate mechanism with its own name, the time-range sync. It’s the same split Demers describes, arrived at from the other direction.

Which puts the interesting work in exactly one place. That sync is currently naive and re-sends things you already have. Replacing it with real range-based reconciliation is the piece I most want to build, and it’ll be the first time in this project that the hard problem in front of me is one with a well-understood answer.

The question #

Here’s the one that actually killed it.

A dat is addressed by its own work hash. To retrieve one, you send GET with that hash, and any peer that has it replies. So: how do you get the hash?

You don’t. Not from the network. There’s no index, no enumeration, no namespace, nothing to browse. You can only ask for something you were already told about through some other channel — and if that channel exists, a lot of the reason for the network existing has quietly moved into it.

It’s worse than that, because GET doesn’t route. A read checks your own store first, and on a hit it returns without sending a single packet:

case dave.Op_GET:
	shardKey, datKey := keys(m.W)
	dat, ok := dats[shardKey][datKey]
	if ok {
		found = true
		apprecv <- &dat
	}
	if !found {
		for _, p := range rndpeers(peerList, trustSum, FANOUT, 0, 0) {
			pktout <- &pkt{&dave.M{Op: dave.Op_GET, W: m.W}, addr}
		}
	}

Left: what godave did on a local miss. Right: what the rewrite does instead. The second panel isn't a better lookup — it's the absence of one.

On a miss it goes to five peers picked at random and repeats on a timer. No key space, no distance metric, nothing pointing the query towards whoever holds the data.

That local check is worth dwelling on, because it looks like the answer and isn’t. It’s a cache lookup, and a cache is only as good as the thing it misses to. What sits in a node’s store is a random sample of recent traffic — you receive one random dat per peer per epoch, and the pruner discards everything past the shard capacity every minute. The chance that an arbitrary hash happens to be in there is low, and it falls as the network holds more data. The fallback is doing the real work, and the fallback is the broken part.

Which is also why I didn’t see this for months. On a dozen nodes on my desk, with capacity far exceeding the data, the local hit rate is close to one and the network path almost never fires — and when it does, five out of twelve peers is most of the network. Dave behaved correctly at every scale I ever ran it at. The lookup only fails once the data in the network exceeds what a single node can hold, which is precisely the point at which you would want it to work. I called Dave a distributed hash table in that first article. It was a distributed hash table with no table, and a small enough network to hide it.

So I gave dats a key, and an owner, and a signature — the commit at the top of this piece. Now data had a name you could form without being told it, and a namespace per public key you could enumerate. That is a genuine fix for a genuine problem, and I’d make the same call again with the information I had.

What followed was mechanical. If dats have keys, the sensible place to put a dat is near its key, so I added XOR distance. If placement is deterministic, some node is supposed to hold each dat, so I added replica tracking. If replicas are tracked, peers leaving means reassigning them, so I added that. If replicas can be reassigned, I need to know whether a peer is actually storing what it claims, so I added storage challenges. Values then outgrew a UDP datagram, so I added a TCP fallback — a second transport, with its own code path for the same operation.

Every one of those follows from the one above it. Not one of them was a bad decision on its own terms. Together they deleted the property the whole thing was for: once placement is deterministic, an observer knows exactly which nodes should hold what, and “random push for anonymity” no longer hides anything at all.

By December I was sketching a third system inside the second — direct messaging, layered on the keyed store, with recipient-keyed namespaces. I never committed it. That’s where I stopped.

Deleting the question #

I’m writing it again, in Zig, and the whole design rests on one rule I wrote down before any code:

A node never needs to know anything about the network beyond its immediate neighbours.

The point of the rule isn’t what it permits. It’s what it refuses. GET(key) means routing a query towards a node you can’t see, so there is no lookup — and with no lookup there is no placement, no replica set, no reassignment, and no storage challenge. The entire collapse is rejected by one sentence, and it’s rejected automatically, before I have a chance to talk myself into each step.

So how do you read anything, with no lookup?

You already have it. Everything floods to everyone, and reading is a query against your own disk. Following someone is a filter over your own store by their key. There’s no relay to ask, no query to route, and nothing to place, because nothing was ever anywhere in particular.

That is the same first move v1 made, with a completely different guarantee behind it. v1 checked a cache and fell back to a broken query; this checks a replica, and there is no fallback because none is needed. The difference isn’t the lookup — both start by looking in the same place. The difference is whether the design is working for or against that lookup succeeding. Random push and mass pruning actively hold a v1 node’s store down to a small sample; flooding and a retention window deliberately fill it.

That sounds like it can’t possibly scale, and it scales further than I expected. At a million users each posting ten times a day, a node carries about 0.6 Mbit/s and 2% of one core. Ingress doesn’t bind until several hundred million users on a normal home connection. Storage is the real cost, and it’s each node’s own choice of how far back to keep — a phone keeps a week, a machine with a spare disk keeps everything and becomes an archive.

That last part is deliberately not a protocol feature. There’s no mechanism to declare an archive, verify one, reward one or challenge one, and there is never going to be, because that mechanism is exactly what “storage challenges” were. Data stays available for as long as somebody chooses to keep it, which is how Usenet worked and how Nostr works. Any protocol promising more than that is either lying or has an accounting system that will eventually eat it, and I’ve now written the accounting system that ate mine.

That is the second answer, and it works by refusing the question. Nobody is doing anyone a favour, so there is nothing to pay for: you receive everything regardless, and you keep the part you want to read. The simplest possible incentive structure is no incentive structure — just self-interest, and no coordination because none is required.

It’s an honest answer and I think it’s the right one to build on. It also has a hole in it, which took me a while to see. Self-interest keeps the data somebody wants. It does nothing at all for data that nobody in particular wants yet — the unpopular, the archival, the thing you publish today that matters in five years. Under this scheme that data quietly evaporates, and there is no way to pay to keep it, because paying is the thing we removed.

Everything on the wire is one fixed size:

Every packet is exactly this shape, so size and structure leak nothing. A message that doesn't fit is split across several cells, each sealed independently.

There’s no recipient field. Every node tries to open every cell against its own key, and the authentication tag says whether it worked — so nobody, including the node holding a cell, can tell who it’s for. On my machine that costs 31 microseconds a cell, essentially all of it the X25519 scalar multiplication.

The peer layer comes across almost unchanged: discovery and liveness in one message pair, gossip only in replies, sharing gated behind a delay longer than the drop timeout, and reputation kept strictly local and never gossiped. The one correction is that identity becomes the public key rather than the address, because a table keyed by ip:port treats a node that moves as a stranger — and on IP, moving is the normal case, not the exception.

The traversal needs writing down this time, which it never was. Nothing in v1 records that the ping interval is load-bearing for anything but liveness, and the obvious optimisation — back off the tick, skip unverified peers — is correct for liveness and quietly fatal for hole punching. I nearly made exactly that change while reviewing this code, having read the same three lines and concluded the traversal was accidental. It wasn’t; it was the reason the discovery design looks the way it does. The mechanism survived, and the note explaining it didn’t.

Why Zig #

Partly because I want to, which is a sufficient reason for a hobby project. But there’s a real argument too, and it’s about where this code can run.

The protocol has no business needing an allocator on the receive path: a cell is a fixed 224 bytes, and opening one is arithmetic on a fixed-size buffer. Zig lets me write that once and compile it for a server or for a microcontroller with no operating system underneath it. I checked rather than assumed — the whole receive path, X25519 and BLAKE3 and ChaCha20-Poly1305 and the framing, cross-compiles to a Cortex-M4 in an eighteen-kilobyte object, with no allocator and no OS.

I’m not building the radio version now; I scoped this rewrite to IP so that it might actually get finished. But it means the door stays open, and it’s the one thing this project can do that the mature alternatives can’t.

Zig 0.16 is also recent enough that most of what’s written about the language online no longer compiles. main takes a std.process.Init, randomness comes through std.Io, std.time.Timer is gone. I’ve been finding these by compiling rather than by reading.

A loose end worth picking up #

The third answer is the one I didn’t recognise until I sat down to write this, and it had been in the code since 2024. It’s also the piece that fills the hole in the second: a way to buy permanence for data nobody else wants, paid for in work rather than money, with nothing keeping accounts.

I want to write it down properly, because it doesn’t need a peer-to-peer network to be interesting and I’d like to come back to it.

The mechanism. Every stored item carries a proof of work of difficulty d, in leading zero bits. The store is partitioned into tiers by d, and each tier has its own capacity. When a tier is full, it evicts within itself — oldest first is fine. A writer chooses d before publishing, so a writer is choosing which queue to stand in.

Why it should clear. Treat each tier as a queue with capacity C and arrival rate λ. In steady state, an item’s expected residency is T = C / λ — Little’s law, and for evict-oldest it’s exact. So the writer faces a straight trade: pay 2^d hashes, receive C_d / λ_d seconds of retention.

Little’s law is L = λW: the average number of items in any stable queue is the arrival rate times the average time each one spends in it. It assumes nothing whatsoever about the arrival distribution, which is why it is safe to lean on here — a store being filled by strangers is not going to be Poisson.

That’s a market, and it clears without anyone running it. If a tier gets popular its arrival rate climbs, residency falls, and the marginal writer moves up a bit. Cost doubles per bit while capacity stays put, so demand spreads itself across the tiers geometrically. No payment, no ledger, no proof of storage, no contract — just local eviction, on every node independently, adding up to a price.

Finding d leading zero bits takes 2^d hashes on average, and the distribution is geometric, so the standard deviation is about the same size as the mean. A writer aiming at a tier will sometimes pay several times over and sometimes get in cheap, which is an odd property for a price to have.

What was missing, and it’s the whole failure. A market needs four things. A cost function: proof of work, present. Differentiated goods: the tiers, present. Independent clearing per good: per-tier caps, present. And a price signal — which was entirely absent. Nothing in Dave ever told a writer that tier 0 was full and tier 12 was empty. There was no way to observe occupancy, so there was no way to act on it, so every writer rationally paid the minimum. The market had a price and no way to display it, and a market with one participant doesn’t clear.

The fix is small: have nodes report how full each of their tiers is, and let a writer sample a few and pick a difficulty from how long they want the thing to live. Occupancy is aggregate information about your own disk, so publishing it leaks very little.

How to test it with no network at all. This is why it’s worth doing separately. Simulate a population of writers with some distribution of how much they care about persistence, and one node’s store with tiers and caps. Feed arrivals in, evict, and measure whether residency time ends up proportional to cost. That’s a discrete-event simulation of a caching policy in a couple of hundred lines, and it answers the question completely — no sockets, no peers, no protocol.

Four things I’d want it to tell me:

How to divide capacity across tiers. v1 gave every tier the same cap, which wastes most of the disk when demand is concentrated at the bottom. Allocating in proportion to observed demand is self-defeating, because the tier you escape to grows to meet you. There’s a right answer here and I don’t know what it is.
Whether it oscillates. Publish occupancy and everyone can see the same empty tier. They all move there, it fills, they all move again. That’s textbook herding, and it probably needs damping or hysteresis to settle.
What lying buys you. A node that reports tier 20 as empty makes writers burn work for nothing. Sampling several nodes and taking a median helps, and a writer can check empirically whether the item actually survived, but this needs thinking about.
Whether the absolute numbers work at all. Dave’s minimum was eight bits — 256 hashes, which is nothing. Twenty bits is about a million, still only seconds of CPU, so the “expensive” tiers can be squatted cheaply. The scheme only prices anything if the top tiers are genuinely costly, which means calibrating against current hardware rather than against a number I picked in 2024.

If it survives that, it’s a retention policy any store could use, and it never needed to be part of a network in the first place.

What I don’t know #

Whether the question at the top has a good answer at all. Three attempts so far. Reward measured contribution — which failed, because anything I could measure locally was something an attacker could manufacture. Refuse the premise and rely on self-interest — which works, and leaves anything unpopular to rot. Price permanence in work and let local eviction clear a market nobody runs — which is untested, and is the only one of the three that’s an incentive structure in the sense I originally meant. One is built, one is a cautionary tale, and one is a weekend of simulation away from being either interesting or dead.

Whether spam is survivable. Flooding means anyone can inject, everyone must process, and now that data is retained, spam persists as well as propagates. My current answer is to ration rather than reward — per-neighbour rate limits, and a single shared budget that all unknown authors compete for, so ten thousand fake identities get one stranger’s worth of throughput between them. That’s an argument, not a result. So was the trust metric.

Whether anyone will use it. Almost certainly not many, and that’s fine. I want it to be good, and used by a handful of people I know, which is a different target from the one I was implicitly aiming at in 2024 and a much more reachable one.

What I do know is the thing I’d tell 2024 me. The first version had no rule that could reject a feature. Every addition answered “is this good?”, and each one honestly was. None of them ever had to answer “is this ours?” — and by the time that question would have been useful, there was no longer a clear enough idea of what “ours” meant to answer it with.

The code is at github.com/intob/godave, pinned where it was still one idea.

How we built Soaring Spirit

Sun, 26 Jul 2026 00:00:00 +0000

Soaring Spirit is my first company. It’s a Swiss GmbH, it runs speedflying and paragliding trips and coaching, and the people in it are pilots rather than programmers — Greg cooks and does the osteopathy, Jamie has more speedwing hours than almost anyone alive, Oli drives the bus and keeps the books, Nika gets us into the parts of Georgia nobody else can reach. So when I say “we built”, I mean the company is a we. The software is mine, and this post is about the software.

First commit was 2025-03-13. Sixteen months and 706 commits later it books trips, takes deposits and balances through Stripe, issues invoices, runs a coaching-availability calendar, hosts a flying syllabus, keeps a database of speedflying incidents scraped from the Swiss and French fatality reports, and logs in every user without ever having stored a password.

All of it is one Go binary, cross-compiled to arm64 and copied onto a Raspberry Pi in my house.

I want to write about the constraints rather than the features, because the constraints are the interesting part. Nearly every decision here was some version of “what is the smallest thing that can possibly work”, and a few of them were wrong in ways that took months to surface.

One binary, one machine #

The deploy script is worth quoting more or less in full, because it is the whole operations story:

GOOS=linux GOARCH=arm64 go build -ldflags "..." -o ss_linux_arm64 .
scp ss_linux_arm64 "$HOST:~/ss.new"
ssh "$HOST" "mv ~/ss.new ~/ss && sudo systemctl restart ss"

That’s it. Build, copy, restart. The Pi sits behind a Cloudflare Tunnel, so there is no public IP, no inbound port forwarding, no load balancer and no bill. Postgres runs on the same machine. pgbackrest ships backups to S3 on a schedule.

The frontend is deployed by rsync, and then the Cloudflare cache is purged. There is no build step for it at all — I’ll come back to that.

The obvious objection is that this is one machine and one power cut. That’s true, and it’s a real risk I’ve accepted rather than solved. The mitigation is entirely on the data side: the backups are off-site and verified, and the binary can be rebuilt and redeployed onto anything with an ARM chip in about ninety seconds. What I’d lose in a hardware failure is availability, not the business. For a company that sells six trips a year to people who mostly book after talking to us in person, that’s the right trade. It would be a bad trade for something with real-time revenue.

The less obvious cost is that one process serving both the API and the site means a deploy takes the whole site down for the second it takes systemd to restart. I’ve never minded this and I doubt anyone has noticed, but it’s the kind of thing that’s fine until it isn’t.

Ten dependencies #

The go.mod has ten direct requirements: a Postgres driver, Stripe, WebAuthn, an SMTP pool, three image libraries, and three golang.org/x packages. No web framework, no ORM, no dependency injection container, no code generation.

The router is 154 lines. Here is essentially all of it:

func (r *Router) ServeHTTP(w http.ResponseWriter, req *http.Request) {
	path := req.URL.Path
	for _, route := range r.routes {
		params, ok := matchPath(path, route.Segments)
		if ok && route.Method == req.Method {
			// ...inject params into the context, wrap in middleware, serve
		}
	}
	r.notFoundHandler.ServeHTTP(w, req)
}

A linear scan over about 110 routes, splitting the path on / and comparing segments. It is not clever and it does not need to be — this is a Raspberry Pi answering maybe a few thousand requests a day, and the linear scan is nowhere near the top of any profile I’ve taken. I wrote it before Go 1.22 gave the standard library pattern matching, and I’ve left it because it works and because the middleware ordering is explicit in a way I like.

I’m not going to pretend this is universally the right call. I wrote a router, a session store, an image pipeline and a mail queue that all exist as libraries, and each of them cost me a weekend I could have spent on the actual product. What I got back is that when something breaks at eleven at night before a trip, the stack trace goes through code I wrote, and I can read all of it. That has paid for itself several times.

The honest counterweight is the test suite: about 11,000 lines of Go tests against 19,000 lines of everything else, across 31 integration test files that run against a real Postgres in Docker Compose. Writing your own primitives is only defensible if you also write the tests that a library would have shipped with. I didn’t understand that at the start and the early code shows it.

No frontend build step #

The frontend is about 20,000 lines of hand-written JavaScript. Lit is vendored as a single file. There is no bundler, no transpiler, no npm run build, no node_modules in the deploy. Files are served from disk, so an edit is live on reload.

Routes are plain objects with a dynamic import:

{
	path: "trip/:id",
	component: "ss-trip-detail",
	action: async () => await import("./components/app/trip/trip-detail.js")
}

The browser does the code splitting, because ES modules already are code splitting. Every component is a LitElement with its styles in a tagged template literal next to its markup.

What this buys is that the edit-to-live loop is a file save, and that there is no build to break. What it costs is that node --check is the entire compile step — nothing catches a typo in a property name until it renders wrong. Over sixteen months, that has cost me less than I expected. Shadow DOM turns out to do most of what a build step’s CSS scoping would.

It has one wart I still haven’t fixed properly. The server pre-renders the SPA shell’s <head>, so it keeps its own table of route titles and descriptions in app/prerender.go, mirroring the one in site/index.js. Two hand-kept copies of the same data. It’s documented in three places and I’ve still got it wrong once.

Money is where the discipline goes #

Everything above is “how little can I get away with”. Bookings are where that stops.

A booking moves through ten states — DRAFT, IN_REVIEW, AWAITING_DEPOSIT, DEPOSIT_PAID, DEPOSIT_TO_REFUND, DEPOSIT_REFUNDED, AWAITING_BALANCE, FINALISED, VOID, REJECTED — and the legal transitions are an explicit switch, not a convention:

switch current {
case BookingStateInReview:
	if next == BookingStateAwaitingDeposit || next == BookingStateRejected || next == BookingStateVoid {
		return nil
	}
case BookingStateDepositPaid:
	if next == BookingStateDepositToRefund || next == BookingStateAwaitingBalance {
		return nil
	}
// ...
}
return errInvalidTransition

Every transition goes through one function, which opens a transaction, takes SELECT state FROM app.bookings WHERE id = $1 FOR UPDATE, validates, and writes with the old state in the WHERE clause. The row lock is there because the check-then-write without it is a textbook TOCTOU: two concurrent callers both read AWAITING_DEPOSIT, both decide the transition is legal, both proceed.

The transaction on its own wouldn’t have saved it. Postgres defaults to READ COMMITTED, where both callers can read AWAITING_DEPOSIT and both carry on quite happily. FOR UPDATE is the part that serialises them: the second one blocks on the row until the first commits, and then reads what it actually wrote.

There’s one flag on it that took me a while to get right. A transition to the state you’re already in is treated as a no-op rather than an error, because Stripe redelivers webhooks and a retried handler re-observing its own applied state is normal. But a caller that’s about to issue an actual refund on the strength of that transition cannot accept a no-op — two callers racing there means refunding twice. So RequireStateChange turns the no-op back into a conflict for exactly those callers:

// Set this for callers about to perform an external side effect (e.g. a
// Stripe refund) on the strength of this transition, so a second,
// concurrent caller that raced past the same check doesn't repeat the
// side effect.

The retry loop #

The bug I’m most glad to have found is not in that machine, it’s in what surrounds it.

Stripe redelivers any webhook that doesn’t return 2xx, backing off over several days. That’s correct for a transient failure — database down, network blip — where a later attempt might work. It is actively harmful for a deterministic rejection.

During the host merge in July, a past invoice.paid event got resent for a booking that had already reached FINALISED. The handler read it as a deposit payment and attempted FINALISED → DEPOSIT_PAID. The state machine rejected it, correctly. The handler returned 500. Stripe retried. The state machine rejected it again, identically, because the state machine is deterministic — that’s the entire point of it.

So it retried for days.

The fix is a distinction rather than a patch: an invalid-transition error is now a sentinel that the webhook handler acknowledges with a 2xx, because “received, nothing to do” is the honest answer when the booking is already in its terminal state. Only genuinely transient errors get a 5xx. It’s obvious in hindsight and it was not obvious at all in advance, and it’s the clearest lesson the project has taught me — retry semantics are a property of the error, not of the endpoint.

Let the database say no #

The application checks that two coaching sessions don’t overlap. That check is a race, always, no matter how carefully it’s written, because two requests can pass it concurrently before either commits.

btree_gist is needed because the constraint mixes two kinds of comparison: plain equality on contact_id and overlap on the date range. A stock gist index handles the range; the extension is what lets an ordinary scalar sit beside it in the same index.

Postgres will just do this for you:

CREATE EXTENSION IF NOT EXISTS btree_gist;

ALTER TABLE app.coaching_sessions
    ADD CONSTRAINT excl_coaching_sessions_overlap
    EXCLUDE USING gist (
        contact_id WITH =,
        daterange(window_start, window_end, '[]') WITH &&
    ) WHERE (status IN ('PENDING', 'CONFIRMED'));

The application check stays, because it produces a friendly error message at the right moment in the UI. The constraint is the thing that’s actually true. I’ve come round to thinking this is the correct shape for most validation: the readable error in the app, the authority in the schema.

The same instinct runs through the rest of the schema — seventy-odd numbered migration files, currently up to 0193. Images are content-addressed by SHA-256 and stored as bytea, so uploading the same photo twice is a no-op; they’re resized and encoded to both JPEG and AVIF in-process, on the Pi, which is slower than I’d like and has never been slow enough to matter.

My favourite migration is 0130_remove_passwords.sql, which drops six columns and is the entire content of the file. Login is passkeys, or a six-digit code by email. Nothing to leak.

Two ways to get email wrong #

Until three weeks ago, sending mail went straight to SMTP inline, and it managed to be wrong in two opposite directions at once.

Most call sites sent from a detached goroutine and logged the error. A momentary SMTP outage meant a user silently never learned their booking was approved — no record, no retry, no way to answer “did they actually get told?”.

The Stripe webhook did the opposite: it returned the send error up, so a failed email failed the whole webhook, and Stripe retried the hook and re-ran every side effect around it. A bounced email could void an invoice twice.

Both are the same bug — treating delivery as part of the transaction that caused it. The fix is a transactional outbox: write the intent to a table, drain it from a worker with backoff and a max attempt count, and key rows on an idempotency key so a redelivered webhook produces one receipt rather than two. Delivered rows are kept for ninety days rather than deleted, because “did that user get told?” should be a query.

The thing that stopped running #

For eleven days in July, every page on the site told Google it was a duplicate of the homepage.

The SPA ships one generic <title>, description and canonical in index.html, and rewrites them in JavaScript once the router resolves. That’s invisible to anything that doesn’t execute JavaScript, so to fix it for crawlers there was a Cloudflare Pages Function — site/functions/_middleware.js — which filled the head in server-side. It worked. It had worked since the day I wrote it.

Added 2026-07-01 with the rest of the SEO work — sitemap, robots.txt, per-route meta. All of that survived the move. The middleware was the only piece that depended on where the site was served from.

Then on 15 July I moved site hosting off Pages and onto the Pi, so the site would deploy by rsync and sit behind the same cache rules as everything else. That was a good change and I’d make it again. It also switched the middleware off, because a Pages Function is a thing Pages does, not a file that runs wherever it happens to sit.

Nothing errored. Nothing logged. site/functions/_middleware.js stayed in the repo looking exactly as maintained as it had the week before, and every page quietly went back to shipping <link rel="canonical" href="https://soaringspirit.ch/"> — each one declaring itself a duplicate of the homepage.

Eleven days is about as cheap as this class of bug gets, and I only caught it because I was planning the host merge and went to check what the middleware would do afterwards. What bothers me isn’t the eleven days, it’s that nothing could have caught it. There’s no test for “this file is still in a position to execute”. The deploy got faster and simpler, and what broke was a capability rather than a line of code.

The fix folded into a merge I wanted anyway. api.soaringspirit.ch and soaringspirit.ch were two processes from the same binary on the same machine behind the same tunnel, and the split existed for no reason beyond history — while costing a CORS middleware, a preflight round-trip on every non-simple request, and a sitemap on the wrong host. Moving the API under /api on the site origin deleted the CORS code entirely and let the head for /trip/:id be rendered from a direct database query instead of an HTTP call to ourselves.

Sixteen route collisions had to be resolved to do it, and http.StripPrefix meant all 110 route registrations stayed exactly as written. Everyone gets the same correct head now — no user-agent sniffing, because varying HTML by user agent would fragment the edge cache and there’s no upside.

What I’d tell myself at the first commit #

Write the integration tests before the third feature, not after the thirtieth. They’re the only reason the no-framework approach is defensible, and I built up months of debt before I understood that.

Push correctness into the schema earlier. Every exclusion constraint and foreign key I added late replaced application code I’d been quietly trusting.

And when you move a platform, inventory what the old one was doing for you. Migration notes list what you’re carrying across, and I wrote good ones. What they don’t list is the work the old platform did for free, because nobody had to think about it while it worked — that’s exactly the part that leaves no diff, throws no error, and shows up weeks later in a search result. A file can keep being correct long after it has stopped being reachable, and those two things fail completely independently.

The company is sixteen months old and the software has never lost a booking. For a first attempt at both, I’ll take it.

Two stories about air

Tue, 28 Apr 2026 00:00:00 +0000

I’ve been flying soft wings for years without really understanding the sections they’re built from. Paragliders, speedwings, kites — they all live or die on the shape of a slice through the wing, and I’ve mostly taken that shape on trust. So I’ve started building something to fix that: a simulation backend called Foilworks, which takes a wing section, an angle, and an airspeed, and tells you what the air does about it.

This is the first of what I expect to be quite a few posts. The early ones are about the field rather than about my code — the physics has to come first, or none of the rest means anything. My maths is not strong, so I’ve had to build the intuition first and let the equations follow, and that’s how I’m going to explain it. If you can picture air moving, you can follow all of this.

Where friction is, and where it isn’t #

Air behaves in two completely different ways depending on how close to the wing you are, and almost every practical aerodynamics tool is built on that split.

Nearly everywhere, air is so slippery that friction may as well not exist. It just has to flow around the shape rather than through it. Solve that, and you get the pressure everywhere on the surface — and pressure is where lift comes from.

Then there’s a film glued to the surface, about one percent of the chord thick, where friction is the only thing that matters. That film is where drag comes from, and where stall comes from.

This post is entirely about the first story. It’s the easier one, it runs in about a millisecond, and it gets you lift and pressure. The second one is much harder, I’m still fighting it, and it gets its own post.

A wing is a list of points #

The first job is turning “NACA 4412” into something a computer can chew on. You walk all the way around the section and write down coordinates, then join consecutive points with straight lines. Those straight segments are the panels.

Chord is always normalised to 1, so the nose is at x=0 and the tail at x=1, whatever the real wing measures.

Two details in that picture do real work.

The points aren’t evenly spaced. They’re bunched at the nose and the tail using a cosine distribution. Over the middle of a section the flow does almost nothing interesting — the pressure changes gently and slowly. Around the nose it goes berserk: air comes to a dead stop, then accelerates to well above freestream speed within a couple of percent of chord. Even spacing would spend most of your panels describing the boring part and smear the most important feature on the whole surface.

And the direction you walk matters. I go clockwise — tail, along the bottom, round the nose, back along the top. Every panel computes its outward normal as “the direction I’m travelling, rotated ninety degrees”. Walk the other way and every normal points into the wing, and the sign of the lift flips. I have this written in capital letters in my notes.

The trick: throw the wing away #

Here’s where it gets clever, and where I had to stop and think for a while.

You’d assume you solve the flow around a solid object. You don’t. You delete the object entirely and replace its surface with a vortex sheet — imagine laying a carpet of tiny spinning rollers along where the skin used to be. Each roller drags the air around with it.

The rollers have no physical existence. They're a device for producing a flow that happens to follow the shape you care about.

Now ask one question: how hard does each roller have to spin so that the wind plus all the rollers together never pushes air through the line where the skin used to be?

Answer that, and you’ve solved the flow — because a flow that doesn’t cross the skin is indistinguishable from a flow around a solid wing.

Why spinning rollers, specifically? Because spin is what makes lift. You could lay down a carpet of little sprinklers instead, blowing air outwards, and they’d make the shape appear fatter — but they’d generate exactly zero lift no matter how you tuned them. Net circulation around the section is lift. Vorticity is the only ingredient that can produce it.

Chop the surface into N panels, let the spin vary linearly along each one, and the unknowns become the values at the N+1 corners. Then write one equation per panel: at this panel’s midpoint, the flow perpendicular to the surface is zero.

That’s N equations and N+1 unknowns. One short.

The bit where nature has to choose #

That missing equation isn’t a bookkeeping mistake. It’s the most interesting thing in the whole method.

Frictionless flow theory genuinely does not have a unique answer. There’s an entire family of valid solutions around any wing section — one for every possible amount of circulation — and every single one satisfies “no air through the skin” perfectly. In one of them the air whips around the sharp trailing edge from below to above. In another it goes the other way. In exactly one, the flow leaves the trailing edge smoothly, both streams merging and departing together.

Both are mathematically valid. Only the right-hand one is ever observed.

Nature picks the smooth one. It picks it because real air has viscosity, and viscosity flatly refuses to let flow turn a sharp corner at infinite speed. The first attempt to do so sheds a vortex downstream — you can watch this happen in a water tank — and that departing vortex leaves behind precisely the circulation needed to make the trailing edge smooth ever after.

So the missing equation is a fact about sticky air, smuggled into a model that has no friction in it anywhere. Written down, it is one line — the Kutta condition:

γ₀ + γₙ = 0

The spin strength just under the tail, plus the spin strength just above it, equals zero. Since the sheet strength turns out to be the local surface speed, that says the speeds on the two sides of the trailing edge match: no jump, no infinite turn.

This is where the lift comes from. Everything else is bookkeeping. When I had a sign error early on and my sections produced negative lift, this line was where the bug was.

It’s all one linear system #

With that last equation in place you have a square system: a matrix, a vector of unknown spin strengths, and a right-hand side.

N rows of “no air through this panel”, plus one row of physics.

Each entry of the matrix answers a very concrete question: if roller j were spinning at unit strength, how much air would it push through panel i? That’s pure geometry — there’s a closed-form expression involving a logarithm and an arctangent, and it’s the same on every panel method ever written.

The bit worth noticing is what isn’t in the matrix. The angle of attack appears nowhere in it. It only touches the right-hand side. So the expensive half — factorising the matrix, which is the cubic-time part — gets done once per shape, and each new angle is a cheap back-substitution. Running a whole polar sweep across twenty angles costs roughly the same as solving one.

Getting the answer back out #

Once you’ve solved for the spin strengths, you might expect to have to sum up every roller’s contribution at every point to find the surface speed. You don’t, and this is my favourite part.

A vortex sheet is defined as a jump in tangential velocity across it. The solution was constructed so the air inside the imaginary body is perfectly still. So the jump across the sheet is simply the speed outside it. The strength you solved for is the speed of the air sliding over the skin.

Then Bernoulli turns speed into pressure — fast air means low pressure — and the whole thing collapses to one line, evaluated at each panel’s midpoint:

Cp = 1 − γ²

Cp is the pressure coefficient: local pressure minus ambient, divided by the pressure you’d get by bringing the freestream to a dead stop. Dividing through like that means it doesn’t depend on how fast you’re going or how thick the air is — the same curve holds at 10 m/s at sea level and 30 m/s at three thousand metres. It’s the universal currency of this field.

Here’s what actually comes out:

Straight out of the solver: 198 panels, Cl = 1.2386. Everyone plots Cp upside down, so that the area between the curves reads as lift.

Three things to read off it. Cp = +1, bottom left, is air brought completely to rest — the stagnation point, and the highest value physically possible. The big negative spike just behind the nose is the suction peak, where air is moving far faster than freestream and pressure has dropped well below ambient; most of the lift lives there. And the long climb from the peak back towards zero at the tail is the pressure recovery — air being asked to move into rising pressure. That climb is the whole ballgame for the second story, because it’s what eventually pushes the boundary layer into separating.

The drag that isn’t drag #

The solver reports a drag number. It’s a lie, and I’ve had to be quite disciplined about saying so.

In genuinely frictionless flow, a closed body experiences no drag at all. The pressure pushing back on the front is exactly cancelled by pressure pushing forward on the rear. This is d’Alembert’s paradox, and it bothered people for about a century before boundary layers were understood.

So the drag my panel solver computes is not physical. It’s the residue of chopping a smooth curve into flat segments — a measure of how good my panelling is, nothing more. On the run above it’s 0.000172, which is a couple of orders of magnitude smaller than the real drag of that section. If it ever gets large, that tells me my resolution is bad, not that I’ve designed a draggy wing.

I have a rule written into the project: never report this number as drag. It would be very easy to ship something that looks like a working aerodynamics tool and is quietly nonsense.

The check that keeps me honest #

The pressure integration gives lift. But there’s a completely independent route to the same number — Kutta–Joukowski, which says lift is just twice the total circulation. Two different calculations from the same solution:

Kutta–Joukowski is L = ρVΓ. The twice is an artefact of the coefficient form: divide through by ½ρV²c, set V and c to 1, and Cl = 2Γ. Nearly every stray factor of two in this field is somebody’s ½ρV² in disguise.

cl      = 1.2386
cl_circ = 1.2387

If those ever diverge by more than a percent, the solver attaches a warning to its own output. It’s a cheap and brutal test: the two agree only if the solution is actually right, and it costs almost nothing because the second number falls out of the solution you already have.

One result that looks like a bug and isn’t: my lift slope comes out at 6.86 per radian, against the textbook thin-aerofoil value of 2π ≈ 6.28. That’s 9.2% high. It’s meant to be. Thin-aerofoil theory linearises the thickness away, and thickness genuinely adds lift. Every thick-section panel method overshoots the textbook line by about this much. It took me an embarrassingly long time to stop trying to “fix” it.

What it can’t do #

This is worth being blunt about, because a plausible-looking wrong answer is worse than no answer.

There is no viscosity in any of this. So: no real drag, no stall, no separation, no transition from laminar to turbulent flow, no compressibility. You cannot get a glide ratio out of it, because you cannot get drag out of it. You cannot find the stall angle, because nothing in the model knows how to stall — push the angle up and the lift just keeps rising happily, forever, which real wings famously do not.

What it does give you, and gives you fast and accurately, is lift and the pressure distribution for attached flow. That turns out to be exactly the input the second story needs.

Next #

The other story: the boundary layer, which is where drag and stall actually live. It’s a much nastier problem, and not only because friction is harder than no friction — the thin film and the outer flow each need the other’s answer before either can start, so the two have to be talked into agreeing.

The diagrams in this post were generated from the solver itself rather than drawn by hand, which felt like the right way to do it. If any of it is wrong, I’d genuinely like to know.

Progress on the Peregrine HTTP server

Thu, 09 Jan 2025 00:00:00 +0000

I’m implementing a HTTP server from scratch in Zig, as a fun way to learn the Zig systems programming language, and also to get more comfortable and familiar with some modern kernel features that make highly performant network IO possible.

I started this project on the 24th December 2024, and within a couple of days, I had a bare-bones TCP server that could parse and respond to HTTP requests. Initially, I followed Karl Seguin’s excellent guide to creating a TCP server in Zig. If you’re interested in learning Zig, after solving some Ziglings, I suggest that you look at Karl’s blog posts on Zig.

After some light optimisation work, I wrote some example applications because I know that the best way to test the usability of your library is to use it.

Initially, the server was initialised with an on_request(request, response) void handler function. I found that this approach made it difficult to create a stateful request handler. After trying a VTable-based interface, I settled on the more idiomatic Zig generic pattern, leveraging Zig’s comptime metaprogramming. Basically, you just call Server(MyHandler).init(config), where Server is a function that returns a type of Server that has your handler built in at compile time. You then simply define your handler as the implementation of an interface, containing at least the required methods. If the user messes anything up, the program will simply fail to compile, and some useful error will be output.

Anyway, with the nicer API, I implemented a simple counter server where the handler increments a counter on each request, and responds with the current value. I then wanted to see my server doing something useful, so I wrote a directory server, and served this website locally. To my delight, the server was working great, and the implementation of the DirServer helper required no significant changes to the server. It was fun to see the server working well, serving binaries and text files that were rendered correctly by the browser.

At this point, the server was slightly more performant than Go’s stdlib HTTP server. Go’s HTTP server is a good benchmark, because it’s well written and used extensively in production. Also, it supports HTTP/2 and TLS out of the box, which is really great. Unfortunately, Peregrine does not yet support HTTP/2 or TLS. However, Peregrine does beat Go’s HTTP server in one regard, and that is built in support for WebSockets. Peregrine features full support for WebSockets, and adding WebSocket support to your application is as easy as handling the connection upgrade by calling the upgrade helper function, and implementing the WS handler methods in your handler. It really could not be easier, and I’m quite happy with how that turned out.

Slightly more performant than Go’s HTTP server? There is no way that we can be happy with that… after all, many HTTP servers outperform Go’s.

From my own tests, NGINX only marginally outperforms Go’s, even with some tuning such as enabling Kqueue/Epoll and setting the thread count equal to the CPU core count. That surprised me, because as far as I know, Go relies on its runtime scheduler, and does not use Kqueue/Epoll for socket readiness notifications, which should eliminate wasted CPU time spent polling the sockets.

At the front of the pack is Zap, built on Facil.io. Zap is basically Zig bindings for the HTTP server part of Facil.io. Facil.io is a marvellous networking library written in C. The more I study and benchmark Facil.io, the more I’ve come to appreciate its stability and performance. On my M2 Pro, Zap will process in excess of 170K static GET requests per second with 1000 connections. I’ve not yet found a HTTP server that outperforms this with 1000 connections. Please let me know if there is one.

As a baseline, on 28th December, 4 days after the start of the project, Peregrine was doing around 86K static GET requests per second with 1000 connections. Similar to Go’s stdlib HTTP server and also similar to (tuned) NGINX. At this point, Zap/Facil.io does more than twice the throughput. We have a lot of work to do…

Step 1: Improve the handler #

As I mentioned earlier, the nature of the library’s on_request hook made implementing a stateful handler inconvenient at best. This was at the top of my list. I implemented a VTable-based interface, but I didn’t like that the user needed to declare the VTable in their handler:

// commit 0beb0019df043abb43e924d4aa8234f479eba607
// ./example/basic.zig
const MyHandler = struct {
    const vtable: pereg.HandlerVTable = .{
        .handle = handle,
    };

    pub fn handle(ptr: *anyopaque, req: *pereg.Request, resp: *pereg.Response) void {
        const self = @as(*@This(), @alignCast(@ptrCast(ptr)));
        self.handleWithError(req, resp) catch |err| {
            std.debug.print("error handling request: {any}\n", .{err});
        };
    }

    inline fn handleWithError(_: *@This(), _: *pereg.Request, resp: *pereg.Response) !void {
        _ = try resp.setBody("Kawww\n");
        const len_header = try pereg.Header.init(.{ .key = "Content-Length", .value = "6" });
        try resp.headers.append(len_header);
    }
};

This version actually performed quite well, doing a little over 160K requests per second. That’s nearly a 2x improvement from our baseline! In retrospect, I’m surprised that this version performs so well, because the indirection of the VTable lookup should cost at least a few nanoseconds, not to mention that this indirection should increase CPU cache misses for calls to handler methods. It seems that function pointers are even more costly than VTables. Anyway, this performs well, but it’s quite ugly, and does not make use of what Zig has to offer. We need a better way to ‘inject’ our handler into the server.

// commit 2d13a14cd5ed77de040bea003ad37a55020fb54d
// ./example/basic.zig
const std = @import("std");
const pereg = @import("peregrine");

const MyHandler = struct {
    pub fn init(allocator: std.mem.Allocator) !*@This() {
        return try allocator.create(@This());
    }

    pub fn deinit(_: *@This()) void {}

    pub fn handle(self: *@This(), req: *pereg.Request, resp: *pereg.Response) void {
        self.handleWithError(req, resp) catch |err| {
            std.debug.print("error handling request: {any}\n", .{err});
        };
    }

    fn handleWithError(_: *@This(), _: *pereg.Request, resp: *pereg.Response) !void {
        _ = try resp.setBody("Kawww\n");
        const len_header = try pereg.Header.init(.{ .key = "Content-Length", .value = "6" });
        try resp.headers.append(len_header);
    }
};

pub fn main() !void {
    var gpa = std.heap.GeneralPurposeAllocator(.{}){};
    const allocator = gpa.allocator();
    defer _ = gpa.deinit();
    const srv = try pereg.Server(MyHandler).init(allocator, 3000, .{});
    std.debug.print("listening on 0.0.0.0:3000\n", .{});
    try srv.start(); // Blocks if there is no error
}

By the way, sorry for @This() in the method signatures, this was pasted as committed. I’ve since started declaring const Self = @This(); to clean up method signatures. This looks much better. Our handler simply implements the 3 required methods; init, deinit, and handle. Our handler is then given to the Server by making Server a function that returns a type. Our handler is passed to the ‘constructor’ as a parameter. Under the hood:

// commit 2d13a14cd5ed77de040bea003ad37a55020fb54d
// ./src/server.zig
pub fn Server(comptime Handler: type) type {
    return struct {
        comptime {
            if (!@hasDecl(Handler, "init")) {
                @compileError("Handler must implement init(std.mem.Allocator) !*@This()");
            }
            if (!@hasDecl(Handler, "deinit")) {
                @compileError("Handler must implement deinit(*@This()) void");
            }
            if (!@hasDecl(Handler, "handle")) {
                @compileError("Handle must implement handle(*@This(), *Request, *Response) void");
            }
        }

        handler: *Handler,
        // ...
    };
}

Pretty simple, right? As it should be. As you can see, we will get a compile error if our handler is missing any of the 3 required methods.

Step 2: optimise the request parser #

Being a text protocol, efficient parsing of a HTTP request comes down to well-implemented buffered reading from the socket, and optimised parsing operations. The parts of a HTTP request are delimited by CRLF (sometimes clients may use bare LF line endings). The first line contains the request method, path, and HTTP version. The following lines contain headers, until an empty line. If present, the request body would follow. The length would be specified in a Content-Length header. For example (without body):

GET /path/to/resource HTTP/1.1\r\n
Host: localhost\r\n
User-Agent: Microsoft Internet Explorer 6\r\n
Accept: */*\r\n
Connection: keep-alive\r\n
\r\n

As a side note, the Connection header is the client’s request for the server to either close or keep-alive the TCP connection. HTTP/1.1 introduced the keep-alive mechanism, allowing connections to be reused. This greatly improves HTTP’s efficiency, especially as modern websites typically consist of many resources that are requested individually. In HTTP/1.0, clients had to create a new connection for each request. Our server must handle both HTTP/1.0 and HTTP/1.1 as a bare minimum. HTTP/2 and HTTP/3 would be great to have, but these are binary protocols, and must be upgraded to from a HTTP/1.1 connection. Additionally, HTTP/3 is not over TCP, but QUIC, which is over UDP. Support for HTTP/2 and 3 may come later.

So, how to parse this request? Logically, we should read it line-by-line. So far, I’ve found that efficient request parsing is about minimising search and copy operations.

If you’re interested, check out ./src/reader.zig. I’ll spare you a complicated code-dump.

Method, path and version #

In essence, the method and version are parsed first, at the beginning and the end of the first line. Parsing the method was originally done using a compile-time generated lookup table of 8-byte masks. SIMD operations were then used to compare the first 8 bytes of the request with the lookup table. In practice, I was not happy with how some methods were parsed fairly quickly (9ns), but other methods further down the lookup table took considerably longer (over 30ns) due them requiring more comparisons.

While a few nanoseconds may seem negligible, I’m trying to keep request parsing under 2000 CPU cycles. That’s around 600ns on my M2 Pro.

I also tried a switch of the method length before byte comparisons, but this necessitated finding the first ’ ‘, which cost more than the saving of narrowing down the number of possible methods.

Today, I settled on the following:

pub const Method = enum(u4) {
    GET,
    PUT,
    HEAD,
    POST,
    PATCH,
    TRACE,
    DELETE,
    CONNECT,
    OPTIONS,

    pub fn parse(bytes: []const u8) !Method {
        if (bytes.len < 3) return error.UnsupportedMethod;
        return switch (bytes[0]) {
            'G' => .GET,
            'P' => switch (bytes[1]) {
                'O' => .POST,
                'U' => .PUT,
                'A' => .PATCH,
                else => error.UnsupportedMethod,
            },
            'H' => .HEAD,
            'T' => .TRACE,
            'D' => .DELETE,
            'C' => .CONNECT,
            'O' => .OPTIONS,
            else => error.UnsupportedMethod,
        };
    }

    pub fn toLength(self: Method) usize {
        return switch (self) {
            .GET => 3,
            .PUT => 3,
            .HEAD => 4,
            .POST => 4,
            .PATCH => 5,
            .TRACE => 5,
            .DELETE => 6,
            .CONNECT => 7,
            .OPTIONS => 7,
        };
    }
};

As beautiful as the comptime-generated lookup table was, I found that the overhead of initialising SIMD vectors was too expensive for this use-case. The above solution is much faster, with methods parsed between 6ns and 9ns. We simply give it at least 3 bytes, and in the fewest operations possible, it will return the method.

We then parse the HTTP version. Not worthy of a code-dump, we simply take the last 8 bytes of the request line, and check that the 6th byte is ‘1’ (HTTP/1), the 7th byte is ‘.’, and switch over the 8th byte to return the version enum 1.0 or 1.1. You can see it in ./src/version.zig.

Next is extracting the path. You probably noticed the toLength method in the Method enum. We know that everything between the method and the version (minus the spaces) is the path and query. We also know that the version is 8 bytes in length. Therefore, by calling toLength() on the method, we have the position of the beginning of the path without a single scalar search for the delimiting spaces (’ ‘).

But what about separating the query from the path? We could simply defer this until (if ever) the query is needed, omitting the search for ‘?’, but I found that this hurt the usability of the library. I may change this again because searching for ‘?’ is relatively expensive, and not always required. For now, the path and query are separated, but the query is never parsed, and instead stored in its raw URL-encoded form. Calling request.parseQuery() will populate a hash map in the request.

Parsing headers #

Here, we simply read the headers line-by-line, parsing them using a scalar search for ‘:’ which is the delimiter for the key-value pair. The header is simply a struct with key and value fields. If you’re looking at ./src/header.zig and wondering why Header contains fixed-sized arrays and length fields… Requests are pre-allocated with a fixed-sized array of 32 Headers. This means that we don’t need to allocate headers for each request.

Some of the most important optimisations can manifest as quite ugly ‘hacky’ tricks. Such is the case for the Connection header. The Connection header is a little special, because it isn’t really for an application/library user to think about, but a client request for the server to handle the connection in a specific way. The value is normally either “keep-alive” or “close”. The server may still choose to close the connection, for example, if the limit of connection requests has been reached. Therefore, when we respond to a HTTP/1.1 request, we most often need to evaluate the Connection header. Initially, I was searching for the Connection header while generating the response, but I noticed that we could save a few nanoseconds by moving this into the header parsing step, as we’re already iterating over the headers. This was as simple as adding a boolean flag to the request keep_alive. Looking at the implementation at time of writing, it’s so ugly that I don’t want to dump the code here! If you want, check it out in the parseHeaders method in ./src/reader.zig. Basically, we do a quick process of elimination by raw length, before checking if the value starts with ‘cl’ for ‘close’. Notice the conn_header_found flag in the method to prevent wasted operations after we’ve found it.

SIMD for `reader.readLine` #

How could I forget? reader.readLine(fd: posix.socket_t) is right in the critical path. See ./src/reader.zig.

One important optimisation here is the use of SIMD (vectors/chunks) when searching for line-endings. Rather than check byte-by-byte if we have a new-line character, we can leverage the SIMD (single-instruction-multiple-data) support of modern CPUs. This allows us to fill a mask with new-line characters, and check whether each 16-byte chunk contains the new-line. If it does, we simply call the built-in @ctz (count trailing zeros) to calculate its offset (position) in the chunk.

For any remaining bytes (if the line-length is not divisible by 16), we simply iterate over them byte-by-byte.

The use of SIMD is often integral to efficient data processing, and you will see it a lot.

Step 3: Write iovecs #

Finally for this article, when the ./src/worker.zig responds to a request, making a syscall to write each part of the response, or concatenating the response status line, headers, and body buffers, we can make use of posix.writev to write a slice of iovecs (IO vectors) with a single syscall. This is much more efficient. Thanks to Karl Seguin for his guide to writing iovecs in Zig. I’d like to reiterate just how helpful I’ve found his guides, so thank you Karl.

Wrap up #

I hope you found some of this interesting. I quite enjoy writing these, as it’s a good way for me to reflect on some of the decisions that I’ve made while. While writing this, I even found a couple of little things that I can improve.

Thanks to Bo (author of Facil.io), for his advice for optimising the server implementation. Facil.io still outperforms Peregrine by around 10%, and shows better stability of response times under load. I think that this is due to Bo’s superior solution for distributing and scheduling the computation.

Today, I implemented a fairly performant ring buffer queue (50ns for one read and one write). Maybe I can still improve this by a factor of 2, as a hardware lock should only take a single cycle.

const std = @import("std");

pub fn RingQueue(comptime T: type) type {
    return struct {
        const Self = @This();

        allocator: std.mem.Allocator,
        buffer: []T,
        read_index: usize = 0,
        write_index: usize = 0,
        mutex: std.Thread.Mutex = .{},
        not_empty: std.Thread.Condition = .{},
        not_full: std.Thread.Condition = .{},

        pub fn init(allocator: std.mem.Allocator, capacity: usize) !*Self {
            const self = try allocator.create(Self);
            self.* = .{
                .allocator = allocator,
                .buffer = try allocator.alloc(T, capacity),
            };
            return self;
        }

        pub fn deinit(self: *Self) void {
            self.allocator.free(self.buffer);
            self.allocator.destroy(self);
        }

        fn isFull(self: *Self) bool {
            return (self.write_index + 1) % self.buffer.len == self.read_index;
        }

        fn isEmpty(self: *Self) bool {
            return self.write_index == self.read_index;
        }

        pub fn write(self: *Self, data: T) void {
            self.mutex.lock();
            defer self.mutex.unlock();
            while (self.isFull()) {
                self.not_full.wait(&self.mutex);
            }
            self.buffer[self.write_index] = data;
            self.write_index = (self.write_index + 1) % self.buffer.len;

            self.not_empty.signal();
        }

        pub fn read(self: *Self) T {
            self.mutex.lock();
            defer self.mutex.unlock();
            while (self.isEmpty()) {
                self.not_empty.wait(&self.mutex);
            }
            const data = self.buffer[self.read_index];
            self.read_index = (self.read_index + 1) % self.buffer.len;
            self.not_full.signal();
            return data;
        }
    };
}

This queue will make it possible to distribute work (requests to handle) to a thread pool on a FIFO basis. This would prevent long-running handlers from blocking an entire worker, which could block multiple connections.

I also learned that Zig supports async/await as this could be an even better way for each worker in the pool to handle multiple requests concurrently.

Peace & love! J

Elegance in Global Cooperation: Engineering Simplicity, Purity and Correctness

Thu, 09 Jan 2025 00:00:00 +0000

Today, after reading a good chunk of Serious Cryptography, by J. P. Aumasson, I was lamenting over the likeness of computer science and society, and the way that game theory and complexity theory are interlinked. Reflecting on my day’s study, and my view of the world in general, I found myself drawing many comparisons that seemed insightful.

For example, the security of a given cryptographic scheme is in practice dependent on its understandability, for if an algorithm is difficult to understand, there is a high likelihood that it will be implemented poorly or misused. That is to say that the number of possible attack vectors scales with complexity.

A system that requires many details to be properly defined is inherently harder to understand. I do not suggest that complexity is bad, but merely a result of solving hard problems. However, the complexity of a solution does not imply its efficacy, but rather increases the probability that there is a flaw. This notion may be reduced to the idea that together simplicity and efficacy is virtuous. One might say that an elegant solution is simplified correctness.

To me, this is one of the most fundamental engineering principles. Our intuition for what is good and what is bad is rooted in the elegance we see in the idea. To me, elegance is purity, simplicity and correctness. Nothing more, and nothing less than fit for purpose.

When solving hard problems to deliver value to the world, we reduce our solution to its essence. Such is a proper definition of our solution.

My view of humanity is an ecosystem of beautiful souls that must cooperate for survival. Productive and fair cooperation is tantamount to our survival.

Before I continue, I must state that I am an optimist, and I believe that in general, humans have made progress toward cooperating on a global scale. As our capabilities of travel and communication have grown, so have the numbers of people we attempt to organise, and the geographic size of these clusters of organised cooperation. I suggest that recent technological progress has not been matched by organisational progress when it comes to organising large groups of people. As such, my feeling is that many of the social constructs that we depend on for productive and fair cooperation are no-longer fit for purpose. I also suggest that in engineering terms, our current legal, political and economic mechanisms are monolithic patchworks of progressive solutions mixed with outdated and sometimes irrelevant requirements. To simplify, I suggest that in order to progress further as a civilisation, we must aim for simpler and clearer terms by which we define fair and productive cooperation.

I suggest that the definition of an ideal societal system is akin to a mathematical formula, just as you can devise a proof of correctness in mathematics. Game theory is about creating a structure of incentives such that the sum of incentives to play fairly is greater than the sum of incentives to cheat.

Rather than hopelessly wishing that everyone would take only what they need and contribute on their own accord, we would better ensure that the incentives of the individual align with the incentives of the masses. In other words, ensuring that the system makes cheating less rewarding than fair play.

That is, of course, what the current mechanisms attempt to do, however they have become overly complex, inefficient, and riddled with vulnerabilities. You could liken it to a very insecure cryptographic scheme that is exploited whenever deployed. The forces at play are the same; there is an arbitrage between the risk and reward for cheating because the formal specification by which we govern ourselves is inadequate.

At this point, a reader could feel that the critique of a system has no value in the absence of proposed solutions. However, the precursor to positive change is the understanding of contemporary challenges and issues. It would be equally naive to suggest that the solutions to such hard problems could or should be defined by a single person. Rather, a sound and logical consensus that fosters fair and productive cooperation must be a collaborative effort. Many perspectives and experiences must be considered.

I believe that many principles of engineering, mathematics and game theory may be applied in the conception and definition of an ideal societal fabric by which we may better cooperate fairly and productively. Let us empathise with one-another, and believe that we are headed towards a healthy, compassionate and productive future.

The beginning of project Peregrine - a bleeding fast HTTP server

Thu, 26 Dec 2024 00:00:00 +0000

If you’re very familiar with Zig, then you’ll know that the standard library HTTP server is not written to be highly performant.

Zap is an alternative. Zap is built on the fantastic Facil.io networking library written in C.

Now, while there are plenty of fast HTTP servers, at time of writing, I couldn’t find a production-ready high-performance HTTP server written purely in Zig. There is http.zig, but according to Karl, the developer, it must sit behind a reverse-proxy because it’s not resilient to DoS attack.

As I need a real project to get deeper into the language, and having never written a HTTP server from scratch before, I felt that this would be a great opportunity to have fun learning about Zig, async IO, vectored IO, memory alignment, zero-alloc, zero-copy and a bunch of other systems stuff. I’ve also found that working with constraints (such as following a spec) is great for forcing one to find a solution, rather than skirting around an issue. Also, writing a library is a particularly good way to learn a language because it forces you to write reusable (and usable) software.

Peregrine - a bleeding fast HTTP server

My goal with this project is to write a robust, performant and simple HTTP server. In doing so, I will learn much of the Zig language, and a bunch about systems programming.

I started this project on 2024-12-24, two days ago at time of writing. So how far have I got in two days?

Currently, we have the main components of the server implemented. The server is written in a library form, and exposes a clean API to create a server and handle requests.

There is a worker pool. Each worker runs on its own thread. For now, connecting sockets are assigned to a worker using round-robin distribution. This means that we don’t need a scheduler.

Kqueue is the FreeBSD/MacOS solution for async IO. It’s a kernel event system. Basically, you register interest in a type of event. You then process the events. We have one of these event buses for our server’s main thread, where we accept connections. Each connecting socket is then assigned to a worker by registering interest in the client socket’s events with the worker’s event bus. This means that when clients write to the socket, the appropriate worker is notified and will process the event. In this case, the worker processing the event means reading from the socket.

For the Linux kernel, there is a different solution, called epoll. In some ways Kqueue is superior to epoll because Kqueue allows registering interest in multiple events with a single system call. Conversely, with epoll, each interest must be registered with separate sys calls. In our case, this won’t make any difference because we make a sys call every time we accept a new connection. If we didn’t, clients would have to wait until their socket was registered. Otherwise, epoll is very similar to kqueue, as far as I know. In the next days, I will implement epoll to support Linux.

When each worker starts, it does all of its heap allocations that it needs for normal operation. Ideally, no further heap allocations will be made per-request. This zero-alloc approach greatly improves performance because we don’t need to wait for memory to be allocated on the heap before progressing with processing each request.

Each worker can handle one request at a time. This constraint allows us to reuse the buffers and data structures that it allocates. The only thing we have to ensure is that they are properly reset before reuse.

Getting deeper into the guts of the server, each worker creates a request reader. This request reader reads from a given socket, and maintains a buffer of read-but-unprocessed data. When we call readRequest, unprocessed data in the buffer will be read before compacting the buffer and making another syscall to get more data. This approach means that fewer syscalls are used to read each request, while not needing a particularly large buffer.

One optimisation that I’ve made already is that buffers are properly aligned. This means that the data’s memory address is a multiple of its size. Memory alignment is crucial for high-performance software because the CPU accesses memory most efficiently when data is naturally aligned. This allows the processor to fetch memory in a single operation, whereas misaligned data may require multiple accesses. Additionally, aligned data structures help to optimise cache efficiency. When data crosses cache line boundaries, it can result in additional cache misses, resulting in performance degradation.

The cache-line half of that is a different number. A line is typically 64 bytes, and it is alignment to that which decides whether a structure straddles two of them. A u64 at an address divisible by 8 is naturally aligned and can still sit across a line boundary, if the buffer around it isn’t.

Below is the critical path of the worker’s event loop. Each kernel event is processed as follows.

// ./src/worker.zig
fn handleKevent(self: *Worker, socket: posix.socket_t, reader: *request.RequestReader) !void {
    defer posix.close(socket);
    self.req.reset();
    try reader.readRequest(socket, self.req);
    self.resp.reset();
    self.on_request(self.req, self.resp);
    if (!self.resp.hijacked) try self.respond(socket);
}

fn respond(self: *Worker, socket: posix.socket_t) !void {
    const headers_len = try self.resp.serialiseHeaders(&self.resp_buf);
    if (self.resp.body_len > 0) {
        var iovecs = [_]posix.iovec_const{
            .{ .base = @ptrCast(self.resp_buf[0..headers_len]), .len = headers_len },
            .{ .base = @ptrCast(self.resp.body[0..self.resp.body_len]), .len = self.resp.body_len },
        };
        const written = try posix.writev(socket, &iovecs);
        if (written != headers_len + self.resp.body_len) {
            return error.WriteError;
        }
        return;
    }
    try writeAll(socket, self.resp_buf[0..headers_len]);
}

One optimisation you can see above is in the respond method. This is called after the user’s on_request handler is called, providing that they did not hijack the response. More on that later. If the response has a body, vectored IO is used to write the headers and body in a single syscall, without needing to concatenate the buffers. If there is no body, the response is written by repeatedly writing to the socket until all bytes are written. Without a body, this is likely to be achieved in a single syscall.

Making use of x86’s SIMD and ARM’s NEON capabilities allow us to speed up a variety of operations. One example that you will see often is chunking. This is where slices are processed in chunks, rather than operating on single bytes. This is important because operating on single bytes does not make good use of the cache. A good example of this is in Zig’s stdlib: std.mem.indexOfScalarPos.

This searches a slice for a specific value. Normally this would be a slice of bytes, and we’d be searching for a byte. If we were to search the slice byte-by-byte, the CPU would have to individually load each byte into a register for comparison. In std.mem.indexOfScalarPos, you can see how a bit mask is used to efficiently test if the target byte is somewhere in the chunk. The naive and much slower implementation would be as follows:

pub fn indexOfPos(comptime T: type, slice []const T, start_index: usize, value T) ?usize
    if (start_index >= slice.len) return null;
    var i: usize = start_index;
    while (i < slice.len) : (i += 1) {
        if (slice[i] == value) return i;
    }
    return null;

Often, adding some room for complexity allows us to solve problems more efficiently. Good engineering is partly about balancing performance gains with simplicity. A good engineer knows what to optimise, and what not to. Tight loops in critical paths tend to offer the greatest reward for optimisation.

Anyway, hopefully that was interesting. After a couple of days, we have the bare bones of a high-performance HTTP server, written purely in Zig. Considering that I’m new to systems programming and Zig, I feel that I’m doing alright. It surely is a very fun and expressive language. I’m enjoying the build speed and the quality of the compiler’s errors.

Stay tuned for my progress, as I will be working on this to make it production-ready. I’m not sure how long it will take, but I am sure that I’ll continue learning a lot.

Peace and love! J

First thoughts on Zig

Mon, 23 Dec 2024 00:00:00 +0000

I’ve done very little systems programming. I wrote Conway’s Game of Life in C as a teenager. I also did a bit of assembly at GE Aviation Systems. A few years ago, I completed the Rust programming book, but struggled with the language beyond that, and never grew to love it.

I’m currently diving quite deep into systems programming in Zig. While there are many things I love about the language, namely its elegant approach to generics and meta-programming, I do feel that the language is lacking in some ways.

Simplicity #

What I like most about the language is its simplicity. Similar to Go in this regard, it didn’t take me long to become comfortable with the syntax.

Errors lack context #

One of the most obvious flaws with the language is its error type. It’s impossible to add context or details to errors. In contrast, an error in Go for example, is just a value. This allows you to add context as the error is passed up the call stack.

I’m not sure if Zig will offer a solution for this.

The idiom that emerged is a separate diagnostics struct the callee writes into, alongside the plain error — std.json and the compiler’s own parser both do it. It keeps the error set exhaustive and switchable, which is the property Zig is protecting, and puts the prose somewhere else.

Despite this, errors in Zig are otherwise great, and very easy to handle or throw up the stack.

Powerful build system #

Zig exposes a very powerful yet usable build API. Linking C libraries is quite effortless, and I’m a noob to this.

While I’m not yet familiar with much of the build system, I have the feeling that it is probably the best build system that I’ve seen yet. For me, it was also the hardest part of Zig to become comfortable with. I’m still not really comfortable with it. In addition, as the language is so young and volatile, much of the 3rd party documentation for the build system is out of date.

Fortunately, the API is very clean, and so it’s not so difficult to figure things out on your own.

Lack of interfaces #

Zig’s philosophy is to provide low-level building blocks, rather than high-level abstractions. As such, the Zig team felt that interfaces are an unnecessary abstraction.

While I understand this view, I feel that the language’s lack of built-in support for interfaces can make interface implementations feel quite clunky. This became particularly apparent to me when I began using the standard library’s filesystem types. I’m sure that this could be greatly improved as the language server matures.

I tried both answers a couple of weeks later while writing an HTTP server: an explicit vtable the user has to declare, and the comptime-generic Server(MyHandler) pattern. The generic one won on readability and on speed — the comparison is here.

Expressive language #

I am often surprised by Zig because it tends to work exactly as you’d hope. This is particularly noticeable when implementing polymorphic types and platform-dependent behaviour. This makes writing cross-platform software remarkably easy. For me, Zig really shines in this regard.

Generics are beautiful #

Thanks to Zig’s expressive compile-time metaprogramming, generics are very easy to implement, and are perfectly type-safe.

The following is a generic thread-safe message queue:

const std = @import("std");

pub fn Queue(comptime T: type) type {
    return struct {
        const Node = struct { data: T, next: ?*Node };
        mutex: std.Thread.Mutex,
        head: ?*Node,
        tail: ?*Node,
        allocator: std.mem.Allocator,

        pub fn init(allocator: std.mem.Allocator) !*Queue(T) {
            const queue = try allocator.create(Queue(T));
            queue.* = .{
                .mutex = std.Thread.Mutex{},
                .head = null,
                .tail = null,
                .allocator = allocator,
            };
            return queue;
        }

        pub fn deinit(self: *Queue(T)) void {
            self.mutex.lock();
            defer self.mutex.unlock();
            while (self.head) |node| {
                const next = node.next;
                self.allocator.destroy(node);
                self.head = next;
            }
            self.tail = null;
        }

        pub fn write(self: *Queue(T), data: T) !void {
            const new_node = try self.allocator.create(Node);
            new_node.* = .{
                .data = data,
                .next = null,
            };
            self.mutex.lock();
            defer self.mutex.unlock();
            if (self.tail) |tail| {
                tail.next = new_node;
                self.tail = new_node;
            } else {
                self.head = new_node;
                self.tail = new_node;
            }
        }

        pub fn read(self: *Queue(T)) ?T {
            self.mutex.lock();
            defer self.mutex.unlock();
            if (self.head) |node| {
                const data = node.data;
                self.head = node.next;
                if (self.head == null) {
                    self.tail = null;
                }
                self.allocator.destroy(node);
                return data;
            }
            return null;
        }
    };
}

test "basic usage" {
    var gpa = std.heap.GeneralPurposeAllocator(.{}){};
    const allocator = gpa.allocator();
    const queue = try Queue(u8).init(allocator);
    try queue.write(1);
    try queue.write(2);
    try queue.write(3);
    try std.testing.expect(queue.read() == 1);
    try std.testing.expect(queue.read() == 2);
    try std.testing.expect(queue.read() == 3);
    try std.testing.expect(queue.read() == null);
}

In Go, I very rarely found myself using generics. Generics in Go feel like an after-thought, and that’s probably because they are.

Conversely, in Zig, I’ve used this comptime struct pattern many times to write reusable components.

Standard library is still rough around the edges #

Some packages in the standard library are not entirely production-ready. For example, the Blake3 implementation uses the reference rather than the optimised implementation.

There are faster and more robust HTTP servers than the one in the standard library, namely Zap built on facil.io (written in C).

As Zig is still young, it seems only natural that the standard library is not yet polished. Obviously, the language should mature and stabilise before polishing the standard library can become a priority.

Memory management made easy #

When I learned Rust (or tried to), I never became comfortable with it, and I was constantly fighting the compiler.

Possibly, I’m simply a better programmer these days, but I find it more likely that Zig’s approach to memory management is just simpler and more explicit.

Zig is not a fancy language, it really does feel close to C. I feel that this inherently makes it easier to learn than Rust.

Mostly writing Go recently, I expected that Zig’s memory management would be a really big hurdle. While it has taken me a few days to become comfortable allocating memory on the heap, I was surprised how easy I found it.

I’m still not fully familiar with each of the standard library’s allocators, and I mostly use the GeneralPurposeAllocator.

GeneralPurposeAllocator has since become std.heap.DebugAllocator, which is the honest name — it was always a debugging allocator rather than a fast default. The release-build answer is now std.heap.smp_allocator. Zig moves like this constantly, and it is the main reason anything written about it goes stale inside a year.

Conclusion #

I’m only a couple of weeks into my journey with Zig.

Would I use Zig for work? If the project would benefit from optimal performance and reliability, absolutely. For bits such as tooling for managing infrastructure or automated key rotation, where performance is not a concern, I’d likely opt for Go. At time of writing, Zig’s ecosystem is still very small compared to Go’s.

My reason for beginning with Zig was to get into systems programming, and I’m really enjoying it. I remember faffing with header files in C as a teenager, and I know that building C projects can be a real pain. So far I’m quite happy with Zig, and I’m optimistic that a lot of the kinks will be ironed out as the language matures.

There will never be a perfect language for everyone for all use-cases. As I’ve been learning Zig, I’ve come to appreciate how hard it is to design a language that is fit for decades of mission-critical software engineering. It’s incredible how good C is, despite its (lack of) build system. I’ve also come to realise how much of Go they got right. Writing concurrent software in Zig has left me with an appreciation for Go’s channels. The only thing that I would change about Go is the verbose error handling.

I feel that learning a new programming language allows us to open our minds to alternative paradigms. Questioning what we already know is fundamental to making progress.

Dave

Wed, 01 May 2024 00:00:00 +0000

EDIT: In September & October I made significant changes to the DHT, and so much of this article is out of date. I have not yet found time to update this article, and I’m not currently working on the project.

So… what am I doing? When I started this project (first commit 2024-04-05), I didn’t know exactly, beyond designing a peer-to-peer network application that disseminates information. Now my vision is finally becoming clear, and I’m able to describe it.

I’m writing a peer-to-peer application that allows anyone, or any device, to read and write information without a value transaction of any kind. A writer pays in CPU cycles and electricity, while a reader pays in network participation.

Simplicity, anonymity, efficiency, performance, and high availability are key aspects that I’m aiming for.

Simplicity will come at the cost of pushing much application-specific complexity up the stack. We must be free to re-imagine a broken game, if it turns out to be broken… At this time, I feel that this approach ensures a modular and upgradeable stack of protocols, while maximising the probability of success of the dave protocol.

I’m aware that such projects have been attempted and failed. I’m also aware that such a challenging project may not be an ideal start for a budding p2p network researcher. With that in mind, I do need a worthy challenge and vision to hold my interest.

Dave - distributed KV store #

Dave is a protocol for continuously disseminating information in a decentralised, anonymous, and censorship-resistant manner, without the need for a native token or value transaction.

New values are available nearly instantly, with sub-second lookup time. Old values have a significantly faster lookup time, because more nodes will likely have the data. Real-world speed depends on the bandwidth provisioned by the nodes. Most likely, the response time is better than average network ping, even with a naive implementation.

Use-cases: Decentralised social media applications, serverless forms, communication layer for any decentralised application.

Design #

The protocol is designed around a single message format with an enumerated operation code. There are 4 operation codes, as follows; GETPEER, PEER, PUT, GET.

Execution is of a cyclic mode, with the minimum period defined by constant EPOCH. The program sends a uniform stream of packets, minimising packet loss and maximising efficiency. This allows the protocol to be adjusted safely, and in a way that could preserve interoperability with a network running variations for different bandwidth ideals & constraints.

GETPEER & PEER Messages #

These are the first two op-codes that I defined, and initially the only operations that the network performed. These two messages allow nodes on the network to discover peers, and to verify their availability.

For every PING EPOCH, iterate over the peer table. If a peer is found that has not been seen in the last PING EPOCH, and has not been pinged within the last PING EPOCH, send to the peer a message with op-code GETPEER.

A protocol-following peer will reply with op-code PEER, a message containing NPEER addresses for other peers.

I often refer to these addresses as peer descriptors, as in future, they may not necessarily be IP addresses. I would like the possibility to cleanly implement interoperable transports. This would allow nodes to serve as bridges. As software-defined radio is a field of interest to me, I imagine that the dave protocol could used to create alternative low-energy networks that could bridge our existing infrastructure built on IP. Imagine a free and open internet that does not require an internet service provider, or any knowledge of its users. Know that I have not yet adjusted the protobuf specification to support this, as I prefer to focus on a building working proof over UDP for now.

If a peer does not respond with any valid message, after DROP * EPOCH has elapsed, the peer is deleted from the peer table. Unresponsive peers are no-longer advertised after OPEN * EPOCH has elapsed without message, so as to ensure that unresponsive peers are not re-added from latent gossip.

PUT Message #

A message with op-code PUT is a packet of data containing a Key, Value, Time, Salt, Work, Sig and PubKey. Salt and Work are outputs of the cost function, into which Key, Value and Time are passed.

Every EPOCH, each node sends one randomly selected dat to one randomly selected peer.

This ensures that information propagates and re-propagates through the network reliably, until it is eventually no-longer stored by any node. I describe the selection algorithm later.

This provides a good level of anonymity for original senders, because it is virtually impossible to discern the origin of a dat, even with a broad view of network traffic.

Anonymity is achieved by ensuring no correlation between the timing of a dat being received for the first time, and its eventual propagation to other nodes. This happens at random, but at a constant interval.

GET Message #

A GET message is a packet with op-code GET, containing a Key. If the remote has the dat, the remote should reply with a message with op-code PUT, containing the data. A node may request the same dat from many peers simultaneously. This is up to the application to optimise, although the reference implementation should be pretty good.

DAT Selection by Mass #

Proof-of-work #

A Salt is found, that when combined with the Key, Value & Time fields using a hash function, the output begins with some desired number of leading zero bits. The number of leading zeros (or some other constraint) probabilistically accurately reflects the energetic cost equivalent of computing the proof. This is commonly known as proof-of-work, and is well known for its use in Bitcoin mining.

The number of leading zero bits is referred to as the “difficulty” throughout the remainder of this document.

We compute a salty hash from an initial hash of key, value and time. For large values, this performs better than a single hash.

work = blake3(salt, blake3(key, value, time)) #

Mass Calculation #

As each machine has limited resources, we need a mechanism by which the software can select which dats should be stored, at the expense of others being dropped. In a peer-to-peer application without any central coordination or authority, each node must be able to decide which dats to keep on its own.

So how can we give priority to some dats over others?

As the cryptographic proof contains the value, and time, neither may be modified without invalidating the proof. As such, we can use the time, and the difficulty in the calculation of a score, referred to here as mass.

mass = difficulty * (1 / millisecondsSinceAdded) #

The mass tends to zero over time. Dats with a harder proof of work persist longer in the network. In addition, difficulty scales exponentially (each added zero byte increases the difficulty by 256 times.

Peer Trust Mechanism #

A decentralised network depends on a trust system that incentivises fair play. It’s much more challenging than operating in a private network, as most web applications do today. So why do it? It allows us to build more powerful software that cannot be controlled by any single entity.

The goal of the trust mechanism is to ensure that energy is not lost to malicious or protocol-deviating peers.

Earning Trust #

Each time a packet is received containing a dat not already stored, the remote peer’s trust value is incremented by the mass of the dat.

If a peer is dropped, and then re-joins the network, they will begin with a trust score of zero.

Trust scores are not gossiped, as this implies additional attack surface and complexity.

Use of Trust #

The trust score is weighed in to the random peer selection. A random threshold between the maximum trust score and zero is chosen. A peer with a trust score greater than the random threshold is selected at random.

Therefore, peers with a higher trust score are more likely to be selected for gossip messages. This in turn increases the chance for the peer to learn of new dats earlier, reinforcing the pattern.

In essence, the longer a peer (ip-port) remains in an other’s peer table, the higher the trust score will likely be, and therefore more bandwidth is allocated to that peer.

Resilience to Attack #

Sybil Attack #

A Sybil attack is one where an attacker creates many peers that join the network. Failing to consider this would result in the network losing energy to these numerous byzantine nodes that are inexpensive to spawn.

Eclipse Attack #

An Eclipse attack is one where an attacker attempts to poison the peer tables of peers that it discovers on the network. This poisoning may be either byzantine nodes controlled by the attacker, or simply bogus peer descriptors.

Denial of Service #

This attack is the simplest to orchestrate, and yet one of the hardest to deal with. A denial of service attack is one where an attacker sends packets that induce the maximum amount of computation on the remote. HTTP servers are commonly the recipients of such attacks, which is why we use CDNs where possible to shield our origin servers. This is obviously not possible for stateful APIs and databases. That’s why we put databases in private networks, and employ efficient filtering of API requests, often using an IP-based rate limiter. Obviously we can’t run our peer-to-peer application in a private network unless we only want to use it in a datacenter.

Unfortunately, if an attacker has more bandwidth than any given node, the node can simply be denied service by exhausting their bandwidth. If the attacker has more bandwidth than the entire network combined, the whole network is denied service. So if you advocate for a given peer-to-peer network, the best way that you can support the network is by giving your bandwidth. Run a node.

Attack Mitigations #

Dat Delay #

A node must wait some time before receiving dats. They must participate in the gossip of peers without receiving any dats, until the defined period has elapsed. They will then start to receive dats very slowly, based on the PROBE constant.

Trust #

A node normally selects only trusted peers to receive dats, but occasionally an untrusted peer is probed. Over time, a node is able to build trust with peers, therefore receiving dats more frequently. As a peer earns trust, they are more likely to be selected, thereby having more bandwidth allocated to them.

NPEER Limit #

We only accept PEER messages containing up to NPEER peer descriptors. I recommend that this value is either 2 or 3. I err on the side of caution with 2, because there is little benefit to speeding up the peer discovery in this way. It would be more secure to ask for peers more often.

Funny! While writing this (2024-05-29), I discovered a vulnerability to eclipse attack. A byzantine peer may send many PEER messages, limited only by the packet filter. This made address poisoning trivial. This was resolved by recording the last time a PEER message was received, and only accepting a PEER message after the expected duration has elapsed since the last.

Storing Large Files #

As mentioned earlier, all application-specific complexity is pushed up the stack. Dave is purely a packet-sharing protocol, with no built-in features for storing large files. That said, I have considered the need for storing large files in the network.

Linked List #

The simplest approach to storing large files, and the earliest proof that I implemented, is a linked-list. Simply prepend or append the hash of the previous dat in next dat. This simple approach has the disadvantage that reading is slow because GETs cannot be made in parallel.

Merkle Tree #

An optimised approach is to write the large file as a sequence of dats, and concurrently build a merkle tree, where additional pointer dats are used to reference dats containing the file’s content. The hash of the root of the tree may then be used as the reference to the file. A reader may then traverse the tree, collecting all of the dats in parallel, and re-assemble the file. Reads of this nature are significantly faster than the linked-list approach.

Some Repositories #

godave is the protocol implementation in library form, written in Go. #

Protocol https://github.com/intob/godave/

daved is a program that executes the protocol, as any other application that may join the network. #

A tiny cli https://github.com/intob/daved/

garry is a HTTP gateway. #

Anything can work with dave. https://github.com/intob/garry/

State of Operations #

I’m no-longer running public edge (bootstrap) nodes myself. I was using AWS, with a bunch of t4g.nano arm64 VMs running Debian 12. I used scripts & programs to control groups of additional machines as I needed. I view logs by grepping the linux system journal. Logs begin with a short path prefix /fn/proc/action, allowing me to efficiently grep logs without need for typing quotes around the query.

Thank you for reading. I value advice and ideas. If you have any, please do reach me.

Run as Node #

Executing with no command (just flags) puts the program in its default mode of operation, participating in the network.

daved # listen on all network interfaces, port 1618
daved -v # verbose logging, use grep to filter logs
daved -v | grep /d/pr # verbose logging, with only the daemon's prune procedure logs output
daved -l :2024 # listen on all network interfaces, port 2024. Same as -l [::]:2024
daved -e :1969 # bootstrap to peer on local machine, port 1969
daved -e 12.34.56.78:1234 # bootstrap to peer at address

Commands #

daved set hello_dave # write hello_dave to the network with default difficulty of 16 bits
daved -d 32 set hello_world # write hello_world to the network with difficulty of 32 bits
daved get $HASH # get the dat with the given work hash from the network
daved setf myfile.txt # write a very small file to the network

logd

Fri, 15 Mar 2024 00:00:00 +0000

Started around December 2023. logd is a log daemon: services fire log lines at it over UDP, and you tail or query them live from anywhere. It ran in production, where it did more for our observability than anything else we had, and it is still the most fun I have had building something.

I had been building protocols for fun since I was young. This is the one that ended up in production — security flaws and all, as it turned out. UDP is most of why it exists. TCP hands you a stream and hides the machinery. UDP hands you a datagram and a socket and leaves the rest to you — who is connected, what arrived, what to do about the packet that didn’t, all of it becomes a decision you have to make on purpose. I wanted to make those decisions.

The tail #

The part I still like best: one command, every service, live.

Log lines are written under a path — /prod/my-app/http, /prod/my-app/udp, /debug. A query carries a key prefix rather than a key, so the store walks every ring whose name starts with what you asked for:

for key, r := range s.rings {
	if strings.HasPrefix(key, keyPrefix) {
		matchedPrefix = true
		for d := range r.Read(offset, limit-count) {
			out <- d
			// ...
		}
	}
}

Which means tailing /prod merges every service beneath it into a single stream. Watching a whole deployment scroll past in one terminal, interleaved in real time, is a genuinely different way of seeing a system than opening one log file per box. That was the feature people actually used.

Ring buffers all the way down #

The store is a map[string]*ring.Ring — one fixed-size ring per key, sized in config, plus a fallback ring for anything unrecognised. A write is an index bump and nothing else:

func (b *Ring) Write(data []byte) {
	head := b.head.Load()
	b.values[head] = data
	b.head.Store((head + 1) % b.size)
}

Memory is decided at startup and never moves. There is no rotation, no compaction, no disk, no vacuum job at 3am. Old lines fall off the back. For what is happening right now — the only question I ever asked it — that is the entire feature set.

The other half of that decision is UDP itself. A logger that blocks the thing it is logging is worse than no logger, and a write here is a sendto that cannot apply backpressure to the application even if the daemon is gone.

Nobody is connected #

UDP has no connections, so who is currently tailing? is a question the socket cannot answer. The server keeps a map of tails, each client sends a PING every two seconds, and missing three drops you. A client that dies stops pinging and is cleaned up by absence.

I like this more than I expected to. A TCP server learns about a dead client from an error on a write; a UDP one just stops hearing from it. Same outcome, less machinery.

The hard part #

Every packet has to authenticate itself, alone. There is no session to establish trust once and then ride on.

The wire format has been the same since v0 — a 32-byte SHA-256 sum, 15 bytes of marshalled time, then the Protobuf payload — and the sum is taken over the secret, the time and the payload together. The time is what bounds the damage: a packet more than 200 ms old is rejected before anything else happens, so a captured packet is only worth replaying for 200 ms.

Which leaves the actual question: have I seen this packet before?

Attempt one: remember every hash #

A ring of the last few thousand sums, compared one at a time under a mutex.

g.history.Do(func(v interface{}) {
	b, ok := v.([]byte)
	if !ok {
		return
	}
	if bytes.Equal(b, sum) {
		found = true
	}
})

This gets worse exactly where it needs to get better. Every packet costs a full scan of the ring, so making the history longer makes each packet more expensive — and because the ring holds a fixed number of packets rather than a fixed duration, the span of time it covers shrinks as traffic rises. The busier the server, the less it remembers. It was backwards in both directions at once.

Attempt two: a cuckoo filter #

Constant time, comfortably over 200,000 packets a second, and about four lines of code:

func (g *Guard) replay(sum []byte) bool {
	return !g.filter.InsertUnique(sum)
}

Two things wrong with it, and I only saw one at the time.

The one I saw: a filter has no notion of when, so it cannot expire anything. The only way to stop it filling is to reset the whole thing on a timer — ten seconds, in the shipped config — and at every reset all memory is dropped at once, so a replay from just before the tick sails straight through. There is a comment in main.go where I worked out the fix and never wrote it:

FilterCap: 16000000, // balance for memory, larger size reduces false positives
// now I know, we need 2 smaller filters, A & B.
// Each must pass, and each is reset off-kilter to prevent
// replay immediately after filter reset.

The one I didn’t see: InsertUnique returns false when the item is already present and when the filter is too full to accept it. So replay() reports a replay in both cases. Fill the filter before its timer comes round and the server rejects every packet until it does.

And the sizing tells the story on its own. Packets are valid for 200 ms. The filter holds ten seconds — fifty times the window — in 16 MB of fingerprints, to answer a question about the last fifth of a second.

The answer #

I found it in the WireGuard spec, which is worth reading whether or not you are ever going to write a protocol. A sequence counter, incremented per packet and included in each one. The server tracks the highest it has seen: anything far behind is rejected outright, without a lookup, and anything close is checked against a small bitmap of which of the recent numbers have already arrived.

The reason it works is that a counter is ordered and a set is not. Ordering makes forgetting free — once the counter has moved on, everything behind it is rejected by arithmetic, with nothing to reset, clear or time out. My filters had to remember far more than they needed to, because the only alternative on offer was remembering nothing at all.

Two of the three remember the wrong amount. The filter is not too slow — it is too broad, and it cannot be anything else.

And it wasn’t HMAC #

I described this, in the README and in the first version of this post, as hash-based message authentication. It isn’t, quite. HMAC is a specific construction — two nested hashes with padded keys — and it exists precisely because the obvious thing, H(secret || message), is not safe with SHA-256.

SHA-256 is a Merkle–Damgård hash, and its output is its internal state. Given H(secret || time || payload) and a guess at the length of the secret, you can resume the hash from that state and compute H(secret || time || payload || padding || anything) without ever knowing the secret. My verifier splits the packet at a fixed offset and recomputes over secret || time || payload, so an attacker who appends to the payload and supplies the extended sum passes the check.

The flaw belongs to the Merkle–Damgård family specifically — MD5, SHA-1, SHA-2. A sponge like SHA-3, or BLAKE2 and BLAKE3, does not hand you its whole internal state in the digest, so H(secret || message) is genuinely fine with those. That is a poor reason to choose one. HMAC works with all of them.

What saves this particular deployment is narrow and accidental. The appended bytes have to include the hash’s own padding, and Protobuf will not parse the result — so the forgery is thrown out by the parser, one step after the check it defeated. And the 200 ms validity window means you have 200 ms to capture a packet, extend it and land it.

Neither of those is a defence. hmac.New(sha256.New, secret) is the same amount of code as the thing I wrote instead.

Looking back #

2024-05. This was an enjoyable project to rediscover writing software. It’s amazing to return to something after a change of mind. A new mind has more fun, and is more productive. The same thing happened when I re-imagined speedflying after my spinal-cord injury. Some years after my recovery, I rediscovered a lost joy with a new mind.

2024-12. I find it amusing that I struggled so long with replays. The sequence counter is simpler than everything I tried before it, and once you have seen it, a membership filter looks like the wrong shape for the job.

I would build it differently now — for most of what this did, a process reading the systemd journal is the honest answer, and the logger abstraction is a layer I would not reach for again. But the daemon did its job for a real team on real traffic, and the parts of it I got wrong are the parts I now understand. You don’t learn why HMAC exists by importing it.

Repo #

https://github.com/intob/logd

arpload

Sat, 09 Mar 2024 00:00:00 +0000

Arweave sells one thing: you pay once, and the network stores your data for at least two hundred years. The bet behind it is that storage keeps getting cheaper, so a one-off fee paid into an endowment outruns the cost of holding the bytes forever. Whether that holds for two centuries is a question for another post. I wanted to upload a 4K video.

The web wallet #

I started with the Arweave web wallet, which would have worked. The upload sat there for a long time with no visible progress — I hadn’t opened dev tools — so I assumed it had failed, and tried again. It had not failed. I had paid for every one of those transactions.

That lesson isn’t really about Arweave. An upload with no progress indicator, on a network where every attempt costs money, will get retried until the user runs out of money.

ArDrive #

Then ArDrive, because what I actually wanted was a nice interface and a large file safely uploaded, and the web interface genuinely is pleasant.

But ArDrive bundles. Your data goes into a bundle transaction alongside other people’s, which is cheaper and faster and entirely reasonable, and not what I was after. I wanted a standalone transaction: one id that is mine, that resolves to my file and nothing else.

So, build it #

goar, from everFinance, does the heavy lifting, which left the uploader small enough to be worth writing rather than looking for.

The piece worth having is resume. A 4K video is a lot of chunks, and an upload that starts again from zero every time the connection hiccups is an upload that never finishes — which, in a roundabout way, was the same problem I’d started with. arpload tracks how far it got and picks the transaction back up.

Demo #

Tx: TK60GE39cvuTh_DlLK5ZCnhcoOkN42YFQjsJfzwvrk4

Repo #

https://github.com/intob/arpload

First time in Madeira

Tue, 19 Dec 2023 00:00:00 +0000

Flare have given me a Line 8 (cheers Beni), and they asked me if I’d kindly take it flying.

Considering the Maritime alps, I looked further afield, to warmer climes. I almost went to Corsica.

After hearing of the Bitcoin conference and arranging with a couple of friends to go together, I saw photos & videos of this incredible island.

My imagination ran wild as I explored 3D maps, marvelling at the topography.

The island is inhabited by one of the friendliest populations I’ve ever seen.

I felt the people to be especially beautiful, relaxed & kind. Very refreshing.

We enjoyed almost every sunrise & sunset on the mountains or by the sea.

Almost every day, we swam in the sea.

We ate only one meal per day. Each meal was incredible.

Enjoy the video of my flying with this interesting wing.

Joey Innes

Designing the pressure, not the shape

There is no such thing as a good aerofoil #

The deliverable is a curve #

The budget #

Buying a laminar run, and what it costs #

Two sections, and neither of them is better #

The knobs #

Stall is a design decision, not an accident #

The same shape is a different design at a different size #

Nobody designs the shape #

The section is not the wing #

Next #

The boundary layer, where drag and stall live

The film #

Nobody solves the profile #

One number decides everything #

Waiting for a wobble #

Drag, at last #

The circular argument #

The shape the air actually sees #

Where the loop breaks #

What this cannot tell you #

Next #

The question that broke Dave

Where the code is now #

What it did #

The part that worked #

One packet is enough #

Mass was recency wearing a costume #

The anonymity claim was too strong #

The one I never solved #

The question #

Deleting the question #

Why Zig #

A loose end worth picking up #

What I don’t know #

How we built Soaring Spirit

One binary, one machine #

Ten dependencies #

No frontend build step #

Money is where the discipline goes #

The retry loop #

Let the database say no #

Two ways to get email wrong #

The thing that stopped running #

What I’d tell myself at the first commit #

Two stories about air

Where friction is, and where it isn’t #

A wing is a list of points #

The trick: throw the wing away #

The bit where nature has to choose #

It’s all one linear system #

Getting the answer back out #

The drag that isn’t drag #

The check that keeps me honest #

What it can’t do #

Next #

Progress on the Peregrine HTTP server

Step 1: Improve the handler #

Step 2: optimise the request parser #

Method, path and version #

Parsing headers #

SIMD for reader.readLine #

Step 3: Write iovecs #

Wrap up #

Elegance in Global Cooperation: Engineering Simplicity, Purity and Correctness

The beginning of project Peregrine - a bleeding fast HTTP server

First thoughts on Zig

Simplicity #

Errors lack context #

Powerful build system #

Lack of interfaces #

Expressive language #

Generics are beautiful #

Standard library is still rough around the edges #

Memory management made easy #

Conclusion #

Dave

Dave - distributed KV store #

SIMD for `reader.readLine` #