Every harmful system was once someone’s good idea.
No one writes “make people feel surveilled” in a product spec. No one sets out to build the thing wearing people down. And yet here we are, surrounded by systems beginning as someone’s sincere attempt to help.
The Beaconwright built a brighter beacon out of pure devotion. The brighter it burned, the more it blinded the very ships it was meant to guide. Here is the folly at the heart of Chapter 3: good intentions are not a design spec, and the gap between what we meant and what we measured is where harm quietly gathers.
This is the third signal from the lighthouse. We have named the quiet erosion of trust, and the forms which erase us. Now we turn to the people building the systems, and to the most uncomfortable truth in the book: most harm is done by people who meant well.
The Beaconwright’s folly
In the parable, the Beaconwright is the most devoted soul on the coast. Where others tend their light out of duty, the Beaconwright tends it out of love. Ships are foundering in the dark, so the Beaconwright resolves to build a brighter beacon, the brightest the coast has ever seen. More light, surely, means more ships brought safely home.
The beacon is magnificent. It can be seen for miles. And one by one, the ships begin to founder anyway, because a light fierce enough to be seen from everywhere is also fierce enough to blind the eye trying to read it. The captains can no longer find the dark water between the rocks. They steer toward the glare, and the glare is exactly where the danger lies.
The Beaconwright did everything right, by the measure the Beaconwright had chosen. Brightness went up. The harbour grew more dangerous. Devotion, pointed at the wrong number, became the instrument of the harm it set out to prevent.
When the proxy becomes the point
You have watched this happen. So have I. It rarely announces itself.
It starts well. We want to help people, so we choose something to measure. Sign-ups, because we want more people served. Time-on-task, because we want them engaged. Resolution rate, because we want their problems solved. The measure is a proxy, a stand-in for the good thing we actually care about. For a while, the proxy and the good thing move together.
Then the proxy becomes the point. We optimise the number, because the number is what we can see, and slowly we stop optimising for the person the number was meant to represent. “Engagement” climbs while the experience curdles. The support queue clears faster because the hard cases get quietly closed. The metric glows brighter, and the harbour fills with rocks no chart admits.
This is the line between helping people and managing them, and almost no one crosses it on purpose. We cross it one reasonable decision at a time, each defensible on its own, each pointed at a number rather than a face.
Why good intentions are not a spec
Here is the claim, plainly. Intentions live in the heads of the people who build. The system inherits none of them. A system inherits its incentives, its metrics, its defaults, and its blind spots. It does precisely what it is measured to do, with complete indifference to what its makers hoped.
So “we meant well” is true, and it is not a defence. The harm is not in the meaning. It hides in the gap between the intention and the measurement, and the gap widens quietly, below the threshold where anyone thinks to look. Trust dies in there, not through malice, through drift.
The repair begins with a humbler question than “are our intentions good?” The Beaconwright never doubted the intentions. The better question is the one the parable forces: what exactly are we optimising, and what is the optimisation quietly costing the people on the other side of it? Ask it of anything you have built, and you will usually find a brighter beacon somewhere, blinding a ship it was raised to guide.
Devotion is not the problem. Aimed devotion is. The light was always meant to serve the ships, and not the other way round.
A new signal lights every Tuesday until launch. Subscribe to follow the crossing, or pre-order and read all fourteen on day one. https://www.amazon.com/Rehumanizing-Trust-fractured-digital-rebuild/dp/180576005X/