Architectural Patterns

Every Trick Has an Out

Neil Sherman ·

Every Trick Has an Out

The card is in the wrong place. I know this because I’m holding the deck, and the seven of hearts I solemnly promised is sitting three cards too deep, for reasons I will never fully understand. The spectator is grinning, waiting for the miracle. And here’s what nobody watching can see: I’m already on plan B. There was always going to be a plan B.

Every trick worth performing has an out. A way to still reach the ending when the middle has quietly fallen apart. Usually several, ranked, decided long before anyone was watching. If this goes wrong, you do that. If that goes wrong too, you reach for the next thing, and it looks for all the world like the plan you had from the start.

The audience never finds out. That’s the entire craft. They don’t see a recovery, they see a trick that worked. An out is not a safety net you produce with an apologetic shrug. It’s invisible, and the invisibility is the whole point. The best magicians aren’t the ones whose tricks never go wrong. They’re the ones you’d never know had a bad night.

Hold onto that, because it’s the part most systems get exactly half right.

You know the display. It says the bus is due. It said the bus was due five minutes ago. There is no bus. There is an empty road and a number radiating a confidence it has not earned, while you stand there composing a strongly worded nothing.

What’s happening is more interesting than the board being wrong. The board runs on a live feed, usually SIRI in this part of the world, the standard that carries real-time data about where a vehicle actually is and when it’ll actually turn up. When the feed’s healthy you get a real prediction. When it isn’t, when the bus drops out of coverage in a dip or the feed simply goes quiet, a well-built board doesn’t throw an error in your face. It falls back to the timetable. The scheduled time. The plan everyone agreed on months ago.

And you have no idea. You’re looking at an out. The live trick died and the board reached for its understudy, the schedule, which knows every line and never once complains.

That’s graceful degradation, and it’s the same move I make with the seven of hearts. Decide the fallback in the calm, then reach for it under pressure with the seam already hidden.

Most engineers know the phrase. The bit that gets skipped is that an out is chosen when you’re calm, not when you’re on fire. I didn’t invent plan B at the table with the spectator staring at me. I worked it out at home, weeks earlier, and rehearsed the reach until the reach itself disappeared. By the time I needed it there was no decision left to make, only the quiet selection of a thing already prepared.

Software has a formal name for this, and I’ve written about it at length: the circuit breaker. When a dependency starts failing you stop hammering it, trip the breaker, and serve the fallback. After a pause you let one careful request through to see whether the thing has recovered. If it has, you close the breaker and carry on. If it hasn’t, you keep serving the out and wait. Which is a magician testing whether they can get back to the real ending, or whether they’re committed to the recovery for the rest of the act.

This is where the magician quietly beats most software.

The magician always knows which out they used.

I look relaxed up there, a man enjoying his evening. Underneath, I know exactly what failed and exactly which recovery I grabbed, because I’ll want that later, when I’m working out why the seven of hearts keeps betraying me in front of strangers.

A lot of software degrades so gracefully that nobody finds out. Including the people who built it. The feed dies, the board shows the schedule, the passenger’s content, and not a single signal goes anywhere. The system fooled the audience, which was the goal, and then fooled its own operator, which absolutely was not.

That’s the trap folded inside graceful degradation. A fallback that’s invisible to everyone isn’t resilience. It’s a slow leak with good manners.

And it gets worse than a quiet hour, which is the part I’d most want a less experienced engineer to understand. A degradation nobody can see doesn’t just hide, it settles in. The board shows tidy scheduled times for a morning, then a week, and at some point the broken state stops being an incident and becomes the weather. Nobody declares it. There’s no outage to point at. The system simply forgot it was ever capable of more, and so did everyone running it. The most expensive failures aren’t the ones that wake you at 3am. They’re the ones that never do, the degraded mode that aged into the baseline while everyone admired how stable things looked.

Amazon’s engineers make a sharper version of the point, and it’s worth pausing on. Their advice is to avoid fallbacks wherever you can, because a path that only runs in a crisis goes years without exercise and quietly rots, so the day you finally need it is the day you discover it broke at some moment you weren’t watching. They open, as it happens, with an airport whose boards fail over to staff scribbling flight times on whiteboards, which is our bus stop with worse handwriting. The lesson isn’t that fallbacks are bad. It’s that an unwatched one is a liability dressed as a safety net. Where you can design the need away, do. Where you can’t, and transit is full of places you can’t, the work is to run the fallback often enough that it stays honest and wire it to announce itself every time it fires.

So a real out points two ways at once. Invisible to the audience. Loud to the operator. The traveller at the stop should never feel the downgrade. The person on call should feel it the instant it lands. Same event, opposite experiences, and getting that asymmetry right is most of the actual job.

It’s the part of the problem we spend real time on at navIQuate, and not because it’s glamorous, because it isn’t. The traveller gets a board that simply works, the schedule covering for the live feed without drama or apology. The operations team gets the truth underneath it, that the feed fell silent at 07:42 and wants a look before it quietly becomes the weather. Both, on purpose, at the same time. Anyone can build the happy path. The craft is in the part the passenger is never meant to notice and the operator can never afford to miss.

So the takeaway, if you want one to carry out of here. Reliability was never the absence of failure. Anyone who’s run something real gave up on that idea years ago. Reliability is the quality of your outs, and the discipline of never letting a good one fool you too.

Which turns into a few concrete habits. Decide your fallbacks in the calm and rank them, so the choice under pressure is a reach and not a scramble. Build every fallback to degrade quietly for the user and shout for the operator, because a silent recovery is just an outage you haven’t been introduced to yet. Alert on the transition — the moment the feed goes stale or the breaker trips — not on the user-facing symptom, because a silent fallback is the one failure where the symptom never reaches you. And treat “suspiciously quiet” as a state worth watching in its own right, because the failure that costs the most is rarely the loud one.

The magician’s secret was never that nothing goes wrong. It’s that they always know which card they’re really holding, and the audience never has to.

Links and references