The Front Line Looks Like a Cost Until It's Gone

Frontline staff, moderators, proxies, and red teams absorb the strain so the rest of the organization runs smoothly. Four rules for keeping them from burning out.

Reading settings

Hospitals, platforms, power grids and web servers all have a layer between the outside world and the organized work inside: the interior. The world arrives as weather, lawsuits, traffic spikes, grief and bad-faith users. The interior needs those arrivals to be predictable. Something in between takes the difference, and the wear shows up there first.

The electrical version is a substation. A transmission line carries far more voltage than a house can use, and a transformer steps it down, running hot and humming while it does. The house gets quiet current and never sees the substation, which sits behind a fence at the edge of town.

Institutions have the same layer, and the same habit of not looking at it. Content moderators, nurses on the ward, frontline support staff, site reliability engineers, border officers and security teams all do the stepping down. Software has its own version in the reverse proxy, which takes raw traffic so the application behind it never has to. Prescribed burns do the same job for a forest. The RLHF labelers and red teams who see a model’s output first do it for an AI lab, and intelligence services do it for a state. The people in this layer speak both the interior’s language and the outside’s. They’re often worn down faster than anyone else in the organization, and the interior counts them as overhead.

How the layer gets cut

The layer’s work is preventive, so cutting it looks like saving money. To hit a quarterly number or a headcount target, the interior trims it. The reliability team shrinks, the spare server becomes a single point of failure, and the untidy parts of a process get filtered into a cleaner one. The cost that kept the system running has moved off the books without going away.

The people who run the interior also want it to look clean, so their reports leave out how much of that cleanness the layer produces. And the layer reports in the outside’s language, which the interior keeps trying to correct into its own.

Then the reports thin out, because people who are punished for saying what they see learn to stay quiet. Middle managers smooth the bad news, the red team becomes a checkbox, and moderators file what won’t get them disciplined. The interior now decides from a picture of the world that no longer matches it, and has lost the means to notice.

Then the deferred cost comes due. The next shock may find nothing left to take it and run straight through to the interior. It might be a flash crash, a ward that can’t staff a night, or a person with no more give. Or the people in the layer, underfunded and ignored, use their access against the interior. The administrator who deploys the ransomware, the source who defects and the exhausted worker who stops protecting anyone all had their access because the interior needed them to have it.

Measure boundary roles by truth, then rotate them

Red teams, crisis managers, frontline support and intelligence analysts are often evaluated on the same terms as the people who run the interior, such as tone, compliance and tidy reporting. This is a mistake. A boundary role that is penalized for unprofessional language when reporting a critical vulnerability will stop reporting the vulnerabilities that can’t be described professionally.

Give boundary roles their own charter, their own reporting line and their own measure. The measure is whether what they report is true and complete, whatever the house style prefers.

The work uses people up. It spends a content moderator’s attention and a nurse’s patience, where other jobs would only occupy them. A role treated as a permanent fixture will produce cynicism, injury, or a slow shift of loyalty toward the outside it was meant to hold off.

Rotate people out of boundary roles into quieter ones on a schedule, before they ask. In software the same rule is circuit breakers and load shedding, so that no single node takes the whole surge.

Fund the boundary and limit its inward reach

Most institutions file maintenance, defense and error correction as overhead to be minimized. If the work isn’t funded, the people doing it make up the difference out of their own energy, and the budget records the result as efficiency.

Budget the boundary explicitly, with a floor that is written down and defended. Google’s site reliability teams offer one model. Their handbook sets a goal of keeping operational work, which it calls toil, below half of each engineer’s time, so the other half goes to engineering that makes the next surge smaller. The exact share matters less than the fact that someone has to argue in public to take it away.

To protect the interior, the boundary layer has to understand it, which means it can also damage it. A privileged service account, a negotiator and an embedded source are all sensors that are also levers.

In 2020 attackers slipped malicious code into updates of SolarWinds Orion, software that companies and federal agencies used to monitor traffic on their own networks. The tool that watched the network became the way into it, and the US Cybersecurity and Infrastructure Security Agency ordered federal agencies to disconnect or power it down.

Separate outward reach from inward reach. The boundary can probe and engage the outside freely, while changes to the interior’s core configuration go through a second signature or an air gap. Whatever watches the interior should never be the easiest way into it, so that one failure at the edge cannot become a failure of the whole.

An organization can’t make the outside world’s demands go away, only decide who carries them. It can take outside pressure through a layer of people and systems it staffs and budgets for, or all at once when that layer fails. Much of what gets called efficiency is a cut to that layer, recorded as a saving until the failure arrives.

Continue reading

Related essays

Next routes

This essay belongs to the Survivability pattern; its topic, guide and toolkit are one step away.

New essays by email

One letter a week: that week’s new essays, and one tool to use.

New essays by email

One letter a week: that week’s new essays, and one tool to use.