Measure the Machine, Not the Worker

A clinic that suspects its AI-drafted notes take too long to check has an obvious way to prove it: time the checking. Proposals to count the unpaid work that keeps an institution running, from reading the fixes already in its logs to pricing what it costs to be heard, run into the same problem. Inside a hierarchy, every measurement arrives with an incentive attached, and the people who want the meter most are the people most likely to turn it around. Campbell’s Law and Goodhart’s Law are the formal names. Once a measure becomes a target, people have reasons to optimize the number at the expense of what it was meant to measure. A diagnostic built to protect human capacity can become a performance quota. Whether a proposed instrument survives depends on whether it watches the operator or the machine.

When the meter turns into bossware

One vulnerable design measures verification latency and real-time cognitive load, tracking how long an operator spends auditing an AI-generated draft to test whether fluent summaries impose fatigue. Continuous, individually attributed measurement can require invasive endpoint telemetry, from screen-dwell trackers and keystroke logs to active-window monitors or gaze tracking. With those records available to management, the tool built to hold a vendor accountable can arrive as bossware. A dashboard built to conclude “this software is dangerously dense and requires six minutes of audit time” concludes instead that Dr. Martinez spends 6.2 minutes verifying notes while Dr. Chen spends 1.8, and that Dr. Martinez sits in the bottom quartile of clinical productivity.

If the institution penalizes the clinician for that diligence, it gives workers a reason to skim, speed-read and click approve faster. Errors then have a better chance of passing unchecked. An isolated acceptance lab, run on the buyer’s real data with volunteer cohorts before the contract is signed, is one way to keep the measure aimed at the software. Production studies can also use voluntary reports, sampling or worker-controlled aggregates. Their protection depends on who can see the records and whether the employer can use them for discipline. A continuous individual ranking gives management a whip with a wellness label.

Warehouses got there first. At Amazon’s BWI2 fulfillment center in Baltimore, software tracked each worker’s “time off task,” the unexplained gaps away from the workstation. A letter from the company’s lawyer to the National Labor Relations Board said that about 300 full-time associates there were terminated for productivity between August 2017 and September 2018. It said the system generated warnings and terminations automatically, without input from supervisors. Amazon then said the letter was wrong about that, and that managers can override the process.1 California answered with a law for large warehouses that took effect in 2022. Employers must give each worker a written description of every quota. A quota that prevents meal breaks, rest breaks or bathroom use is unlawful, and a worker who thinks a quota did that can demand the last 90 days of their own work-speed data.2 The law limits what the meter may demand. It leaves the meter pointed at the operator.

The informant trap

A second vulnerable design hunts for unreported workarounds algorithmically. The intent is sound: shadow spreadsheets, paper notes and back-channel chats are outage reports filed by users, and the institution that reads them learns where its software broke. The catch is where workarounds live. They’re in the unmonitored margins exactly because the monitored surface failed. A hunt through those margins can mean surveilling unapproved storage, clipboard transfers, messaging traffic and desk drawers. Workers can also choose to report the failure without exposing everything they did to survive it.

That turns the diagnostic into an informant network, and the punishment lands on the wrong party. A nurse who keeps a personal spreadsheet tracking patient dialysis schedules because the EHR tab is illegible gets flagged as a data-security violation rather than triggering a redesign. The person who saved the treatment gets a reprimand, the vendor stays blameless, and the workforce learns to bury its survival strategies deeper. The same evidence enters the record another way. A worker who declares the workaround on the record is giving testimony, and testimony isn’t surveillance. But the moment IT goes hunting for it, the instrument has turned.

The aggregate exception

Off-hours charting, pajama time, sits on the knife edge. The raw data already exists in audit logs. Using it adds no new collection, but giving more people access to a record of evening work still expands surveillance. As an aggregate it’s devastating in the right way: it can show that claimed operational efficiency rests on unbilled evening labor.

The failure mode arrives when the figure attaches to a manager’s bonus or a hospital’s rating. Administrators then take the path of least resistance, which is the forced lockout: sever remote access at 7 PM and declare victory. The work survives anyway: paper summaries, memorized lab values, notebooks at the kitchen table, a frantic typing session at 6:30 the next morning, and the handoff errors that come of it. The verdict: the measure is viable as a collective bargaining instrument, reported as a clinic-level aggregate across cohorts, barred by contract from individual view. An aggregate can’t name Dr. Martinez. It can prove the ward is running on its own evenings, and it’s worth what the forced lockout proves about management’s real options.

When the aggregate answers back

An aggregate can still turn, in a direction that needs no surveillance at all. Suppose someone takes all of this seriously. They instrument the queues, watch the after-hours record activity, build a continuous index of compensatory labor, run it for a year, find the signal clean, and set a ceiling. Then the instrument built to make invisible work visible starts answering complaints: your burden is six point eight, the limit is seven point two, your objection is unfounded. The index was built to describe a distribution across many people, and it gets used to adjudicate one, because a number is the cheapest available way to end a conversation.

Flint, Michigan, went through this conversion in 2015. Federal rules judge a water system’s lead by the 90th percentile of samples drawn from high-risk homes, against an action level of 15 parts per billion. That year the city sent the state 71 samples, and the state had the city drop two of them. The dropped samples were 20 parts per billion from what the state said was a business, and 104 from the home of LeeAnne Walters, one of the residents who’d been complaining about the water. One sample, the state said, had come from a home with a filter. With all 71, the city was over the action level. Without the two, it was under, and for more than a year state officials told residents their water was safe.3 The highest reading in the pool came from a resident who had complained, and the percentile was what answered the complaint.

Three defects make the move possible, and none of them is a bug in the arithmetic. The institution whose performance is in question chooses the categories an index aggregates, so the person being scored has no standing over the taxonomy that scores them. The center of a distribution hides its tail, and compensatory labor concentrates in the tail. The same demand costs more the less reserve it lands on. So the depleted build the shadow spreadsheet while a mean or a comfortable percentile reports a working system. And compensation produces clean telemetry. One person absorbs the four hours of prerequisite gathering, and the application arrives. Another works the queue past hours, and every ticket closes. An index built to detect compensatory labor reports a good result in the presence of exactly the condition it was built to detect, without any dishonesty anywhere in the chain.

Gates, timers and rollback graphs watch the institution’s own state

A safer class of instruments inspects the state machine of the institution and its software, without tracking the biology of the worker.

The synchronous execution gate carries no surveillance footprint of its own. It refuses to execute a database write or queue an order unless a specific, authenticated, bound token authorizes it. It enforces the distinction between capability and authority at the API boundary. The deterministic countdown timer watches the payer’s own database entry: how long the institution has sat on a claim, with the clock resolving in the applicant’s favor when it expires. Pending is a no nobody signed, and the stop button any Toyota worker can pull is the same species of constraint aimed at live harm. The rollback graph is an internal database transaction. When an upstream premise is refuted, the dependency tree finds every downstream record that consumed it and reverses them, with zero visibility into human movement, keystrokes or off-duty hours. All three surveil power rather than vulnerability, which makes them far harder to turn against the person they were built for. Harder isn’t immune: their direction still depends on who controls eligibility, when the clock starts, who sets the token rules, who may grant exceptions, and whether retaliation for an honest reading is punished. Those safeguards have to be written down, or a machine-side meter becomes the next whip with better optics.

A reform that watches the worker gives management a weapon

The boundary between a structural reform and an oppressive one comes down to a single engineering rule: measure the state transitions of the machine, and never the nervous system of the operator.

A reform that records how fast a human reads, where their eyes move, how many clicks they make, or what they do away from the keyboard supplies data an employer can use against them. Calling the purpose protective doesn’t remove that incentive. Enforceable limits on access and use have to survive the next manager, too. What a capacity meter can instrument is not the operator’s biology but the workload the machine imposes: hours, breaks, staffing, error rates, the interval between demand and recovery. Those are state transitions of the machine and its schedule, so they sit on the machine’s side of the line. Capacity politics that passes the rule builds hard barriers into software and law and forces the institution to adjust its throughput to an unmonitored pace of human life. Capacity politics that fails the rule builds the instruments a manager will use on the people they were meant to protect.

The aggregate needs a second rule beside the first, and it’s the narrowest. An index may inform whether a particular objection deserves to be heard, and it may never be what dismisses the objection. A specific complaint is a fact about a person: their hours, their body, their household, their particular Tuesday. Its categories, weights and ceiling have to be contestable by the people it scores, and contesting them has to be able to change them. Crossing the ceiling has to produce a consequence for the institution rather than a reassurance for the individual. Refusing to measure at all would hand the argument to people who dismiss quantification on principle, and the compensatory labor stays invisible until somebody looks. What looking can’t supply is authority. A ward or a union asked to approve the next dashboard can ask two things before signing: does it watch the machine or the operator, and can the people it scores change what it counts?