Beyond the "Off Switch": The Case for Coactive AI in a World of Automated Decisions
The recent blacklisting of Anthropic by the Department of War (DoW) has stripped away the polite veneer of "AI Safety." This conflict wasn’t a technical glitch; it was a philosophical war over autonomy. The DoW demanded "all lawful use," including the ability for AI to identify and engage targets without a mandatory Human-in-the-Loop (HITL). Anthropic refused.
In the field of Human Factors (HF), this represents a dangerous push to move the user from the center of the system to the periphery, treating AI as an autonomous "blackbox" rather than a collaborative tool. It is dangerously naive to assume that maximizing automation is synonymous with maximizing efficiency. Instead, the high-stakes integration of AI requires a coactive design framework—one that ensures these systems remain under meaningful human moral and legal control, even when operating at machine speeds.
1. The Fallacy of the "Keyword Kill-Switch"
We are currently witnessing a catastrophic real-time experiment in substituting human judgment with crude automation. Recent revelations regarding the Department of Government Efficiency (DOGE) illustrate how the "efficiency" of AI can be weaponized to bypass due process. By using AI models to mass-cancel thousands of federal grants based on simple keyword searches, the system stripped away all meaningful context in favor of speed.
From a Human Factors perspective, this is a total collapse of the Joint Cognitive System (JCS) (Hollnagel & Woods, 2005). By deploying AI to flag and terminate grants without "Glass Box" reasoning, DOGE created a Moral Crumple Zone (Elish, 2019). The staff "reviewed" the lists, but because the AI provided no Uncertainty Metadata or contextual logic, there was no Common Ground (Klein et al., 2005)—the essential shared understanding between human and machine. Without knowing why the AI flagged a specific study, the human cannot effectively audit the decision.
Notably, while human oversight was technically present, the staffing of the agency with personnel who lacked deep domain expertise created a Mismatched Mental Model. These "reviewers" were unable to recognize when the AI was hallucinating political bias into historical research. The UI provided no Affordances for an operator to challenge the machine; it was built to facilitate compliance, not critical thinking.
Sidebar: The "120-Character" Trap
In the DOGE grant reviews, staffers reportedly utilized a prompt that forced the AI to generate a binary "Yes/No" regarding "DEI" content, constrained to under 120 characters.
The Result: The system suffered from Data Reduction Bias. The AI flagged a Holocaust study simply because it "focused on a marginalized culture"—a nuance completely erased by the character limit.
The Intentional Failure: Pairing a "Black Box" AI with an operator who lacks a robust mental model of the task isn't designing for efficiency; it is designing for Unaccountability.
2. The Automation Paradox: Why We Build "Silent Partners"
We are currently witnessing a frantic race toward full automation, driven by a fundamental misalignment of goals. On one side, there is the seductive pull of the "Easy Button"—the promise of "pure" automation that removes the burden of a task entirely. This is the Substitution Myth (Feltovich et al., 2004): the false belief that automation simply "replaces" human work without changing the nature of the task.
On the other side, there is the economic reality: it is mathematically simpler and cheaper to build a "Silent Partner" AI—a system that processes data in a vacuum and outputs a result without explanation. But as former Volkswagen CEO Bernd Pischetsrieder famously noted: “If you think good design is expensive, you should look at the cost of bad design.” When an organization prioritizes "Silent Partner" AI because it is "economically sound," they are ignoring the Lumberjack Effect (Onnasch et al., 2014): the higher the level of automation, the harder the system falls when it encounters a "Black Swan" event it wasn't programmed to handle. Automation doesn't simplify the task; it creates new, invisible cognitive work—monitoring, verifying, and recovering from errors. If the UI doesn't facilitate this, "quick" automation becomes a high-interest Interaction Debt that eventually comes due in legal fallout and the total loss of public trust and reputation loss. It is always "cheaper" to design a black-box command prompt than a UI that provides actionable, understandable information. Good design is hard because it requires anticipating failure.
To break this cycle, we must design for Coactivity, which requires two non-negotiable pillars (Christoffersen & Woods, 2002):
Observability: The AI must signal its internal state and intent, especially when operating at the edge of its competence.
Directability: The operator must be able to "nudge" or adjust the AI’s attention in real-time, redirecting focal parameters mid-stream.
3. Engineering Calibrated Trust
The path to preventing catastrophic AI failure is found in the engineering of Calibrated Trust (Lee & See, 2004). To understand why calibration is critical, we must look to the elevator. Elisha Otis didn’t actually invent the elevator; he spectacularly invented Trust in the elevator.
At the 1854 World’s Fair, Otis stood on a platform, hoisted it high, and ordered the hoisting cable to be cut. When the platform held fast, it wasn't because of a "friendly" interface; it was because a Hard Constraint, a mechanical safety brake, had snapped into place. Otis proved that trust is a structural guarantee. It is the result of making the system’s safety limits observable and its fail-safes absolute.
We must apply this same rigor to AI. Using "friendliness" to mask a lack of transparency is a form of Deceptive UX that leads to overtrust and disaster. Instead, we build trust by acknowledging that high-severity tasks require Human Authority Points. By realizing that AI and humans form a Joint Cognitive System, we can build in digital "safety brakes" that prevent the system from neglecting the user for the sake of speed. Any system that treats human oversight as a bottleneck to be bypassed is inherently malicious by architecture.
4. The Severity-to-Interaction Pipeline
To prevent failures like DOGE or the DoW’s pursuit of autonomous targeting, we must assign a Severity Tier to every interaction, dictating the Information Exchange Requirements of the UI:
We have already seen the lethal consequences of "Tier-Shifting"—treating a high-risk task as a low-risk automated one. Consider the Boeing 737 MAX and the MCAS system. Boeing designed MCAS as a "Silent Partner" that could override pilots based on a single sensor. By suppressing the pilot’s Situation Awareness (Endsley, 1995), Boeing didn't just cause a tragedy; they poisoned global trust in their engineering culture. When you remove the "safety brakes" from a high-stakes cable, you aren't seeking efficiency; you are seeking Unaccountability.
“Just as we cannot allow flight controls to operate in the dark, we cannot allow agentic AI to execute high-severity social or military decisions without the digital equivalent of a redundant sensor: the expert human mind.”
Conclusion: The Architecture of Accountability
The shift toward Calibrated Trust begins when cross-functional teams—engineers, designers, and product managers alike—move away from the allure of the "AI black box" and toward a shared architecture of accountability. The Service Blueprint must become our primary space to audit intent, ensuring that "efficiency" never overrides a design-led safety brake. In this blueprint, we strategically map exactly where a human remains a coactive, non-negotiable partner.
Being the "Ethical Voice" in these rooms can be difficult; it is far easier to facilitate increased KPIs than to pause a launch to demand a UI that plans for failure. However, by applying a Severity-to-Interaction framework, we move beyond the "Easy Button" to provide the Observability and Directability required when an AI reaches its limits. This transforms the UI into a functional Human Authority Point—turning the user from a passive witness into an active partner in a Joint Cognitive System, equipped to intervene before the cable snaps.
This modern implementation of service design is our version of Asimov’s Three Laws: a backbone for safe human-AI cohabitation. We must remember that in a world of automated drift, not every process should be automated. We are the architects who decide where the human must stay in control.
Elisha Otis didn’t just sell a brake; he sold the ability to build skyscrapers. As the safety engineers of the AI era, our job is to ensure that as the cables of automation get thinner, the "safety brakes" of Coactive Design get stronger. We must remind our stakeholders that while AI can move at machine speeds, only Coactivity ensures we don't hit the ground.
Human Factors Definition List
Joint Cognitive System (JCS): A system in which humans and technology act as co-agents to achieve a goal, emphasizing the interdependence of the two rather than seeing them as separate entities.
Moral Crumple Zone: A term describing how responsibility for a system's failure may be disproportionately attributed to a human operator, even when the failure was primarily driven by the system's design or automation.
Lumberjack Effect: The phenomenon where higher levels of automation lead to a more severe "fall" (failure) because the human operator has lost the situational awareness required to recover the system manually.
Substitution Myth: The mistaken belief that automation "replaces" a human task. In reality, automation transforms the task, often introducing new, more complex cognitive demands.
Calibrated Trust: The state where a user’s trust in an automated system accurately matches the system’s actual capabilities and reliability.
Observability: The degree to which a system allows a human operator to perceive and understand its internal state, logic, and future intentions.
Directability: The ability of a human operator to influence or "nudge" an automated system's behavior and focus in real-time.
Automation Bias: The human tendency to favor suggestions from automated systems and ignore contradictory information, even when the automated suggestion is incorrect.
Common Ground: The shared understanding and mutual knowledge between two agents (e.g., a human and an AI) that is required for effective coordination.
Situation Awareness (SA): The perception of elements in the environment, the comprehension of their meaning, and the projection of their status in the near future.