Beyond the "Off Switch": The Case for Coactive AI in a World of Automated Decisions
Joseph Everett Grgic • March 22, 2026
In early 2025, a quiet but consequential conflict played out between Anthropic and the U.S. Department of Defense. The DoD sought AI systems cleared for “all lawful use”—explicitly including the ability to identify and engage targets without a mandatory Human-in-the-Loop (HITL) constraint. Anthropic refused. What the Pentagon framed as an operational requirement, Anthropic recognized as a philosophical line: one that would move the human from the center of the system to its periphery. The contract fell through. The polite veneer of “AI Safety” had been stripped away.
From the perspective of Human Factors (HF), this was not merely a policy dispute—it was a warning. The push toward autonomous decision-making treats AI as a replacement for human judgment rather than an extension of it. It reflects a dangerous assumption: that maximizing automation is synonymous with maximizing efficiency. The evidence, from aviation to public administration, tells a different story. High-stakes AI integration requires a coactive design framework—one that keeps humans as meaningful moral and legal authorities, even when systems operate at machine speeds.
1. The Fallacy of the “Keyword Kill-Switch”
We are currently witnessing a catastrophic real-time experiment in substituting human judgment with crude automation. Recent revelations regarding the Department of Government Efficiency (DOGE) illustrate how the “efficiency” of AI can be weaponized to bypass due process. By using AI models to mass-cancel thousands of federal grants based on simple keyword searches, the system stripped away all meaningful context in favor of speed.
From a Human Factors perspective, this is a total collapse of the Joint Cognitive System (JCS) (Hollnagel & Woods, 2005). By deploying AI to flag and terminate grants without “Glass Box” reasoning, DOGE created a Moral Crumple Zone (Elish, 2019). The staff “reviewed” the lists, but because the AI provided no Uncertainty Metadata or contextual logic, there was no Common Ground (Klein et al., 2005)—the essential shared understanding between human and machine. Without knowing why the AI flagged a specific study, the human cannot effectively audit the decision.
Critically, while human oversight was technically present, the staffing of the agency with personnel who lacked deep domain expertise created a Mismatched Mental Model. These “reviewers” were unable to recognize when the AI was hallucinating political bias into historical research. The UI provided no Affordances for an operator to challenge the machine; it was built to facilitate compliance, not critical thinking.
Sidebar: The “120-Character” Trap
In the DOGE grant reviews, staffers reportedly used a prompt that forced the AI to generate a binary “Yes/No” regarding “DEI” content, constrained to under 120 characters.
The Result: The system suffered from Data Reduction Bias. The AI flagged a Holocaust study simply because it “focused on a marginalized culture”—a nuance completely erased by the character limit.
The Intentional Failure: Pairing a Black Box AI with an operator who lacks a robust mental model isn’t designing for efficiency—it is designing for Unaccountability.
2. The Automation Paradox: Why We Build “Silent Partners”
We are currently witnessing a frantic race toward full automation, driven by a fundamental misalignment of goals. On one side, there is the seductive pull of the “Easy Button”—the promise of “pure” automation that removes the burden of a task entirely. This is the Substitution Myth (Feltovich et al., 2004): the false belief that automation simply “replaces” human work without changing the nature of the task.
On the other side, there is the economic reality: it is mathematically simpler and cheaper to build a “Silent Partner” AI—a system that processes data in a vacuum and outputs a result without explanation. But this is a false economy. As a principle well-established in product design holds: the cost of bad design always exceeds the cost of good design. When an organization prioritizes “Silent Partner” AI because it is “economically sound,” they are ignoring the Lumberjack Effect (Onnasch et al., 2014): the higher the level of automation, the harder the system falls when it encounters a “Black Swan” event it wasn’t programmed to handle.
Automation doesn’t simplify the task; it creates new, invisible cognitive work—monitoring, verifying, and recovering from errors. If the UI doesn’t facilitate this, “quick” automation becomes a high-interest Interaction Debt that eventually comes due in legal fallout and the total loss of public trust. It is always “cheaper” to design a black-box command prompt than a UI that provides actionable, understandable information. Good design is hard because it requires anticipating failure.
To break this cycle, we must design for Coactivity, which requires two non-negotiable pillars (Christoffersen & Woods, 2002):
• Observability: The AI must signal its internal state and intent, especially when operating at the edge of its competence.
• Directability: The operator must be able to “nudge” or adjust the AI’s attention in real-time, redirecting focal parameters mid-stream.
3. Engineering Calibrated Trust
The path to preventing catastrophic AI failure is found in the engineering of Calibrated Trust (Lee & See, 2004). To understand why calibration matters, we must look to the elevator. Elisha Otis didn’t actually invent the elevator; he spectacularly invented Trust in the elevator.
At the 1854 World’s Fair, Otis stood on a platform, hoisted it high, and ordered the hoisting cable to be cut. When the platform held fast, it wasn’t because of a “friendly” interface; it was because a Hard Constraint—a mechanical safety brake—had snapped into place. Otis proved that trust is a structural guarantee. It is the result of making a system’s safety limits observable and its fail-safes absolute.
We must apply this same rigor to AI. Using “friendliness” to mask a lack of transparency is a form of Deceptive UX that leads to overtrust and disaster. Instead, we build trust by acknowledging that high-severity tasks require Human Authority Points. By recognizing that AI and humans form a Joint Cognitive System, we can build in digital “safety brakes” that prevent the system from sidelining the user for the sake of speed. Any system that treats human oversight as a bottleneck to be bypassed is inherently malicious by architecture.
4. The Severity-to-Interaction Pipeline
To prevent failures like DOGE or the DoD’s pursuit of autonomous targeting, every AI-mediated action must be assigned a Severity Tier that dictates the Information Exchange Requirements of the UI. The framework operates on a spectrum:
• Tier 1 – Low Severity: Routine, reversible tasks (e.g., content sorting, scheduling suggestions). AI can act with light-touch logging. Human review is retrospective.
• Tier 2 – Moderate Severity: Consequential but correctable decisions (e.g., flagging applications, prioritizing resources). AI must surface its reasoning. Human confirmation is required before action executes.
• Tier 3 – High Severity: Irreversible or high-stakes outcomes (e.g., grant terminations, targeting decisions, medical triage). Full Glass Box reasoning is mandatory. A qualified human authority must actively approve—not merely “review.”
The critical danger is Tier-Shifting: when organizations treat high-severity tasks as low-severity ones to gain speed. We have already seen its lethal consequences in the Boeing 737 MAX and the MCAS system. Boeing designed MCAS as a “Silent Partner” that could override pilots based on a single sensor. By suppressing the pilot’s Situation Awareness (Endsley, 1995), Boeing didn’t just cause a tragedy—they poisoned global trust in their entire engineering culture.
“Just as we cannot allow flight controls to operate in the dark, we cannot allow agentic AI to execute high-severity social or military decisions without the digital equivalent of a redundant sensor: the expert human mind.”
Conclusion: The Architecture of Accountability
The shift toward Calibrated Trust begins when cross-functional teams—engineers, designers, ethicists, and policy leads—move away from the allure of the “AI black box” and toward a shared architecture of accountability. The Service Blueprint must become the primary space to audit intent, ensuring that “efficiency” never overrides a design-led safety brake. In this blueprint, we map exactly where a human remains a coactive, non-negotiable partner.
Being the “Ethical Voice” in these rooms can be difficult; it is far easier to report increased KPIs than to pause a launch to demand a UI that plans for failure. However, by applying a Severity-to-Interaction framework, we move beyond the “Easy Button” to provide the Observability and Directability required when an AI reaches its limits. This transforms the UI into a functional Human Authority Point—turning the user from a passive witness into an active partner in a Joint Cognitive System, equipped to intervene before the cable snaps.
This approach is our generation’s answer to the challenge Asimov’s Three Laws posed theoretically: how do you encode ethical constraints directly into the architecture of an autonomous system? Asimov’s laws were embedded in the robot’s hardware; our “laws” are embedded in the UI, the approval workflow, and the severity tier. They are the backbone for safe human-AI cohabitation—not as a philosophical ideal, but as a structural design requirement.
We must remember that not every process should be automated. We are the architects who decide where the human must stay in control.
Elisha Otis didn’t just sell a brake; he sold the ability to build skyscrapers. As the safety engineers of the AI era, our job is to ensure that as the cables of automation get thinner, the “safety brakes” of Coactive Design get stronger. We must remind our stakeholders that while AI can move at machine speeds, only Coactivity ensures we don’t hit the ground.
Human Factors Definition List
• Joint Cognitive System (JCS): A system in which humans and technology act as co-agents to achieve a goal, emphasizing their interdependence rather than treating them as separate entities.
• Moral Crumple Zone: The phenomenon where responsibility for a system’s failure is disproportionately attributed to a human operator, even when the failure was primarily driven by the system’s design or automation.
• Lumberjack Effect: The phenomenon where higher levels of automation lead to a more severe failure because the human operator has lost the situational awareness required to recover the system manually.
• Substitution Myth: The mistaken belief that automation “replaces” a human task. In reality, automation transforms the task, often introducing new, more complex cognitive demands.
• Calibrated Trust: The state where a user’s trust in an automated system accurately matches that system’s actual capabilities and reliability.
• Observability: The degree to which a system allows a human operator to perceive and understand its internal state, logic, and future intentions.
• Directability: The ability of a human operator to influence or “nudge” an automated system’s behavior and focus in real-time.
• Automation Bias: The human tendency to favor suggestions from automated systems and ignore contradictory information, even when the automated suggestion is incorrect.
• Common Ground: The shared understanding and mutual knowledge between two agents (e.g., a human and an AI) that is required for effective coordination.
• Situation Awareness (SA): The perception of elements in the environment, the comprehension of their meaning, and the projection of their status in the near future.