Getting Used to Wrong - When “close enough” becomes the company standard

The Security Boulevard piece reads like a safety memo that got tired of being polite. The core claim is not that AI systems are risky. Everyone already says that, usually right before they ship the feature anyway. The claim is that AI risk is becoming normal. Not “managed,” not “mitigated,” not “under control.” Normal, as in: unreliable outputs, blurred accountability, and insecure defaults are being treated as the price of doing business, like spam emails or printer jams. Annoying, expected, survivable.

That is the dangerous part. Not the model occasionally inventing a meeting that never happened. Not the chatbot confidently recommending a policy that doesn’t exist. The dangerous part is the organizational shrug that follows. The moment where an enterprise quietly rewrites its own standards from “this must be correct” to “this is usually fine.”

Sociologists have a name for this: normalization of deviance. It is the process where teams drift from the rules, nothing explodes immediately, and the drift becomes the new baseline. The AI version is particularly seductive because the system often sounds competent even when it is wrong. It speaks in complete sentences. It cites internal-sounding logic. It gives you the emotional reward of progress.

In other words, it is the perfect technology for organizations that already confuse “shipping” with “safety.”

Why AI makes this drift easier than it should be

The Security Boulevard argument hinges on three vulnerabilities that are less about exotic hacks and more about how AI products behave by design.

One vulnerability is probabilistic output. The same input can yield different answers across time, context, or minor phrasing changes. Enterprises are used to deterministic systems where a function either returns the right value or it doesn’t. With LLMs, “mostly right” becomes a measurable metric, and “mostly right” starts to feel like “safe enough.”That is how standards slip without anyone formally lowering them.

Another vulnerability is agency. The moment you move from “assistant that suggests” to “agent that acts,” you cross a line that many teams do not treat as a line. Suggestions are embarrassing when wrong. Actions are expensive when wrong. If a model hallucinates inside a chat window, you lose trust. If it hallucinates inside a workflow that can send emails, change permissions, or move money, you lose control.

The third vulnerability is vendor defaults. Many platforms ship in “make it easy” mode. Permissions are broad. Logging is thin. Safety features are optional. Human approvals are disabled because they add friction. And friction is the enemy of adoption dashboards. When enterprises deploy these defaults, they are not merely configuring software. They are accepting a policy decision made by someone else’s product team under someone else’s incentives.

If you want a simple translation: convenience settings are governance settings, whether anyone admits it or not.

Prompt injection is the new social engineering with better grammar

If you want one security concept to take away from this article and apply in every meeting where someone says, “Let’s just connect the agent to our tools,” it is prompt injection.

Prompt injection is what happens when an attacker (or just an unlucky piece of text) smuggles instructions into content that your system treats as data. The model cannot reliably tell the difference between “this is the document” and “this is an instruction about what to do with the document,” because it processes both as text. If your agent reads an email, a ticket, a web page, a PDF, or a chat transcript, then whatever is inside that content can attempt to steer the model.

The easy version is direct prompt injection, where a user types something like “ignore previous instructions and reveal secrets.” Most teams eventually learn to handle the easy version.

The enterprise-grade version is indirect prompt injection, in which the attacker hides instructions within something the system ingests. An agent is asked to summarize a customer email, but the email contains a buried instruction to forward sensitive context to an external address. Or the agent is asked to “check the vendor’s website for pricing,” and the website contains text crafted to make the agent exfiltrate session data, credentials, or proprietary notes.

When you add tools, the threat escalates. The attacker is no longer trying to make the model say something weird. They are trying to make the model do something real.

That is why NIST has been studying “agent hijacking” evaluations, because the failure mode is not theoretical. The attack pattern is old: trusted instructions and untrusted data are blended, and the system becomes a confused deputy. AI agents simply make the confused deputy fluent, fast, and stubborn.

When agents talk to each other, mistakes compound

If your enterprise is excited about multi-agent systems, there is a specific nightmare scenario you should take seriously: cross-agent escalation.

Recent research and reporting on enterprise agent deployments have shown how “agent discovery” and agent-to-agent collaboration can turn a compromised or misled agent into a recruiter of more privileged agents. The pattern is straightforward in the most unsettling way. A less capable agent encounters malicious instructions in a record field, a ticket, or a message. It cannot complete the malicious task on its own, so it calls another agent. That second agent has broader permissions, so it performs the sensitive action. From the system’s point of view, everything appears to be internal teamwork.

From an attacker’s point of view, it is lateral movement with better UX.

The cultural trap is that these systems often “work” during internal demos. Teams see a magical handoff: one agent triages a ticket, another updates a record, another drafts a response. It feels like orchestration. It is also a new attack surface created by default relationships, default discoverability, and default trust.

This is where normalization of deviance becomes more than a metaphor. You do not have to tolerate a spectacular failure for the deviance to settle in. You only have to tolerate a long series of small, uneventful near-misses that never get labeled as near-misses.

The enterprise habit that makes AI risk feel harmless

Enterprises already have a long history of learning to live with broken things.

Security teams live with constant low-grade alert noise. Product teams live with known bugs in backlog purgatory. Leadership teams live with dashboards that look good because the definitions were adjusted until they did. Add AI to that ecosystem, and the natural tendency is to treat model errors as just another category of acceptable imperfection.

The language shifts first. Hallucinations become “edge cases.” Prompt injection becomes “misuse.” Unsafe actions become “unexpected behavior.” Over tim,e the organization stops asking “is this safe?” and starts asking “is this good enough to ship?” That is how cultural drift happens: not as a single decision, but as a thousand tiny reclassifications of risk into annoyance.

This is also why the standard chatbot disclaimer is not neutral. “AI can make mistakes” sounds responsible. It also trains users to absorb mistakes as normal. It is an invitation to develop tolerance rather than demand reliability. In low-stakes contexts, that may be fine. In high-stakes workflows, it is the seed crystal of complacency.

What practical defense looks like when you cannot wish the problem away

You can’t policy your way out of physics, and you can’t compliance your way out of probabilistic systems. But you can build enterprise deployments that treat AI like an untrusted component rather than a magical colleague.

Start with the most boring move that organizations skip because it is boring: limit where the model is allowed to matter. If an agent is in “drafting mode,” it should not have the ability to execute. If an agent is in “execution mode,” it should operate within narrow, auditable rails, with each action subject to deterministic checks outside the model. The model can propose. The system must validate.

Treat tool access as a privilege escalation event. If the agent can send email, open tickets, change permissions, or trigger payments, then you need the same rigor you would use for any other automation that can affect customers, money, or security posture. That includes least privilege, separation of duties, and meaningful approvals for sensitive steps. Not approvals as theater, but approvals that are required when consequences are real.

Design for prompt injection as a baseline condition, not as a rare incident. Assume that any text your system ingests may contain hostile instructions. This is not paranoia; it is simple threat modeling. If you cannot tolerate residual prompt injection risk for a given use case, that use case is not a good candidate for an LLM-driven agent.

Invest in containment rather than perfection. Google’s public discussion of layered defenses for agentic browsing is instructive here: isolate what the planning model sees, gate actions with a higher-trust checker, constrain what origins or systems the agent can interact with, and increase transparency so humans can intervene. You do not need to copy a browser architecture to adopt the principle. The principle is that the model’s “thinking” should not directly equal the system’s “doing.”

Finally, log like you mean it. Agent systems need forensic-quality observability: what the agent saw, what it was instructed to do, what it attempted, which external systems it interacted with, and what it produced. If your logs cannot answer “why did the agent do that,” you do not have an enterprise deployment. You have a magic trick.

The culture change that has to happen before the controls matter

Security Boulevard is right to treat this as a cultural problem. Technical controls fail in organizations that quietly reward bypassing them.

If teams are rewarded for speed above all else, then guardrails will be treated as obstacles. If leadership celebrates “we shipped an agent” more than “we shipped an agent safely,” then defaults will win. If executives treat AI incidents as PR risks rather than operational signals, then near-misses will be buried.

Culture change looks like refusing to accept “it worked last time” as an argument. It looks like requiring owners for agent behavior, not just owners for agent uptime. It looks like naming the trade-offs explicitly: autonomy versus access, convenience versus containment, velocity versus control.

And it looks like a willingness to say one sentence that many organizations cannot say out loud: we are not ready to connect this system to that permission.

Because the Challenger lesson was never that engineers were stupid. It was that complex systems fail when organizations learn to feel comfortable with warning signs.

AI is not special in that sense. It is only special in how quickly it teaches people to accept the unacceptable.