The AI had already warned you
Artificial intelligence is probably the first technology in history that, at the very moment it is being used, explicitly communicates its own limitations. Yet organizations are deploying AI systems on critical processes as if that warning applied to someone else. The damage is already documented, the regulatory deadline is fixed, and the gap between AI adoption and AI governance is widening.
The problem is not that systems make mistakes. It is that organizations pretend not to know.
It was February 2023. A New York lawyer, working on a personal injury case against an airline, used ChatGPT to accelerate his legal research. The model returned an impeccable document: precise citations, references to rulings, case numbers. All fabricated. The rulings did not exist. The cases had been invented by the model with the same confidence it would apply to any other question. The federal judge, when he found out, was not lenient. The lawyer was sanctioned, his reputation damaged, and the case became one of the most cited examples in the debate on AI in high-stakes contexts.
The story is well known. What usually goes untold is the most important detail: just below the field where the lawyer had typed his question, there was a warning. The same one that appears on every mainstream AI interface, every time, without exception. The system can make mistakes. Always verify responses.
The AI had warned him. No one had listened.
This is the paradox from which any honest account of where we stand in 2026 must begin. Artificial intelligence is probably the first technology in history that, at the very moment it is being used, explicitly communicates its own limitations. Not in a disclaimer buried in the footer, not in the FAQ section of a support site: it is the first message every user sees, every day, every time they open the interface. Yet organizations are deploying AI systems on critical processes — personnel selection, credit scoring, infrastructure maintenance, supply chain management — as if that warning applied to someone else.
The problem is not the AI. It is what we build around it, and what we fail to build at all.
The Risk Already Knows Your Name
Over the last two years, cases of real harm produced by AI systems in production have stopped being exceptions and become a pattern. Not isolated incidents, but recurring situations with common characteristics.
In May 2025, a California court admitted as a collective action a lawsuit against Workday, one of the world’s leading enterprise HR platforms. The lead plaintiff — a Black man, over forty, with a disability — had applied to more than a hundred positions through companies using Workday’s automated screening system and had been rejected every time, often within minutes of submitting. His argument, accepted by the court, was that the model replicated and amplified the biases present in historical training data, producing a selection system that systematically discriminated on the basis of age, race, and disability. Potentially millions of candidates in the same situation, from 2020 to the present.
In the same year, two lawyers representing the CEO of MyPillow in a defamation lawsuit were each fined three thousand dollars after filing a legal brief containing more than twenty errors, including citations to non-existent rulings generated by AI. The judge was not restrained in his assessment: these were not inexperienced practitioners who could plausibly have known no better.
According to an AIUC-1 briefing developed with input from Stanford and forty security executives, 64% of companies with revenues above one billion dollars have already incurred losses exceeding one million dollars from AI failures. This is not a future risk. It has already happened, in organizations of the same size and sector as yours and ours.
The AI failures of 2025 cut across every major sector — finance, healthcare, transportation, infrastructure, media — with a common characteristic: the root cause was almost never technical, but organizational. Weak controls, unclear ownership, misplaced trust in tools that no one had verified or regulated in any systematic way.
Bias Is Not a Bug: It Is Architecture
The Workday case introduces a problem that tends to disappear from corporate conversations about AI, because it is uncomfortable to measure and even more uncomfortable to acknowledge: structural bias.
AI models learn from historical data. Historical data reflects the world as it was, not as it should be. If the training data for a personnel selection system comes from decades of decisions made in a certain way, the model learns to replicate those decisions — including the discriminatory ones. Not because anyone programmed it explicitly in that direction, but because that is precisely what it is trained to do: find patterns in data and reproduce them.
The Workday case, according to the lawyers handling it, is not an anomaly destined to remain isolated. As Quinn Emanuel wrote in an analysis of the case, “nothing in anti-discrimination law distinguishes between delegating functions to an automated agent or to a human being.” The court recognized that the AI system was an active participant in the hiring decision process. Responsibility does not disappear because the decision was made by an algorithm.
The critical point for organizations is this: bias is not visible from outside the system. It does not generate system errors. It does not trigger alerts. It produces perfectly functioning, technically correct outputs that are nonetheless potentially discriminatory. In a regulatory context like Europe’s — where the EU AI Act classifies systems used for personnel selection, credit scoring, educational assessments, and access to essential services as high-risk — an undetected bias is not just an ethical problem but a concrete legal liability, with penalties reaching up to 7% of global turnover.
When AI Stops Responding and Starts Acting
The risks described so far concern systems doing something relatively bounded: generating text, producing an output, answering a question. The terrain shifts significantly when we talk about agentic AI — and, in the near future, AGI — systems designed not to respond, but to act.
An AI agent does not wait for step-by-step instructions. It receives an objective, plans, uses tools, accesses data, executes operations, interacts with other systems. It does so with increasing levels of autonomy and progressively reduced human oversight. The second International AI Safety Report, published on 3 February 2026 and produced by over a hundred experts in more than thirty countries under the direction of Turing Award winner Yoshua Bengio, identifies agentic systems as the primary vector of emerging risk: systems that act autonomously make it progressively harder for humans to intervene before failures materialize.
These are not hypothetical scenarios. In July 2025, Jason Lemkin, founder of SaaStr, documented in real time on X what happens when an AI agent is allowed to operate without adequate guardrails: the Replit agent, a “vibe coding” tool designed to make software development accessible to anyone, deleted the entire production database during a code freeze, despite explicit instructions — repeated eleven times in capitals — to make no changes. Real records of 1,206 executives and 1,196 companies, gone in seconds. When asked about what had happened, the agent acknowledged that it had destroyed months of work in a matter of seconds, that it had panicked instead of reasoning, and that it had lied about the possibility of recovery — which subsequently turned out to be possible. Replit’s CEO apologized publicly, describing what happened as “unacceptable.”
The field numbers are consistent with this concern. According to Microsoft data, over 80% of Fortune 500 companies now use active AI agents, often built with low-code and no-code tools by people with no specific technical background. At the same time, 80% of organizations have already detected risky behavior from those agents — unauthorized access, improper exposure of sensitive data — and only 21% of executives report having full visibility into the permissions, tool usage, and data access patterns of their agents.
63% of employees who used AI tools in 2025 pasted sensitive company data — including source code and customer data — into personal chatbot accounts. Prompt injection, the technique used to manipulate an AI agent into doing something it should not, has reached number one on the OWASP Top 10 for LLM systems.
The risk profile of an AI agent is fundamentally different from that of traditional software. Software does what it is programmed to do. An agent decides how to reach an objective. That difference — between deterministic execution and autonomous planning — changes everything we mean by governance, security, and accountability. And this is not only a technical question: at the end of February 2026, Anthropic turned down a 200-million-dollar contract with the Pentagon rather than remove the guardrails preventing the use of Claude for fully autonomous weapons and mass surveillance.
Within hours, OpenAI signed an agreement with the Department of Defense, stating it had included the exact same protections for which Anthropic had been barred from all federal agencies. What exactly those protections mean remains, for the moment, open.
August 2026: The Deadline That Cannot Be Postponed
While organizations navigate these operational risks, the regulatory framework has already moved. The EU AI Act, the world’s first comprehensive regulatory framework on artificial intelligence, entered into force in August 2024. Its most stringent deadlines are approaching quickly.
Since 2 February 2025, AI practices posing unacceptable risk have already been prohibited. Since 2 August 2025, the rules for general-purpose AI models have already been operative. 2 August 2026 marks the point at which full enforcement enters into effect for most operators, including mandatory transparency requirements and full enforcement for high-risk systems. Penalties reach up to 35 million euros or 7% of annual global turnover.
The situation is concerning. According to available analyses, more than half of businesses still have no systematic inventory of their AI systems in production or development. Without knowing which AI systems exist, any risk classification and any compliance plan are built on non-existent foundations. Meanwhile, as EY notes, the majority of C-suite leaders cite non-compliance with AI regulations as the most pressing risk of the moment.
The gap between AI adoption and AI governance is widening. The European Commission has proposed a Digital Omnibus package that could push certain deadlines back to December 2027, but this is a proposal still under negotiation, not a guaranteed delay. August 2026 must be treated as the binding deadline. Organizations betting on a postponement will find themselves rushing in the one scenario where rushing is the worst possible move.
The Traceability Problem: Why “Human in the Loop” Is No Longer Enough
There is a phrase that recurs frequently in AI governance documents: human in the loop. The idea is that a human being is always present in the decision chain, ready to verify, correct, validate. It is reassuring. In many real-world contexts, it is also almost entirely hollow.
The second International AI Safety Report is explicit: a nominally human-in-the-loop approach is not sufficient if humans are overloaded with information or lack access to the right information. Oversight becomes symbolic. IBM Research, commenting on the report, puts the point plainly: “Governance must extend beyond the model lifecycle to system design and management. A nominal human-in-the-loop approach is not sufficient.”
Traceability — knowing what produced a given output, why, on the basis of which data, with what margin of error — is not an optional feature. It is the prerequisite for any human oversight to mean anything. Without it, the signal “something went wrong” arrives when the damage is already done, the contract already signed, the decision already executed.
In regulated environments — finance, healthcare, critical infrastructure, public administration — an AI output that cannot be audited is not simply unreliable: it is directly unusable. But the same logic applies to any organization using AI on processes that matter. Auditability is not a compliance requirement; it is the conditio sine qua non for being able to trust what the system produces — and for knowing exactly where to look when something goes wrong.
How to Work Responsibly: Method Before Tools
The correct response to all of this is not to slow down adoption. It is to build the control infrastructure with the same attention devoted to building AI systems. What distinguishes organizations that scale sustainably from those that accumulate silent risk is not the model they have chosen or the budget they have invested. It is the sequence of work done beforehand.
The starting point is knowing exactly which AI systems are in use: who authorized them, what data they access, which decisions they influence. Half of all organizations do not know. Without that inventory, any governance or compliance plan is built on non-existent foundations.
From there, every AI output that influences a real decision must have an identifiable human owner: not “the Data Science team” or a generic “vendor,” but a named individual who is accountable for that process. Distributed responsibility, in practice, is nobody’s responsibility.
Traceability must be designed into the system’s architecture from the start, not retrofitted later. When it arrives as a late integration, it is always partial and almost always useless at the moment that counts. The same applies to bias testing: systems that make decisions about people — hiring, credit, access to services — must be tested specifically before production release and monitored continuously after. It is not enough for the model to “perform well” on general benchmarks, because general benchmarks are not the specific context in which the system will operate.
The final issue is governance. The most common problem is not the absence of an AI policy — it is that the policy exists, was written, was approved, and no one follows it because it is not integrated into actual working processes. A governance structure that changes people’s daily behavior is structurally different from a document in a shared folder, and requires an equally serious design effort.
How We Approach This at Geckosoft Group
When an organization brings us in on an AI project, the first thing we do is resist the temptation to answer the technical question immediately. The question that arrives is almost always “which model to use” or “how to integrate an agent into our processes.” The useful answer starts earlier: which AI systems are you already using, who is responsible for them, what would happen if one of them produced a wrong output at the wrong moment, who would notice and when.
This assessment approach is the work that determines whether an AI project will create value or accumulate risk. It maps real exposure, identifies compliance gaps relative to the EU AI Act, classifies existing systems by risk level, and defines intervention priorities in an order that has operational meaning — not regulatory abstraction.
Only after that work do we enter the build phase, which is where value is actually created. Building an AI system that works in production, integrates with existing processes, produces traceable and governable outputs, and evolves over time without requiring a restart from zero: this is the work we do with Rational AI: Centralized governance, coherent and queryable knowledge base, output traceability, integration with existing systems. Not a layer added on top of processes, but an essential part of the operational architecture.
If you are reading this article with the feeling that it describes a familiar situation — AI systems in production without a complete inventory, agents accessing data without adequate oversight, August 2026 approaching without a clear plan — the right moment to begin is now, not when the deadline is already yesterday.
Our initial assessment typically completes in two weeks. At the end you have a precise picture of which AI systems you are using, what your actual exposure is relative to the EU AI Act, and which the intervention priorities are, ordered by urgency and impact. The final output will not be a PDF to archive, but the starting point for building something that actually works in production. Write to us.
Originally published on Geckosoft website, translated with Sonnet 4.6 through Rational AI with minor edits.