Let's talk

The Future of AI

Agent Liability Law: Navigating Accountability in an Agentic World

As AI agents evolve from simple chatbots to autonomous actors capable of executing workflows, the legal and operational focus is shifting from intelligence to liability.

By Divyaprakash JhaUpdated 5 min readTopic: Future of AI

Photo for Agent Liability Law: Navigating Accountability in an Agentic World

The Core Challenge of Agent Liability

As AI agents move from "assisting" to "acting," the primary challenge for businesses is no longer just about how smart the AI is, but about who is legally and operationally responsible when an agent makes a mistake. In a world of autonomous agents, the hardest problems to solve are identity, payment authority, and determining who is liable when an agent negotiates or transacts on behalf of a human.

We are entering an agentic economy where the focus is shifting from simple chat interfaces to autonomous actors capable of executing entire workflows. For a business owner in India, this might mean an agent that handles GST paperwork or follows up with leads via WhatsApp. While this increases efficiency, it introduces a massive friction point: liability. If an agent promises a discount it wasn't supposed to, or sends an incorrect invoice, the question isn't just "why did the AI do that?" but "who pays for the error?"

As these systems evolve, the legal and operational focus is moving from intelligence to accountability. We are moving toward a future where agents are not just tools, but digital entities that require clear boundaries of authority.

The Ladder of Trust: Managing Risk Through Design

To manage the risks of autonomous action, businesses cannot simply "turn on" an agent and walk away. Instead, they must move agents up a "ladder of trust" through intentional engineering. You cannot jump from a chatbot to a fully autonomous employee in one step; you must climb the rungs one by one.

Rung Mode of Operation Human Role Example Use Case
1. Assist The agent drafts content or plans. The human reads every word and hits "send." Drafting a customer email or a sales proposal.
2. Act with Approval The agent prepares a specific action. The human clicks "approve" on a specific task. Preparing a refund or a bank transfer for review.
3. Act within Limits The agent acts alone up to a threshold. The human sets the limits and audits the results. Approving small refunds or reordering stock.
4. Run the Process The agent owns a workflow end-to-end. The human reviews summaries and exceptions. Managing the entire accounts receivable process.

My forecast is that by 2028, many routine business processes will sit on rung two or three. By the early 2030s, a good share of back-office work in well-run companies will sit on rung four. If you design your systems so they can operate at rung two—with clear proposed actions and approvals—you can move them up the ladder later without rebuilding the entire system.

The Rise of AI Audit and Verification

As agents move from drafting to acting, the need for verification becomes a professional requirement. We expect "AI audit, evaluation, and assurance" to become recognized professions, much like financial auditing is today. Just as a company hires an auditor to sign off on their books, businesses will soon require experts to sign off on their AI agents.

Verification involves several critical layers:

  • Testing: Proactively trying to break the agent to see how it handles errors.
  • Log Review: Examining the step-by-step history of an agent's decisions.
  • Assurance: Providing formal sign-off that a system behaves exactly as the business claims it does.

To make this possible, robust audit logs are essential. Without a clear, unchangeable record of every decision and action an agent takes, it is impossible to trace a mistake or undo a wrong transaction. In the agentic commerce landscape, these logs will be the primary evidence used to settle disputes between agents and humans.

Security and Operational Risks

Giving an agent the power to act is equivalent to giving it a credit card and a company email address. This creates a high-value target for attackers. One of the most significant, and currently unsolved, threats is prompt injection. This occurs when hidden instructions—perhaps tucked away in a website or a document the agent is reading—hijack the agent's behavior, forcing it to ignore its original instructions and follow the attacker's instead.

For a small business owner, this could mean an agent reading a malicious email and, instead of filing it, transferring funds to a fraudulent account. To mitigate these risks, builders and owners must implement three core principles:

  1. Least Privilege: An agent should only have the minimum permissions necessary to do its job. An agent that manages customer queries should not have the technical ability to change payroll records.
  2. Strict Approval Steps: High-stakes actions, such as large financial transfers, must always require a human "handshake."
  3. Comprehensive Audit Logs: Every action must be recorded in a way that allows for a full forensic reconstruction of what happened.

The Human Role: Ownership and Judgment

There is a common fear that AI agents will replace human responsibility. The reality is likely the opposite. As AI makes technical and repetitive tasks cheaper, human judgment, trust, and responsibility become more valuable, not less.

An AI can propose an action, but a human must still own the outcome. You can automate the "doing," but you cannot automate the "owning." In a world where anyone can generate a polished, professional-looking proposal in seconds using an agent, the business that wins the contract will be the one that provides a trusted name and a real relationship. The human element provides the "proof of personhood" that agents currently lack.

As we move toward the future of AI, the most successful businesses will be those that combine agentic efficiency with a trusted human name. The human role shifts from being a "doer" of tasks to being a "supervisor" of systems, focusing on high-level strategy, complex problem-solving, and maintaining customer relationships.

How Builders Can Mitigate Liability

If you are building agentic software, your goal is to build "durable workflows." A durable workflow is one that doesn't just "chat," but actually completes a job through state management, retries, and scheduled checkpoints. To build for the long term and minimize liability, follow these guidelines:

  • Design for Checkpoints: Build your architecture so that an agent can pause and wait for human approval at critical junctions. This makes it easy to move the agent up the "ladder of trust" later.
  • Use Open Protocols: Build on open standards like the Model Context Protocol (MCP). This prevents vendor lock-in and gives you more control over how your agents interact with different tools and data.
  • Integrate Auditing from Day One: Do not treat logging and evaluation as an afterthought. If you want to sell your agent to a large enterprise, you must be able to prove exactly how it makes decisions.

The direction of the industry is clear: AI is moving from answering to acting. The builders who succeed will be those who focus on making that action reliable, verifiable, and safe.

Aerial view of a modern city at night with illuminated skyscrapers and streets.
Photo: Free Tokyo city lights by unknown, CC0

Questions people ask

Who is liable when an AI agent makes a mistake?

As agents move up the ladder of trust, liability often rests with the human or business that sets the agent's limits and provides the final approval for high-stakes actions.

How can I reduce the risk of an AI agent acting incorrectly?

Implement a 'human-in-the-loop' system where the agent requires approval for significant actions, and ensure every decision is captured in an audit log.

What is the difference between an assistant and an agent in terms of liability?

An assistant merely drafts content for human review, whereas an agent is designed to execute actions (like payments or orders) autonomously within set limits.

Is prompt injection a major liability risk?

Yes, prompt injection is an unsolved security risk where hidden instructions can hijack an agent's behavior, making security measures like 'least privilege' essential.

Will AI agents eventually replace human responsibility?

No. While agents can run processes, humans remain responsible for judgment, trust, and owning the ultimate outcome of the work.