Loading...
Research & Insights

When AI Agents Act in Public Services, Who Is Accountable?

November 16, 2026 · Jason Ellis

South Dade Justice Center and Government Building facade in Miami, Florida showcases modern architecture.
Photo by DΛVΞ GΛRCIΛ on Pexels

Something is changing in how governments use AI, and the change is easy to miss because it sounds like a small technical step. The systems governments bought in 2023 and 2024 mostly answered questions and summarized documents. The systems now being proposed and piloted can do more: look up records, fill in forms, call other software, and set actions in motion across connected systems. Standards bodies call these "AI agents," and the distinction matters. A chatbot that gives a wrong answer wastes your time. An agent that takes a wrong action can change a record, flag a claim as fraudulent, or trigger a process that someone then has to notice and unwind.

The US Government Accountability Office puts the shift bluntly: deploying an AI agent in the public sector is better understood as delegated authority, not just software automation, because agents can invoke tools, alter records, and trigger actions across connected systems (GAO). Delegated authority is a legal idea, and it raises a legal question. If an agent acts on behalf of a government office, and the action harms someone, who explains it, who corrects it, and who is accountable?

That question does not yet have a settled answer. What exists instead is a fast-assembling set of rules and technical safeguards, some enforceable and some aspirational, and a growing body of evidence about whether they actually hold. This article walks through what is really deployed today, how the safeguards are supposed to work, what independent testing shows about their failure modes, and what the evidence says should be required before an agent is allowed to act for people and businesses.

What governments are actually letting agents do

The honest starting point is that fully autonomous agents making consequential public decisions are rare. GAO found that reported AI use cases at US federal agencies nearly doubled, from 571 in 2023 to 1,110 in 2024, but most remained assistive, experimental, or internal rather than autonomous public-service agents (GAO-25-107653). The clearest deployments in 2024 and 2025 cluster around document intake, records search, consultation analysis, casework support, and resident navigation, not independent benefit adjudication or payment authorization (GAO-25-107653).

The UK's "Humphrey" program illustrates the current frontier. Its "Consult" tool analyzed more than 2,000 responses to a Scottish Government consultation, grouping themes and ranking submissions for officials, while expert reviewers kept responsibility for interpretation (GOV.UK). A later deployment at Defra processed over 50,000 water-policy consultation responses, with 83% reviewer agreement and no changes to mapped themes, cutting estimated analysis time from 2,440 hours to roughly 22 (Government Analysis Function). Humphrey's "Minute" tool was trialed by 25 councils for meeting transcription and record-keeping, with early tests saving about an hour of administrative work per meeting hour (GOV.UK).

In the United States, Indiana's Secretary of State converted more than 20 million historical records into searchable form and deployed bilingual agents, Liz and Otto, for resident assistance and business-service interactions (Google Cloud case study).

Notice the pattern. These systems read, sort, summarize, and answer. They mostly do not yet approve, deny, or pay. But the pressure to extend them is real, because the same capabilities that sort 50,000 consultation responses could, in principle, sort benefit applications, and the step from "recommend" to "decide" can be little more than a configuration change rather than a new technology. The safeguards question has to be answered before that step, not after.

How does an agent get the authority to act?

Behind the accountability question sits a mechanical one: when an agent calls a tool, what proves it is allowed to?

The answer emerging from the US National Institute of Standards and Technology is that an agent needs its own identity, not a borrowed human account. Authority should be explicit, task-bound, and verifiable, with the agent's identity linked to the human or organization that originated the request (NIST). NIST proposes that authorization follow the full delegation chain: the originating user, the organization, the specific task, and every handoff in between. Permissions should follow least privilege, separation of duties, task-bounded access, short-lived credentials, and continuous authorization checks at each tool call, not a single login that unlocks everything.

One NIST idea deserves plain-language explanation because it does a lot of work: the cryptographically signed mandate. This is a machine-readable, tamper-evident statement of what the agent was asked to achieve and under what constraints. Its purpose is to give the system's policy engine something to check against, so it can distinguish actions the human actually authorized from instructions that were injected into the agent from somewhere else (NIST). If an agent processing a benefits file suddenly tries to email records to an outside address, the mandate shows that was never the task, and the policy engine can refuse the request before it executes.

Microsoft's guidance for its own agent identity system makes the same argument from the vendor side: least privilege should limit each agent's identity, data scope, tools, operations, and delegation rights, rather than letting agents inherit generalized credentials. And the logging has to capture more than the conversation: the agent, the human principal, the tool, the inputs, the outputs, the policy version, the authorization decision, the approval status, and any downstream handoffs (Microsoft Learn). NIST adds that the logs must be tamper-resistant, stored independently, and inaccessible for alteration by the agent itself, with monitoring systems cross-checking what the agent reports against what the systems actually recorded (NIST).

A woman reads a book in a card catalog room, surrounded by archival records.
Photo by Tima Miroshnichenko on Pexels

Why such suspicion of the agent's own account of events? Because of what independent testing has found.

What happens when someone hides instructions in the data

The defining security problem for agents is prompt injection: an attacker places instructions inside content the agent will process as data, such as a webpage, an email, a document, or the output of another tool. The agent cannot reliably tell the difference between the data it was asked to read and a command embedded in that data.

This is not a theoretical concern. The INJECAGENT benchmark, published in peer-reviewed computational linguistics research, treats an attack as successful when external content causes a tool-integrated model to execute a malicious command, and demonstrates that prompt injection is an agent-control problem, not merely a chatbot content-moderation issue (ACL Findings 2024). A large public red-teaming competition tested 22 frontier agents across 44 realistic scenarios, collecting 1.8 million attack attempts and more than 60,000 successful policy violations, including unauthorized data access, illicit financial actions, and regulatory noncompliance (competition report). OpenAI's own testing of its browser agent found that untrusted instructions can arrive through attachments, calendar invitations, shared documents, forums, social media, and webpages, and that attacks can steer long-horizon workflows, not just single responses (OpenAI).

The public-sector implications are specific. Research on indirect prompt injection in municipal document-processing copilots found the failure mode especially relevant to fraud screening, procurement review, benefits administration, and investigative workflows, because an altered document can influence a decision without visibly changing anything the official asked for (SAUC). Related work on automated fact-checking shows adversaries can manipulate retrieved evidence, potentially causing systems to accept misleading claims or discount corrective evidence (EMNLP 2025). Imagine a benefits applicant, or someone targeting an applicant, embedding hidden text in a submitted PDF. If the agent reading that PDF has the power to flag, escalate, or suspend, the hidden text is effectively an unauthorized command to a government system.

Canada's Cyber Centre draws the operational conclusion: it is not just the agent's inputs that need validation but its tool responses too, because malicious or unsafe instructions can enter through external services, plugins, retrieved content, or other agents (Canadian Centre for Cyber Security). The UK security standard for AI systems says the same thing in mandatory language: agent outputs and tool parameters must be treated as untrusted until validated through schemas, sanitization, and policy controls, with human approval where appropriate, and execution must happen in controlled environments restricted to approved networks, data, and tools, with rapid suspension, isolation, credential revocation, and rollback capabilities (UK Security Standard SS-040).

Where this story sits

5 related pieces

When AI Agents Act in Public Services, Who Is Accountable? and 5 related articlesNo. 105You are hereNo. 66: When AI Agents Escape the Sandbox: Inside the July 2026 Hugging Face Breach66When AI Agents Escapethe Sandbox: Inside the…No. 23: Inside India's Plan to Let AI Agents Spend Money on UPI23Inside India's Plan toLet AI Agents Spend…No. 49: Replicator Promised Thousands of Autonomous Systems. The Public Count Stands at Hundreds49Replicator PromisedThousands of Autonomous…No. 64: The ARTEX AI bank hack claim: what South Korea's record actually shows64The ARTEX AI bank hackclaim: what South…No. 42: After the Last Dollar: What Actually Catches People at Zero42After the Last Dollar:What Actually Catches…

Hover a story · click to read

  1. No. 66 · Oct 12, 2026When AI Agents Escape the Sandbox: Inside the July 2026 Hugging Face Breach
  2. No. 23 · Sep 1, 2026Inside India's Plan to Let AI Agents Spend Money on UPI
  3. No. 49 · Sep 15, 2026Replicator Promised Thousands of Autonomous Systems. The Public Count Stands at Hundreds
  4. No. 64 · Oct 11, 2026The ARTEX AI bank hack claim: what South Korea's record actually shows
  5. No. 42 · Sep 7, 2026After the Last Dollar: What Actually Catches People at Zero
Drawn from the blog's own index. See every story

Which laws already apply, and where they fall short

Close-up of a business professional reviewing an application form at a desk.
Photo by Kampus Production on Pexels

The legal picture is more developed than many assume, but it was mostly written for systems that decide, not systems that act, and the gaps show.

The EU AI Act (Regulation 2024/1689) regulates agents by risk category rather than creating a distinct "agent" class. Many systems used in essential services, law enforcement, migration, education, employment, and justice fall into the high-risk regime, which obliges providers and deployers to maintain risk management, data governance, technical documentation, logging, human oversight, accuracy, robustness, and cybersecurity, with registration, monitoring, and incident duties for public authorities (European Commission). The Act also prohibits certain practices outright, including some social scoring and manipulative techniques. But its effectiveness for autonomous agents depends on an unresolved classification question: how do you classify a system whose objectives, tools, and outputs change continuously (EUR-Lex)? The Commission's guidance adds that public-service deployments should undergo pre-deployment risk and fundamental-rights assessments covering discrimination, privacy, security, accessibility, legality, and impacts on affected groups (European Commission).

Data protection law supplies a second layer. GDPR Article 22 restricts decisions based solely on automated processing that produce legal or similarly significant effects, requiring exceptions, safeguards, and avenues for human intervention. The difficulty for agents is the word "solely": when a human supervises an agent loosely, is the decision solely automated or not? Interpretation is genuinely hard (EUR-Lex). The UK retains UK GDPR and the Data Protection Act 2018, but its 2025 Data (Use and Access) Act created a more permissive automated-decision framework with specified safeguards, particularly for adverse significant decisions and sensitive data, which means the UK and EU are now diverging on exactly the question agents make hardest.

In the United States, the Office of Management and Budget's memorandum M-25-21 requires pre-deployment testing, impact assessments, monitoring, and public reporting for high-impact federal AI uses, and requires agencies to discontinue use when minimum practices are not met (OMB M-25-21). And a principle that predates all of this still carries the most weight: existing public-law duties do not disappear when agencies use vendors, algorithms, or autonomous agents. Agencies remain responsible for benefit, privacy, civil-rights, due-process, and confidentiality requirements regardless of what software sits in the middle (HHS). As NYU's Development Research Institute puts it, an AI agent has no independent legal authority of its own; its actions must stay within powers delegated to accountable public officials and authorized by law (NYU DRI).

The gap, then, is less about missing law than about uneven enforcement machinery. The OECD's 2026 Digital Government Outlook finds that government AI safeguards are shifting from ethical principles toward enforceable lifecycle controls, but implementation is patchy: only 14 of 36 OECD countries require pre-deployment risk assessments, 11 conduct post-deployment audits, and 6 maintain open algorithm registers (OECD). A rule that is never audited is a hope, not a safeguard.

The safeguards that actually limit an agent

Close-up of a smartphone displaying a fraud alert notification on a wooden surface.
Photo by RDNE Stock project on Pexels

Pull the technical guidance together across NIST, the UK's National Cyber Security Centre, Canada's Cyber Centre, and the UK security standard, and a reasonably consistent control set emerges:

  • A distinct agent identity, tied to the human or organization that originated the task, with authorization checked at every tool call rather than once at login (NIST).
  • Least privilege and task-bounded access, so the agent can touch only the data, tools, and operations the specific task requires, using short-lived credentials (NIST; Microsoft Learn).
  • Allowlists and server-side policy enforcement, which validate both the requested operation and its arguments before anything executes, so a compromised agent cannot simply ask for more (NIST NCCoE).
  • Treating agent outputs as untrusted, with schema validation, sanitization, and policy checks between the agent and any system of record (UK SS-040).
  • Sandboxed execution in controlled environments, with the ability to rapidly suspend the agent, isolate it, revoke its credentials, and roll back its actions (UK SS-040).
  • Independent, tamper-evident logs recording identity, intent, delegations, decisions, tool calls, outputs, and results, stored where the agent cannot alter them (NIST).
  • Human approval gates placed where they matter. Canada's guidance is usefully precise here: human handoff is most important for high-impact, irreversible, legally sensitive, or hard-to-reverse actions, not for every routine, low-risk step (Canadian Centre for Cyber Security). A human who must click "approve" four hundred times a day is not oversight; it is a rubber stamp with extra steps.

Governance controls wrap around the technical ones. The UK's NCSC says organisations need documented ownership, named accountable officials, defined escalation routes, and an empowered person who can stop the agent before it is deployed at all (NCSC), and that deployment should be incremental, starting with repetitive, well-understood, low-risk tasks and expanding only after testing and monitoring demonstrate safety (NCSC).

How do you test whether any of this works?

The uncomfortable lesson from the red-team evidence is that safeguards must be tested adversarially, under realistic conditions, before deployment, because the attacks succeed against polished frontier systems, not just prototypes. Australia's National AI Centre makes this explicit: pre-deployment testing must reflect real operating conditions and cover adversarial inputs, prompt injection, privacy leakage, unauthorized actions, excessive agency, and cyber-enabled harms, rather than relying on vendor assurances (National AI Centre).

Close-up of a parking meter with digital interface in a city street.
Photo by Egor Komarov on Pexels

Three kinds of evidence should exist before an agent touches a public service, and should be refreshed whenever the system materially changes:

  1. Adversarial test results against the known attack classes: injected instructions in documents and web content, poisoned tool responses, manipulated retrieved evidence, and conflicting instructions. The benchmarks exist; INJECAGENT and the public competition dataset show what realistic testing looks like (ACL Findings 2024).
  2. Accuracy evidence against human judgment in the actual domain. The Defra consultation deployment is a model of the genre: a measured 83% reviewer-agreement rate, checked against expert reviewers, published as a case study (Government Analysis Function).
  3. Operational drill evidence that the kill switch, credential revocation, and rollback actually work, because a shutdown capability that has never been exercised is an assumption.

The OECD figures suggest most governments do not yet require even the first category. That is the gap between the guidance and the ground.

When an agent gets it wrong, who fixes it?

Here is where the stakes become personal. In benefits programs, an "improper payment" includes underpayments as well as overpayments, so an agent's errors can harm people through wrongful recovery demands as well as through missed or reduced entitlements (GAO-25-108412). The same systems that can detect anomalous patterns at a scale impractical for staff can also hallucinate facts, misclassify applicants, work from defective data, automate fraud accusations, or trigger payment suspensions without understanding statutory exceptions or individual circumstances (NYU DRI).

The remedy framework that has emerged from US benefits-administration guidance is the most concrete available. Human oversight must be substantive rather than ceremonial: trained reviewers must be able to inspect the evidence, reject outputs, pause operations, and reverse erroneous actions. For high-impact decisions such as reducing, terminating, or denying benefits, systems should preserve staff discretion, provide notice, retain durable decision logs, and allow review of the logic and evidence used (USDA framework). Affected people need accessible explanations, non-digital channels, timely human reconsideration, and, critically, appeals that are not decided solely by the same automated process that produced the disputed result (House committee record). UK public-sector guidance points the same way: where agents influence legally or similarly significant decisions, people must receive accessible information about the automation and a straightforward route to human intervention, review, and correction (GOV.UK guide).

Two call center agents working together with headsets in a modern office environment.
Photo by Kampus Production on Pexels

So the answer to the headline question is clearer than the technology suggests. Accountability does not transfer to the agent, because the agent has none to receive. It stays with the named officials who authorized the deployment, the agency that owns the legal duty, and the vendor under its contract. The real risk is not that accountability disappears in law but that it becomes practically unusable: decisions nobody can explain, logs nobody kept, and an appeals process that routes the complaint back into the system that made the error.

The cost of being careful

Every safeguard above has a price, and pretending otherwise would be dishonest. Approval gates slow services. Sandboxing and separate agent identities cost engineering effort. Mandatory human review can recreate exactly the backlog the agent was bought to clear. OMB names this trade-off directly: rigid safeguards can delay critical operations or reduce accessibility, which is why it permits documented, publicly summarized waivers when compliance would create greater overall risk or an unacceptable operational impediment. The same memo supplies the discipline that stops the exception from swallowing the rule: waivers must be justified, centrally tracked, reassessed after material changes, and annually certified by accountable officials (OMB M-25-21).

That is the right shape for the whole problem. The question is never "safeguards or speed" in the abstract. It is whether each deviation from the baseline is documented, visible, owned by a named person, and revisited when the system changes.

What should be required before an agent acts for the public

The evidence across standards bodies, regulators, auditors, and security researchers converges on a minimum bar, and it is worth stating plainly:

  • A named accountable official and an empowered person who can stop the agent, documented before deployment, not after an incident (NCSC).
  • A distinct agent identity with least-privilege, task-bound permissions and policy checks at every tool call (NIST).
  • Published pre-deployment risk and rights assessments covering discrimination, privacy, security, and accessibility, as the EU framework requires for public-service uses (European Commission).
  • Adversarial testing against injection and misuse under realistic conditions, with results that would survive independent scrutiny (National AI Centre).
  • Tamper-evident logs the agent cannot alter, retained long enough to support appeals (NIST).
  • Human approval for high-impact, irreversible, or legally sensitive actions, and human appeal routes independent of the automated process (Canadian Centre for Cyber Security; House committee record).
  • Post-deployment monitoring, audits, and public reporting, with discontinuation when minimum practices fail (OMB M-25-21).

The encouraging finding is that today's real deployments, consultation analysis, meeting minutes, record search, resident Q&A, mostly sit on the safe side of this line, with humans keeping the consequential judgments. The sobering finding is that the controls that would make riskier delegation safe are specified in detail by standards bodies but required by only a minority of governments, while independent testing shows that unprotected agents fail against exactly the attacks public services would face. The technology to let agents act is here. The evidence says the permission to act should still be earned, one documented, tested, reversible step at a time.

Sources / References

Close-up of a professional audio and video editing software interface with waveform displays.
Photo by Pixabay on Pexels
  • National Cyber Security Centre (UK), "Thinking carefully before adopting agentic AI," blog and guidance PDF: https://www.ncsc.gov.uk/blogs/thinking-carefully-before-adopting-agentic-ai and https://www.ncsc.gov.uk/sites/default/files/2026-06/Thinking-carefully-before-adopting-agentic-AI.pdf
  • NIST, "Design Tenets for Agent Identity Management Systems": https://www.nist.gov/blogs/cybersecurity-insights/design-tenets-agent-identity-management-systems
  • NIST NCCoE, Agentic AI Identity and Authorization Project, summary of comments: https://pages.nist.gov/nccoe-ai-identity/summary-of-comments.html
  • Microsoft Learn, "Least privilege for AI agents with Microsoft Entra Agent ID": https://learn.microsoft.com/en-us/security/zero-trust/sfi/least-privilege-for-ai-agents
  • UK Government Security Standard SS-040, Artificial Intelligence and Machine Learning: https://assets.publishing.service.gov.uk/media/6af5d060cc0dbfec767a50bd/standard-artificial-intelligence-and-machine-learning-version-1.pdf
  • UK Government, "A guide to using AI in the public sector": https://assets.publishing.service.gov.uk/media/60828077d3bf7f013e451bc8/A_guide_to_using_AI_in_the_public_sector__Print_version_.pdf
  • GAO, "Science & Tech Spotlight: AI Agents" (GAO-25-108519): https://www.gao.gov/products/gao-25-108519
  • GAO, "Artificial Intelligence: Generative AI Use and Management at Federal Agencies" (GAO-25-107653): https://www.gao.gov/assets/gao-25-107653.pdf and https://files.gao.gov/reports/GAO-25-107653/index.html
  • GAO, "Fraud and Improper Payments: Data Quality and a Skilled Workforce Are Essential for Unlocking the Benefits of Artificial Intelligence" (GAO-25-108412): https://files.gao.gov/reports/GAO-25-108412/index.html
  • GOV.UK, "Government-built 'Humphrey' AI tool reviews responses to consultation for first time": https://www.gov.uk/government/news/government-built-humphrey-ai-tool-reviews-responses-to-consultation-for-first-time-in-bid-to-save-millions
  • Government Analysis Function (UK), AI case studies: https://analysisfunction.civilservice.gov.uk/ai-hub/ai-case-studies/
  • GOV.UK, "AI experiments see 'Humphrey' help townhalls cut costs and improve services": https://www.gov.uk/government/news/ai-experiments-see-humphrey-help-townhalls-cut-costs-and-improve-services
  • Google Cloud, Indiana Secretary of State case study: https://cloud.google.com/customers/indiana-secretary-of-state
  • European Commission, "AI Act" regulatory framework overview: https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai
  • European Commission, "Navigating the AI Act" FAQ: https://digital-strategy.ec.europa.eu/en/faqs/navigating-ai-act
  • Regulation (EU) 2024/1689 (AI Act), consolidated text: https://eur-lex.europa.eu/eli/reg/2024/1689
  • Regulation (EU) 2016/679 (GDPR), Article 22: https://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1772981449498&uri=CELEX%3A32016R0679
  • OMB Memorandum M-25-21, "Accelerating Federal Use of AI through Innovation, Governance, and Public Trust": https://www.whitehouse.gov/wp-content/uploads/2025/02/M-25-21-Accelerating-Federal-Use-of-AI-through-Innovation-Governance-and-Public-Trust.pdf
  • HHS, "Public Benefits and AI": https://www.hhs.gov/sites/default/files/public-benefits-and-ai.pdf
  • USDA, "Framework for State, Local, Tribal, and Territorial Use of Artificial Intelligence for Public Benefit Administration": https://digitalgovernmenthub.org/wp-content/uploads/2024/05/2024-usda-ai-framework-benefits.pdf
  • US House Committee record on automated systems in public benefits: https://docs.house.gov/meetings/FD/FD00/20240206/116793/HHRG-118-FD00-20240206-SD001.pdf
  • NYU Development Research Institute, "Harnessing Agentic AI: The Public Sector Can Gain as Well": https://nyudri.org/publications/harnessing-agentic-ai-the-public-sector-can-gain-as-well/
  • OECD, "Adopting and governing AI in government," Digital Government Outlook 2026: https://www.oecd.org/en/publications/digital-government-outlook_0496b2bc-en/full-report/adopting-and-governing-ai-in-government_7ef312a9.html
  • Canadian Centre for Cyber Security, "Careful adoption of agentic AI": https://www.cyber.gc.ca/en/guidance/careful-adoption-agentic-ai
  • "INJECAGENT: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents," Findings of ACL 2024: https://aclanthology.org/2024.findings-acl.624.pdf
  • "Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition": https://kingsley.idehen.net/DAV/home/kidehen/Public/PDFs/ai-agent-red-tape-testing-for-injection-issues.pdf
  • "Attacks by Content: Automated Fact-checking is an AI Security Issue," EMNLP 2025: https://aclanthology.org/2025.emnlp-main.431.pdf
  • "Indirect Prompt Injection in Municipal Document-Processing Copilots," SAUC: https://visualcompublications.es/SAUC/article/view/6399
  • OpenAI, "Continuously hardening ChatGPT Atlas against prompt injection": https://openai.com/index/hardening-atlas-against-prompt-injection/
  • National AI Centre (Australia), "Guidance for AI adoption: implementation guidance": https://www.ai.gov.au/staying-safe-and-responsible/essential-ai-practices/guidance-ai-adoption-implementation-guidance
Share this article
Go deeper

The reading behind this research

What the desk leaned on for this piece. Each link goes to the retailer, where the price is shown.

Paid links. Calibre Code USA may earn a commission from purchases made through these links. This does not affect what we write or what we recommend.

Comments

No comments yet. Start the conversation below.

Comments are reviewed before they appear.

Continue exploring