AI agent governance in government and public sector: transparency, due process and sovereign AI

A municipal benefits agency deployed an AI agent to pre-screen disability benefit applications. The agent reviewed medical documentation, cross-referenced employment records and produced an eligibility recommendation for each application. A human caseworker received the recommendation and approved or denied the application. In 94% of cases, the caseworker followed the agent’s recommendation.

An applicant whose claim was denied requested an explanation. The caseworker explained the denial based on the factors the agent had identified. The applicant appealed, and the appeals board asked how the agent had weighed the medical evidence against the employment history. The caseworker could not explain the agent’s reasoning. The agent’s documentation recorded the inputs and the output but not the decision logic that connected them.

The appeals board overturned the denial. Not because the decision was wrong, but because the agency could not explain how it was made. In administrative law, a decision that cannot be explained is a decision that cannot be defended.

The governance environment#

Government AI agents operate in a governance environment that differs from commercial AI in three fundamental ways:

Public accountability. Government agencies serve citizens, not customers. Citizens cannot choose a competing provider. The power asymmetry between a government agency and a citizen means that governance must be stronger, not weaker, than in commercial contexts.

Administrative law. Decisions that affect individual rights, benefits or legal status must comply with procedural due process requirements. These requirements predate AI but apply to AI-made decisions. An automated decision must be explainable, documentable and appealable.

Transparency mandates. Government operations are subject to transparency obligations that do not apply to private entities. Public records laws, freedom of information requirements and specific AI transparency mandates require that government AI use is documented and publicly accessible.

The regulatory overlay#

EU AI Act: public sector as high-risk deployer#

The EU AI Act classifies virtually all public sector AI applications that directly affect citizen interests as high-risk. Annex III explicitly covers:

• AI systems used to evaluate eligibility for essential public assistance benefits and services
• AI systems used in law enforcement for individual risk assessment and crime analytics
• AI systems used in migration and border control for risk assessments, document verification and application processing
• AI systems used in the administration of justice for legal research, case analysis and judicial support
• AI systems used for biometric identification in publicly accessible spaces (with some prohibited uses)

High-risk obligations apply from August 2026 and include: risk management systems, data governance, technical documentation, transparency to users and affected persons, human oversight, accuracy standards and registration in the EU database.

Public authorities deploying high-risk AI face an additional obligation: fundamental rights impact assessments (Article 27) before deploying any high-risk system.

Administrative law and due process#

In the US, administrative law’s due process protections require that decisions affecting individual rights or entitlements be:

• Based on evidence: the decision must rest on documented facts and established criteria, so an AI agent’s recommendation must be traceable to specific evidence in the applicant’s record
• Explainable: the agency must be able to articulate why the decision was made, which for AI agents requires decision explanations that go beyond “the model said so”
• Appealable: individuals must have access to a meaningful appeal process, so if the original decision was made by an AI agent, the appeal must involve a human reviewer who can independently assess the merits
• Consistent: similar cases must be treated similarly, so AI agents must be monitored for disparate treatment across demographic groups, geographic regions or case types

In the EU, GDPR Article 22 provides the right not to be subject to a decision based solely on automated processing that produces legal effects or similarly significant effects. This right requires human involvement in consequential government decisions.

Federal procurement directives#

US federal procurement policy now includes specific AI governance requirements:

OMB M-25-21 (Accelerating Federal Use of AI) requires agencies to designate chief AI officers, maintain AI use case inventories, implement risk management frameworks and publicly report AI deployments annually.

OMB M-25-22 (Driving Efficient Acquisition of AI) establishes procurement standards for AI acquisitions, applying to contracts awarded after September 30, 2025.

GSA’s proposed GSAR 552.239-7001 introduces the most prescriptive requirements:

• Government owns all data inputs and outputs
• Contractors cannot use government data to train models for other customers
• 72-hour incident reporting with daily updates until resolution
• Prohibition on foreign AI systems in contract performance
• Prime contractor liability for all downstream subcontractor compliance

National AI strategies#

Individual nations add their own governance layers:

• Denmark launched an AI regulatory sandbox in 2021 to test AI compliance with GDPR, with specific focus on public sector applications
• Norway established a regulatory sandbox through its Data Protection Authority for developing ethical AI solutions, with requirements for transparency and accountability in public administration AI
• California (Executive Order N-5-26, March 2026) directs state agencies to develop AI certification requirements for vendors, including attestations on bias, civil rights and content exploitation

Governing government AI agents by type#

Citizen-facing agents#

Citizen-facing agents handle information requests, application processing, appointment scheduling and service navigation. They are the public face of government AI.

Transparency at every interaction. Citizens must know they are interacting with an AI agent, not a human. Disclosure must be clear, immediate and unmissable. The EU AI Act specifically requires that people be notified when they interact with an AI system.

Language and accessibility. Government agents must serve all citizens, including those with limited language proficiency, disabilities or limited digital literacy. Governance must verify that agents provide equitable service quality across population groups.

Scope limitation. Citizen-facing agents should provide information and navigation assistance. They should not make determinations about eligibility, rights or legal status. Any query that touches substantive government decision-making must route to a qualified human official.

Data protection. Citizens interacting with government agents may disclose sensitive personal information. Governance must ensure that this data is protected under applicable data protection laws (GDPR, Privacy Act), not retained beyond the necessary period and not used for purposes beyond the original interaction.

Decision-making agents#

Decision-making agents evaluate applications, assess eligibility, calculate benefits and produce recommendations or determinations. They present the highest governance stakes in government.

Explainability requirement. Every decision affecting an individual must be explainable in terms that the individual and a reviewing authority can understand. The agent must produce decision explanations that identify: which factors were considered, how they were weighted, what evidence supported the conclusion and why alternative outcomes were not selected.

Human-in-the-loop mandate. For consequential decisions (benefit determinations, enforcement actions, permit approvals/denials), a qualified human official must review the agent’s recommendation before the decision takes effect. The human reviewer must have sufficient information and authority to override the recommendation.

Bias monitoring. Government decision-making must be equitable across protected classes. Governance must include regular bias testing across demographic groups, geographic regions and case types. Disparate impact that exceeds statistical thresholds must trigger review and remediation.

Appeal infrastructure. The governance framework must include a clear appeal pathway for individuals affected by AI-assisted decisions. The appeal process must involve human review by officials who did not participate in the original decision and who have access to the agent’s full decision trail.

Procurement agents#

AI agents assisting in government procurement must maintain public accountability standards.

Conflict of interest detection. Procurement agents must be monitored for patterns that could indicate bias toward specific vendors, products or services. Governance must include regular audits of procurement agent recommendations against actual vendor performance.

Public accountability. Government procurement decisions are subject to protest and review. The procurement agent’s evaluation methodology must be defensible before a reviewing authority. This requires documentation of evaluation criteria, scoring methodology and the rationale for rankings.

Small business and socioeconomic compliance. Federal procurement agents must apply small business set-aside requirements, socioeconomic preferences and other statutory procurement obligations. Governance must verify that agents apply these requirements consistently.

Cross-agency agents#

Agents that operate across multiple government agencies or share data between agencies face additional governance challenges.

Data sharing agreements. Cross-agency data sharing must comply with applicable legal authorities. An agent that accesses data from multiple agencies must operate under data sharing agreements that authorize each access point.

Jurisdictional boundaries. Agents operating across jurisdictions must comply with the most restrictive applicable requirements. A federal agent accessing state data must comply with both federal and state governance requirements.

Interoperability standards. Cross-agency agents must use standardized data formats and communication protocols. Governance must verify that interoperability does not create security vulnerabilities or data protection gaps.

Defense and intelligence agents#

Defense and intelligence agents operate under a separate governance tier with additional requirements for classification, operational security and mission assurance.

Classification governance. Agents handling classified information must operate within accredited environments and comply with applicable security frameworks.

Operational testing. Defense agents must undergo rigorous testing in operational scenarios, including adversarial conditions, degraded environments and edge cases that could affect mission outcomes.

Human control over lethal decisions. International humanitarian law and Department of Defense policy require meaningful human control over decisions involving the use of force. AI agents must not make or execute lethal decisions without appropriate human authorization.

The government AI governance maturity model#

Level 1: ad hoc#

No formal AI governance framework. Individual agencies or departments experiment with AI agents independently. No standardized risk assessment, no central inventory, no transparency reporting. Most government organizations started here.

Level 2: documented#

Formal AI policies exist. Agencies maintain AI use case inventories. Risk assessment procedures are defined but applied inconsistently. Transparency reporting occurs but covers deployment, not operational behavior.

Level 3: managed#

Centralized governance oversight through a chief AI officer or equivalent. Standardized risk classification applied across agencies. Pre-deployment review required for high-risk agents. Monitoring exists but may not be continuous. The 8 pillars of AI agent governance provide a structure for this level.

Level 4: measured#

Continuous monitoring of agent behavior with automated drift detection. Governance metrics tracked and reported. Bias testing performed regularly with documented results. Audit evidence generated as a byproduct of operations. Agent registries maintain real-time inventories across all agencies.

Level 5: optimized#

Governance informs agent design from inception. Citizen feedback mechanisms built into governance cycles. Cross-agency governance coordination through shared platforms. Governance maturity measured and benchmarked against international standards. Observability infrastructure provides continuous evidence for transparency reporting and regulatory compliance.

Sovereign AI considerations#

Sovereign AI has emerged as a governance priority for governments concerned about dependency on foreign AI infrastructure.

Domestic model requirements. Some jurisdictions mandate or prefer AI models developed domestically. GSA’s proposed clause prohibits foreign AI systems in federal contracts. The European Commission’s sovereignty discussions focus on reducing dependency on non-EU AI providers.

Data residency. Citizen data processed by government AI agents must remain within jurisdictional boundaries. For cloud-deployed agents, this requires data residency controls enforced at the infrastructure level, not just contractually.

Training data transparency. Government procurement increasingly requires vendors to disclose training data sources, methodology and composition. This enables assessment of potential biases and verification that training data does not include materials that violate data protection or intellectual property laws.

Audit access. Sovereign AI governance requires that government auditors can inspect AI systems, including model architecture, training data and decision processes. Black-box AI systems that resist inspection may not meet government procurement requirements.

Building public trust#

Government AI governance has a purpose beyond compliance: maintaining public trust in democratic institutions. Citizens who believe that government AI makes decisions they cannot understand, challenge or influence will lose trust in the agencies that deploy it.

The governance framework that maintains trust is one that:

• Makes AI use visible through public inventories and transparency reports
• Ensures that AI-assisted decisions can be explained in plain language
• Provides meaningful appeal pathways for individuals affected by AI decisions
• Monitors for bias and takes corrective action when disparities are found
• Includes citizen input in governance design and evaluation

The regulatory requirements, EU AI Act, administrative law, procurement directives, are the floor. Governments that build governance above that floor, governance that proactively earns citizen trust rather than merely complying with minimum requirements, will define the standard for responsible government AI adoption.