Artificial intelligence (AI) has swiftly transitioned from pilot programs into core clinical and administrative workflows across US hospitals, reshaping how care is delivered, documented and managed. The pace of adoption is outstripping the pace of oversight and federal regulators, state legislatures and enforcement agencies are all responding at once. AI is advancing rapidly. For health system boards, keeping pace with these developments requires more than awareness. Governance and oversight must evolve alongside the technology itself. The critical question is whether your board is actively governing AI or just managing the fallout after the fact.
The regulatory landscape: What health system leaders need to know
Health system boards must be aware of significant and accelerating activity across federal agencies, state legislatures and enforcement bodies, all moving on parallel tracks and collectively raising the bar for AI governance in healthcare.
At the federal level, the White House has directed agencies to accelerate AI adoption while building governance guardrails. The US Department of Health and Human Services is aligning its operating divisions around common standards while creating channels for providers to participate in shaping implementation. The Centers for Medicare & Medicaid Services (CMS) has already incorporated AI into prior authorization, program integrity, payment models and analytics, while the Office for Civil Rights (OCR) has made clear that automated systems do not diminish privacy or civil rights obligations and that discriminatory effects in federally funded programs must be prevented.
States are moving with equal urgency to advance both comprehensive and sector-specific AI statutes, including laws that restrict or condition automated decision-making in utilization management. These measures, in some cases, go further than existing federal requirements and, in others, may conflict with them, requiring careful preemption analysis and coordinated implementation.
On the enforcement front, the Department of Justice and US Attorneys' Offices are expanding their use of advanced analytics, including AI-enabled techniques, to identify billing anomalies, kickback schemes and patterns associated with fraud and abuse. This posture raises expectations for corporate integrity and is likely to influence monitoring standards and documentation requirements in future corporate integrity agreements.
The current administration has made clear a preference that federal regulation should trump state regulatory schemes.
Fiduciary duties in the age of AI
AI does not create new fiduciary duties for health system board members. However, it raises the stakes for how existing ones are fulfilled.
Duty of care: Stay informed, ask hard questions
Board members are expected to act with the care and diligence a reasonably prudent person would use when making decisions of significance. In the AI context, that means boards cannot treat AI as purely a management or technology concern. When an organization deploys AI tools that affect patient care, billing or operations, board members are expected to understand the material risks involved and to ask informed questions about how those risks are being managed. This does not require board members to become AI experts; it does require insisting on clear, meaningful reporting and following up when answers are incomplete.
The Caremark doctrine establishes that directors can face personal liability for a sustained or systematic failure to implement and monitor adequate oversight systems. In re Caremark Int'l Inc. Derivative Litigation, 698 A.2d 959 (Del. Ch. 1996). Courts have increasingly applied this standard to technology-related failures and AI is a natural next step. Delaware courts have shown a greater willingness to allow Caremark claims to survive dismissal where boards allegedly ignored red flags. In re The Boeing Co. Derivative Litig., 2021 WL 4059934 (Del. Ch. 2021).
Duty of loyalty: Watch for conflicts
AI procurement is becoming a significant organizational investment and the vendor landscape is complex and evolving. Directors with financial ties to AI vendors or whose affiliated entities stand to benefit, face conflicts of interest that must be disclosed and managed. In In re Walt Disney Co. Derivative Litig. 907 A.2d 693 (Del. Ch. 2005), aff’d, 906 A.2d 27 (Del. 2006), the court signaled that conscious disregard of known risks can implicate the duty of loyalty. As AI becomes central to operations, boards should ensure that conflict-of-interest policies are up to date and consistently applied.
Duty of obedience: Keep AI aligned with your mission
For nonprofit health systems, the duty of obedience requires ensuring the organization remains true to its charitable mission. AI deployments that introduce disparate impacts on vulnerable populations, compromise patient privacy or systematically disadvantage certain groups could jeopardize mission commitments, tax-exempt status and compliance with federal civil rights laws. As AI moves to the core of health system operations, including data monetization, algorithmic resource allocation and automated care denials, regulators and communities are likely to heighten scrutiny.
A practical note on liability
The business judgment rule generally protects directors who make informed, good-faith decisions that they reasonably believe are in the organization’s best interest. The risk lies not in imperfect decisions but in uninformed ones. Robust minutes, regular AI-risk reporting and documented committee oversight are the building blocks of a credible defense.
Questions every health system board should be asking today
Accountability: Who is responsible for AI decisions across the organization?
AI adoption is outpacing the development of governance frameworks in healthcare, creating an evolving oversight challenge. Boards should be satisfied that management has assigned clear ownership of the full AI lifecycle, including authorization, performance monitoring and decommissioning authority. Without that clarity, AI can quietly enter workflows, risking patient safety, privacy and compliance.
This is not a theoretical concern. A recent survey found that 40 percent of healthcare respondents had encountered unsanctioned AI tools in the workplace and nearly 20 percent reported using them personally. Of those using unauthorized tools, almost half did so to speed up workflows and one in three cited either a lack of approved tools or approved tools that lacked the functionality they needed.
The same survey revealed a significant gap in how AI policies are created and communicated: nearly a third of administrators reported involvement in developing AI policies compared to only 9 percent of providers and 21 percent of providers were unclear whether policies had been communicated at all, compared to 11 percent of administrators.
For boards, the takeaway is straightforward: governance only works if it reaches the people it is meant to govern. That means providing secure, authorized tools, communicating policies where staff work, including within the electronic health record and investing in ongoing training that keeps pace with constantly evolving technology.
Transparency: What AI tools are in use and where?
AI functions are embedded across electronic health records, billing systems and vendor applications, often invisibly. Boards should require management to maintain an up-to-date inventory of AI, identify high-risk systems and track vendor AI integrations.
Recent litigation highlights the stakes. In Estate of Lokken v. UnitedHealth Group, Inc., 766 F.Supp.3d 835 (D. Minn. 2025), the court dismissed several tort claims but allowed breach-of-contract and related theories to proceed regarding alleged algorithmic denials of post-acute-care benefits. Notably, the District Court recently issued an order granting the plaintiffs significant discovery into United Healthcare’s AI tool at issue in the case. This case underscores the need for transparency into vendor AI logic, data use, model updates and human-in-the-loop safeguards.
Control: Are there guardrails in place before new AI tools go live?
Bringing a new AI tool into a clinical or operational environment carries legal, safety and compliance implications that require structured pre-deployment review. That review is no longer optional.
OCR's 2024 final rule implementing Section 1557 of the Affordable Care Act prohibits covered entities from discriminating through patient care decision support tools and imposes affirmative duties to identify such tools that rely on protected-class variables and to mitigate discrimination risk posed by each. 45 CFR part 92 (Nondiscrimination in Health Programs and Activities).
For Medicare Advantage organizations, CMS regulations require that any AI-driven adverse coverage determination be reviewed by a licensed physician or other qualified healthcare professional with appropriate expertise before it is issued. A binding “human-in-the-loop” requirement that cannot be satisfied by an algorithm alone. 42 CFR § 422.122.
At the state level, Colorado’s AI Act (SB 24-205) imposes formal impact assessments and risk management programs for high-risk AI systems, with enforcement scheduled for June 30, 2026. California's SB 1120, effective Jan. 1, 2025, prohibits AI tools from supplanting licensed clinician judgment in utilization management decisions.
Legal liability in this space cannot be outsourced. A federal court recently allowed a lawsuit to proceed against a company that relied on a vendor's AI tool for consequential decisions, finding the deploying organization could be held responsible for the allegedly discriminatory outputs that the tool produced. The message for boards is direct: choosing a vendor does not transfer your organization's legal accountability for what that vendor's tool does inside your workflows.
Oversight: Are we getting the right information at the right time?
Board oversight of AI requires more than periodic updates; it requires a functioning, documented monitoring system that surfaces meaningful risks before they become serious problems. Under Caremark and its progeny, boards that fail to implement mission-critical risk monitoring face personal liability exposure. Health system boards should demand regular, structured reporting on AI tool performance, incidents and corrective actions, supplemented by independent verification from Internal Audit and Compliance that AI policies are being followed in practice, not just on paper.
Conclusion
The next era of AI in healthcare will reward organizations that pair innovation with disciplined governance. Health system boards that insist on clear reporting, tested controls and continuous improvement will position their institutions to capture AI's benefits while protecting patient safety, privacy, civil rights compliance and corporate integrity. The time is to act now.
The Norton Rose Fulbright team continues to monitor evolving federal and state requirements, enforcement priorities and industry practices to keep governance structures current and defensible. Please do not hesitate to contact us with any questions