On January 28, the Information and Privacy Commissioner of Ontario (OIPC) issued guidance on the responsible development, procurement, and use of AI scribes. The guidance reflects the growing adoption of AI transcription tools in the healthcare sector, focusing on upholding privacy, security, and human rights obligations in accordance with the Personal Health Information Protection Act, 2004 (PHIPA).

This publication coincides with similar guidance recently issued by the Office of the Information and Privacy Commissioner for British Columbia (BC-IPC), reflecting an emerging regulatory focus on AI accountability in healthcare. However, largely due to differences between privacy laws in Ontario and BC, the OIPC and BC-IPC take notably different approaches.

Ontario’s PHIPA applies broadly to all “health information custodians,” whereas in BC different legislation applies to private sector and public sector healthcare organizations. Accordingly, the OIPC guidance covers all custodians, whereas the BC guidance applies only to private healthcare organizations. The OIPC guidance also addresses the full lifecycle of AI scribes, outlining obligations for developers, procurers and users, while the BC-IPC guidance focuses on helping private healthcare providers using AI scribes comply with PIPA.

Background on AI scribes

AI scribes are tools that use generative artificial intelligence, speech recognition, and natural language processing to transcribe healthcare visits and generate clinical notes, summaries, and other related documentation. The use of AI scribe tools is expanding rapidly across Ontario, with the goal of reducing the administrative burden on healthcare practitioners. While early deployments show promising efficiency gains, AI scribes also introduce new data privacy, security, and clinical accuracy risks that custodians must manage.

Scope and principles of the new guidance

The guidance focuses on privacy-related considerations that arise from using AI scribes to transcribe and generate notes.

The guidance aligns with the six fundamental principles for the responsible use of AI previously set out by the OIPC jointly with the Ontario Human Rights Commission: Valid and reliable; safe; privacy protective; human rights affirming; transparent; and accountable.

Governance and accountability for AI scribes

Health information custodians are ultimately responsible for the personal health information in their custody and control, and their obligations under PHIPA continue to apply when they collect, use, and disclose personal health information using an AI system, like an AI scribe. Accordingly, the OIPC recommends custodians establish a strong governance and accountability framework to ensure continued compliance with PHIPA when AI scribes are used.

A governance and accountability framework should include the following:

  • AI governance committee. Establish an AI governance committee and risk management framework anchored in PHIPA accountability. This should be integrated into existing governance structures and processes.
  • Data minimization. Apply PHIPA’s data minimization principles throughout the AI scribe’s lifecycle. This can include scrutinizing whether it is necessary to retain full audio recordings or transcripts and limiting personal health information disclosure to vendors.
  • Privacy impact assessments. Conduct privacy impact assessments before introducing AI scribes and update them as purposes, systems, or risks change. Implement threat risk assessments and AI-specific assessments, and refresh these assessments, as needed.
  • Policies, procedures, and practices. Maintain comprehensive written policies, practices, and procedures, regularly reviewed to reflect legal changes, OIPC guidance, and technology updates. This includes end-user agreements and training for human oversight of AI outputs.
  • Transparency and inquiries. Prepare patient-facing transparency materials and establish a process for responding to inquiries from individuals regarding the AI system.
  • Accuracy and human oversight. Establish procedures for identifying inaccuracies or biases, and introduce adequately trained humans in the loop to oversee quality and performance of the AI system. This is important as the risks associated with common AI-generated inaccuracies, like hallucinations or transcription errors, are heightened in the healthcare context. Mistakes in clinical notes or other materials relied on by practitioners can reduce quality of care and negatively impact patient safety and wellbeing, enforcing the need for accurate AI systems.
  • Consent to AI scribe use. Ensure patients who withhold or withdraw consent to AI scribe use receive the same level of care as consenting patients. If information collected during AI scribe use contributes to AI model development, address the risk of bias introduced by excluding non-consenting patients’ data.

Developing and procuring AI scribes – considerations for custodians

Custodians may either develop or procure AI systems, including AI scribes. Depending on whether the tool is developed or procured, there are different risks for custodians to consider.

Custodians who develop AI scribes in-house should ensure the system is safe, trained on lawful data, produces accurate and explainable outputs, and includes bias mitigation and cybersecurity safeguards. Ongoing monitoring and AI-specific assessments will be essential.

Custodians procuring AI scribes from third-party vendors should conduct thorough due diligence and assess whether a vendor meets their expectations and agrees to obligations for intended use, lawful training data, model validation, monitoring, explainability, and incident reporting.

To meet their own PHIPA obligations, a custodian should negotiate strong contractual safeguards and verify a vendor’s commitments to performance and ongoing monitoring. This can include contractual limits on vendor’s access to and use of personal health information, data retention and destruction obligations, subcontractor controls, security and PHIPA-specific breach notification commitments, and ongoing performance monitoring, among other safeguards.

Looking ahead

As AI scribe technologies continue to evolve, custodians should expect increasing regulatory scrutiny and expectations for governance, transparency, and human oversight. Custodians will need to continuously monitor legal and regulatory developments and update internal frameworks accordingly. They will need to balance the need for innovation and improved efficiency without compromising adherence to PHIPA compliance and other human rights affirming safeguards.

The authors would like to thank Sulayman Syed, articling student, for his contribution to preparing this legal update.

Contact