AI & Privacy: The Struggle is Real: meAnIngful consent and retAIning data

August 29, 2019
By Amanda Branch

A key concept of AI is that it is an intelligent system that is able to change its behaviour or adapt based on information or experience. AI is a powerful tool, and for AI to work meaningfully, it requires data – lots of data. Since data often contains personal information, there are real concerns about the privacy implications of AI, like what is required for personal information to be used and what privacy and security measures are required to protect that data. Data is essential to AI, and it is important to have complete data sets to try to manage concerns surrounding for example bias. Further, the output of AI is often unpredictable and how AI learns may not always be obvious or transparent. This creates tension with privacy laws. This article, the first in a series looking at AI and privacy issues, will discuss retention of data and the challenge of obtaining appropriate consent from users. Our next articles will discuss some of the approaches that companies are taking to manage these issues and the ethical frameworks that are beginning to emerge.

PIPEDA – A brief overview

Privacy law in Canada is governed by a regulatory framework and the common law. This article will focus on the Personal Information Protection and Electronic Documents Act, (“PIPEDA”) which establishes the basic rules governing how private-sector organizations must collect, use or disclose personal information in the course of their commercial activities in Canada.

As a quick refresher:

  • Personal information is defined as “information about an identifiable individual” and is given a broad and expansive definition.
  • It can include things like name, contact information, health information, financial information, biometrics and tracking information. Information does not need to directly identify an individual to be “about” an individual; it only needs to permit or lead to the possible identification of the individual.
  • Organizations may collect, use and disclose personal information only to the extent it is required to fulfill an explicitly specified reasonable purpose. Reasonableness is an overarching standard in PIPEDA that applies even if the individual has consented to the collection, use or disclosure of their personal information.


To retain or not to retain, that is the question.

AI may depend on large volumes of data to learn, however, this can be at odds with privacy legislation.

Under PIPEDA, the collection of personal information must be limited to that which is needed for the purposes identified by the organization. Personal information may be retained only so long as necessary to fulfill the reasonably stated purpose for which it was initially collected. There is no hard rule on how long an appropriate retention period is; however, the OPC has stated that indefinite retention is generally not appropriate.

The limitation principle can lead to tension with the necessity for AI to use all of the available information to learn. By limiting certain datasets, AI may have bias introduced because it is only being trained on subsets of the dataset, instead of all the available data.

Retaining data for long periods of time may also result in having a larger data pool. In these circumstances, there may be a risk that data that has been de-identified in isolation may be capable of re-identification when arranged and analyzed as part of a larger data set. It is also possible that AI can recreate identities, or minimally, can recreate portions of identities that were originally removed to protect against discrimination. There are examples where gender and/or name were removed from resumes in order to help protect against discrimination, however, the AI tool was able to pick up subtle nuances in language use that allowed it to recreate and determine the candidate’s gender.  

How do you request consent for something you can’t explain?

Consent is an important element of privacy law. Organizations must be open and transparent about their privacy practices. Consent should be obtained at the time of collection, as well as for new uses of personal information.

Consent is considered valid only if it is “meaningful” – that is, if it is reasonable to expect that individuals to whom a business’ activities are directed would understand the nature, purpose and consequences of the collection, use or disclosure to which they are consenting.

One of the major challenges organizations face is how to communicate their privacy practices to users. Studies have demonstrated that many people simply do not read privacy policies. This has fueled the open question of whether privacy policies are truly collecting meaningful consent. The Office of the Privacy Commissioner of Canada released new guidance that has been applied since January 1, 2019. The Guidelines for obtaining meaningful consent are intended to provide practical and actionable guidance for organizations to obtain meaningful consent under PIPEDA, and set out the seven guiding principles. Read more on the Guidelines here.

AI presents particular challenges from a consent perspective. For example, a goal of machine learning can be to create a system that can teach itself, but how can you obtain consent for something that a machine created with limited human intervention?

It may be important to ensure there is sufficient human oversight at each step of the process in order to be able to provide meaningful information about the logic involved and the characteristics considered in reaching a particular decision. Transparency and accountability for personal information are essential privacy principles.  On the other hand, it may be that human oversight creates other privacy issues since the anonymity and aggregation that occurs by AI may be risked with human intervention.  In any event, organizations should incorporate and consider privacy principles at each step in the design and implementation process (known as Privacy By Design). Notification to consumers should be done in a user-friendly way, such as using clear language, layered policies and just-in-time notices at the point where particularly sensitive data is being collected.

Subscribe to our newsletter

You can unsubscribe at any time. This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

This site is registered on as a development site.