With Risk Comes Responsibility: EU Proposes High-Risk AI Regulation Affecting Device Manufacturers

In April 2021, the European Commission proposed the EU Artificial Intelligence Act,¹Proposal for a Regulation of the European Parliament and of the Council: Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts, European Commission, 21 April 2021, https://eur-lex.europa.eu/resource.html?uri=cellar:e0649735-a372-11eb-9585-01aa75ed71a1.0001.02/DOC_1&format=PDF. a first-of-its-kind regulation by any global regulatory authority. The proposed Act is intended to establish harmonized rules on artificial intelligence (“AI”) in the European Union (“EU”).²Id.  In December 2022, following the first steps in the EU ordinary legislative procedure, the EU Council adopted a revised version of the AI Act proposal (the “Draft AI Act”). ^{3Proposal for a Regulation of the European Parliament and of the Council: Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts - General approach, Council of the European Union, 6 December 2022, [hereinafter AI Act].}  The Draft AI Act establishes the EU Council's provisional position on the proposal, and forms the basis for negotiations with the European Parliament to adopt the final text. Article 23a of the Draft AI Act may impose significant compliance obligations on manufacturers of medical devices that involve “high-risk” AI systems. ^{4Proposal for a Regulation of the European Parliament and of the Council: Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts - General approach, Council of the European Union, 6 December 2022, Article 23a.} Further, as a “first-of-its-kind” regulation, its adoption could have significant influence on other jurisdictions to follow.

In the EU, there is a noteworthy difference between the different types of legal acts such as a “Directive” or a “Regulation.”  An EU Directive sets binding objectives to be achieved by the EU Member States, it is a mandate that must be implemented. However, Member States retain some discretion in the implementation. In that way, a Directive acts as a floor, rather than a ceiling. On the other hand, EU Regulations are binding in their entirety and directly applicable in all EU Member States and are therefore inflexible.  The Draft AI Act falls into this latter category, necessitating a careful understanding of its provisions which will be directly applicable in all Member States following the relevant transition period.

AI SYSTEM

Under the Draft AI Act, an AI system is “a system that is designed to operate with elements of autonomy and that, based on machine and/or human-provided data and inputs, infers how to achieve a given set of objectives using machine learning and/or logic- and knowledge-based approaches, and produces system-generated outputs such as content (generative AI systems), predictions, recommendations or decisions, influencing the environments with which the AI system interacts.”⁵AI Act, Article 3(1).

THE AI ACT: A TIER BASED APPROACH

The Draft AI Act addresses risks of specific uses of AI and seeks to regulate AI systems based on their classification into four risk categories (high risk, limited risk, minimal risk, and unacceptable risk).⁶Regulatory Framework Proposal on Artificial Intelligence, European Commission. (Sept. 29, 2022), https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai.  

Similar to the classification of medical devices, we may expect the European Commission to provide some guidance on the classification of AI systems. Ultimately, however, it will be up to the Court of Justice of the European Union (CJEU) to decide whether a specific AI system carries an unacceptable risk and must therefore be banned under the eventually adopted AI Act.  

Article 23a: NOTE TO DEVICE Manufacturers

The Draft AI Act defines a “provider” as a “natural or legal person, public authority, agency or other body that develops an AI system or that has an AI system developed and places that system on the market or puts it into service under its own name or trademark, whether for payment or free of charge.”¹¹AI Act, Article 3(2).

Pursuant to Article 23A of the Draft AI Act,  where high-risk AI systems are safety components of medical devices and/or in vitro diagnostic medical devices or they are medical devices themselves, the device manufacturer is considered  to be the provider of the high-risk AI system and will be subject to the obligations of providers under either of the following scenarios:

• the high-risk AI system is placed on the market together with the product under the name or trademark of the device manufacturer;
  
  
• the high-risk AI system is put into service under the name or trademark of the device manufacturer after the device has been placed on the market.¹²AI Act, Article 23a.

If either scenario is present, the manufacturer must comply with the obligations of providers of high-risk AI systems as set out in Article 16 of the Draft AI Act. The obligations include ensuring, among other things, that the AI system complies with the AI Act, and that a quality management system and a conformity assessment and certification of the AI system have been implemented.

article 43(3): CONFORMITY ASSESSMENT

According to Article 43(3) of the Draft AI Act, for high-risk AI systems under the scope of the Medical Device Regulation and the In Vitro Diagnostic Medical Devices Regulation, the provider will have to comply not only with the relevant conformity assessment as required under these legal acts, but also with other relevant requirements set out in the Draft AI Act. ¹³AI Act, Article 43(3).

In case of a substantial change which may affect the compliance of a high-risk AI system with the Draft AI Act (e.g. change of operating system or software architecture), or when the intended purpose of the system changes, that AI system should be considered a new AI system which must undergo a new conformity assessment. However, changes occurring to the algorithm and the performance of AI systems which continue to ‘learn’ after being placed on the market or put into service (i.e. automatically adapting how functions are carried out) do not constitute a substantial modification, provided that those changes have been pre-determined by the provider” ¹⁴AI Act, Recital 66. and assessed at the time of the initial conformity assessment and are part of the information provided in the required technical documentation. ¹⁵AI Act, Article 3(23).

ARTICLE 10 ObLIGATIONS: dATA AND DATA GOVERNANCE

Of the obligations likely to be most important to manufacturers, obligations under Article 10 relate to data and data governance.¹⁶AI Act, Article 10.  Article 10 states that “[h]igh-risk AI systems which make use of techniques involving the training of models with data shall be developed on the basis of training, validation and testing data sets that meet the quality criteria” set forth in the Article.¹⁷Id. 

• The training, validation, and testing data sets must be subject to “appropriate data governance and management practices,” which “shall concern in particular: (a) the relevant design choices; (b) data collection; (c) relevant data preparation processing operations, such as annotation, labelling, cleaning, enrichment and aggregation; (d) the formulation of relevant assumptions, notably with respect to the information that the data are supposed to measure and represent; (e) a prior assessment of the availability, quantity and suitability of the data sets that are needed; (f) examination in view of possible biases that are likely to affect health and safety of natural persons or lead to discrimination prohibited by Union law; (g) the identification of any possible data gaps or shortcomings, and how those gaps and shortcomings can be addressed.”¹⁸Id.
  
  
• The data sets used in connection with the AI system must be “relevant, representative, and to the best extent possible, free of errors and complete,” with “appropriate statistical properties, including, where applicable, as regards the persons or groups of persons on which the high-risk AI system is intended to be used.”¹⁹Id.  Notably, “[t]hese characteristics of the data sets may be met at the level of individual data sets or a combination thereof.”²⁰Id.
  
  
• The training, validation and testing data sets must also “take into account, to the extent required by the intended purpose, the characteristics or elements that are particular to the specific geographical, behavioural or functional setting within which the high-risk AI system is intended to be used.”²¹Id.
  
  
• The providers of high-risk AI systems may process special categories of personal data (as defined in the EU General Data Protection Regulation) “[t]o the extent that it is strictly necessary for the purposes of ensuring bias monitoring, detection and correction.”²²Id.  Yet, the use must be “subject to appropriate safeguards for the fundamental rights and freedoms of natural persons, including technical limitations on the re-use and use of state-of-the-art security and privacy-preserving measures, such as pseudonymisation, or encryption where anonymisation may significantly affect the purpose pursued.”²³Id. 

• The development of high-risk AI systems not using techniques involving the training of models would only be subject to certain Article 10 provisions governing the testing data sets.²⁴Id.

Other Requirements

The following non-exhaustive list notes other compliance requirements that those deemed “providers” must follow under the Draft AI Act:

• Technical documentation – Prior to the placing on the market or the putting into service of the AI system, the provider must draw up technical documentation to demonstrate that the high-risk AI system complies with the requirements set out in the Draft AI Act, and the provider must give national competent authorities and notified bodies “all the necessary information in a clear and comprehensive form to assess the compliance of the AI system with those requirements.”²⁵AI Act, Article 11.
  
  
• Human oversight – “High-risk AI systems shall be designed and developed in such a way, including with appropriate human-machine interface tools, that they can be effectively overseen by natural persons during the period in which the AI system is in use.”²⁶AI Act, Article 14. The Draft AI Act provides a variety of methods to ensure human oversight.²⁷Id.
  
  
• Document retention – “The provider shall, for a period ending 10 years after the AI system has been placed on the market or put into service, keep at the disposal of the national competent authorities: (a) technical documentation; (b) documentation concerning the quality management system; (c) the documentation concerning the changes approved by notified bodies where applicable; (d) the decisions and other documents issued by the notified bodies where applicable; (e) the EU declaration of conformity.”²⁸AI Act, Article 18.

Reaction from Industry: concerns about overregulation and lack of technical feasibility

The day after the European Council adopted its revised version of the Draft AI Act, MedTech Europe, the European trade association for the medical technology industry including diagnostics, medical devices and digital health, released a statement detailing its reaction to the Council’s approach.²⁹MedTech Europe’s reaction to the EU Council’s General Approach on the AI Act, MedTech Europe (Dec. 7, 2022).  The trade association was quick to note that the Medical Devices Regulation and In Vitro Diagnostic Medical Devices Regulation already detail stringent requirements aimed at ensuring a “high level of protection of health and safety before the technologies are placed on the market.”³⁰Id.  MedTech Europe stated that “[b]y adding another regulatory layer, the [Draft] AI Act risks creating legal uncertainty and unnecessary regulatory burdens on providers of AI-enabled medical technologies because of potential duplicate or contradicting requirements.”³¹Id.  The association also shared its reaction to the Council’s changes to the Draft AI Act’s provisions related to sectoral alignment, definitions of AI systems, and requirements for high-risk AI.³²Id.  For each of these areas, MedTech Europe shared its key concerns and proposed various recommendations.³³Id.

In February 2023, a list of 14 organizations representing European and non-European designers, developers, deployers, and users of AI (including the Confederation of European Business and MedTech Europe), published a joint industry statement on the Draft AI Act and the current discussions for its final form. ^{34Joint Industry Statement on the EU Artificial Intelligence Act, (Feb. 23, 2022)} The joint statement highlighted the high number of modifications to/deviations from the original AI Act proposal and urged the EU Parliament which currently drafting its position, “to ensure that any possible new requirements and amendments to the Draft AI Act are introduced taking into account their technical feasibility, impact on legal certainty, and the ability of AI developers, deployers, and users to comply with them”. ³⁵Id. The joint statement addressed further recommendations to the EU Parliament for the next steps of drafting. ³⁶Id.

It remains to be seen how the above concerns about overregulation and lack of technical feasibility and recommendations may be reflected in the following revisions of the Draft AI Act, as the official public consultation period is closed.

NEXT STEPS

A general transition period of three (3) years is expected following the adoption/entering into force of the AI Act.³⁷AI Act, Article 85. Following the transition period, products that are not in compliance with the Draft AI Act may not be placed on the market, put into service, or used.³⁸See id.

The Draft AI Act is currently under discussion in the EU Parliament before the initiation of negotiations between the EU Council and Parliament for purposes of agreeing on the final text. Press reports indicate that further amendments have been added to the draft proposal, including clarifications and updated definitions. The European Parliament rapporteurs for the Draft AI Act indicate that the consultations are to be completed in April 2023. Two (2) more technical and three (3) political meetings are scheduled within that timeframe. Medical device companies that could potentially be subject to the Draft AI Act should now assess whether their products fall under the high risk, limited risk or unacceptable risk categories and, under which requirements their devices may be placed on the market. Manufacturers of devices, which qualify as high-risk AI systems, should begin to evaluate the extent to which their data governance and other programs are compliant with these proposed requirements and, in particular, whether the data sets used or planned to be used on product development fulfill the quality criteria under the Draft AI Act. 

King & Spalding regularly counsels medical device manufacturers and can support device manufacturers, among other issues, on market access, risk evaluation and conformity assessment procedures, communication with Notified Bodies and national competent authorities, questions regarding the interplay of the Draft AI Act with the EU Medical Devices and In Vitro Diagnostic Medical Devices Regulation, AI system risk assessments, data assessments, data governance and management policies and procedures, and quality management system policies and procedures. If you have questions regarding the Draft AI Act or would like assistance in ensuring compliance readiness, please contact Jarno Vanto and Elisabeth Kohoutek for more information.