What should legal counsel consider when procuring, developing, deploying AI systems?

If a business is procuring, developing or deploying AI systems, in-house legal counsel will usually be involved. 

Here we summarise some of the key areas which in-house legal counsel should consider. 

1. Consider what AI systems the business uses now and intends to use in the future. Each may require a different approach to their procurement, development and deployment. For example, they may each pose different risks, require different stakeholder engagement, and have different compliance requirements.
2. Legislation and guidance.  Remember that existing laws may apply to the business’ procurement, development and deployment of an AI System and new laws may apply soon. There may also be sector specific guidance that is applicable.  For example,  the EU AI Act (see our helpful flowchart: Navigating the EU AI Act: flowchart, Tom Whittaker, Brian Wong (burges-salmon.com)) and guidance released by sector-specific governing bodies and regulators such as the ICO and MHRA (for example, see MHRA guidance for Software and AI as a medical device).
3. Procuring AI systems:
   1. Public procurement. Be mindful of the specific guidelines and requirements in the public sector. There are significant legislative changes also in the UK, including Procurement Act.  Public procurement rules are relevant to those selling into the regulated public sector, too.
   2. Contract. AI may present novel risks.  For example, what are foreseeable harms for an AI system may be difficult to identify with precision.  Contracts may follow a standard framework of terms, but the detail will need to be tailored to the AI system being procured.  If the business is a public body contracting for the procurement of an AI system, consider whether the EU AI Model clauses would be useful to include in contractual documents. Even those based in the UK and/or in the private sector may find these to be a helpful starting point. See our blog post Public procurement of AI - EU AI Act model clauses, Tom Whittaker (burges-salmon.com) for more information. The clauses should be customised to each specific contractual context and as guidance and technical standards are published in respect of the EU AI Act. They may also need to be read in light of other model clauses being developed, such as those by the (UK) Society for Computers and Law.
4. Vendor due diligence. If the business is procuring an AI system, ensure to undertake appropriate due diligence on the supplier’s processes and practices recognising that transactions for emerging technologies may require a bespoke approach.
5. Data Protection. Consider the data protection implications. For example, the role of the business in the context of data protection legislation (i.e. Data Controller, Data Processor or Join Controller), whether a Data Protection Impact Assessment is required, if updates are required to the business’ data protection polices, privacy notices, and contracts with suppliers and customers. 
6. Who are your stakeholders. Think about who are the internal stakeholders and assemble a team.
7. Keep records. Whether the business is procuring, developing or deploying an AI system it is important to keep records of all aspects of the system, such as the use case, the data used (for example, the input data and output data in the case of generative AI) and the potential risks (both to the business and individuals) also being mindful of specific record keeping required by legislation.
8. “Explainability”. A number of laws touch on “explainable” AI or AI output. Generally, being able to explain AI produced decisions and avoiding “black box AI” (namely where the decision-making process used cannot be explained in a way that can easily be understood by humans) will assist in giving the business better assurance of legal compliance and mitigating risks associated with legal compliance. 
9. Training. Ensure there is appropriate training for staff on, where applicable, internal use, development and deployment of the system.

If you have any questions or would otherwise like to discuss any of the issues raised in this article, please contact David Varney, Tom Whittaker, Liz Smith, or another member of our Technology Team. For the latest updates on AI law, regulation, and governance, see our AI blog at: AI: Burges Salmon blog (burges-salmon.com)