Building equality and data protection into AI commissioning and procurement

The Local Government Association (LGA), London Office of Technology and Innovation (LOTI), the Equality and Human Rights Commission (EHRC), and the Information Commissioner's Office (ICO), have in collaboration developed a guide to help councils in England responsibly procure and commission AI-based technologies (both goods and services).  The guide aims to ensure compliance with the Public Sector Equality Duty (PSED) and data protection law.

What is the guide?

The guide provides practical support for those in various roles involved in the AI procurement process.  It offers questions and prompts tailored to each role, helping to navigate the complexities of AI procurement and ensure responsible deployment.  Some of the questions and prompts have been adapted from the UNESCO Ethical Assessment Template and EHRC and ICO guidance.

The guide is intended to act as a framework to be used to: 

• consider whether an AI-based technology:
  • is likely to deliver positive benefits to council and the public;
  • advances equal opportunities for people with protected characteristics or whether it instead may lead to discrimination; and
  • fosters good relations between communities who engage with public services;
• help councils obtain the information required to report on the benefits of the AI technology to the council workforce, communities and the public as a whole; and
• encourage staff who are considering using an AI-based technology to engage with specialist teams early on and throughout the procurement and contracting stages.

Whilst the guide is aimed at councils, it notes that other public sector bodies subject to the PSED may find it useful.

What does the guide aim to cover?

1. Assessing equality and data protection risks:  The guide emphasises the importance of thoroughly assessing the equality impact and data protection risks before and throughout the procurement process. 
2. Probing AI providers:  Councils are encouraged to ask AI technology providers about the equality and data protection considerations that they took into account during the development and testing of their technologies.
3. Designing tendering and contract arrangements which build in equality and data protection requirements. This helps councils design successful procurement processes, monitor the impacts of using AI technologies, and ensure equitable, lawful and safe use.
4. Offering practical tools and recommendations through the lifecycle of a contract for best practices in compliance with data protection law and the PSED (though it does not intend to replace other specialist or regulator guidance).

Which roles are the guide aimed at supporting?

The guide is split into six sections aimed at different roles within councils identified as key to commissioning and procurement of AI:

• (1) & (2) – All officers involved in purchasing and contracting out AI-based technologies: focusing on consideration and review of equality and data protection impacts.
• (3) – Commissioners and project managers: focussing on assessment of the particular AI-based technology in the pre-tender stage, including organising the project team before deciding to purchase AI-based goods or contract out for an AI-based service.
• (4) – Procurement officers: focusing on building equality and data protection into contract specifications, tender documents and interview questions.
• (5) – Council officers: who are evaluating tender answers, focussing on ensuring equality and data protection considerations are included in the tender evaluation process.
• (6) – Contract owners: focussing on building equality and data protection clauses into contract and service agreements and ensuring they obtain data from the AI providers to allow them to monitor the equality and data protection impacts of using the technology procured.

The full ‘How to build equality & data protection into your AI commissioning and procurement processes: a guide for councils in England’ guide can be assessed here.

If you would like to discuss how current or future regulations impact what you do with AI, please contact Tom Whittaker, Brian Wong, Lucy Pegler, Martin Cook, Liz Smith or any other member in our Technology team.

For the latest on AI law and regulation, see our blog and sign-up to our AI newsletter.

This article was written by Ryan Jenkins.