As AI Act vote nears, the EU needs to draw a red line on racist surveillance

The EU Artificial Intelligence Act, commonly known as the AI Act, is the first of its kind.

Not only will it be a landmark as the first binding legislation on AI in the world – it is also one of the first tech-focused laws to meaningfully address how technologies perpetuate structural racism.

From the racially discriminatory impact of predictive policing systems to the use of AI systems to falsely label (mostly racialised) people as fraudsters when claiming benefits, this legislation is deeply informed by a growing awareness of how technology can perpetuate harm.

Lawmakers have taken steps that would ban certain uses of AI when "incompatible with fundamental rights". This is the case for some uses of facial recognition to identify people in public places and AI systems used to predict where and by whom crimes may occur.

Negotiators in the European Parliament are, however, stopping short of recognising the specific harms that come when AI systems are used in the migration context.

From AI lie-detectors and AI “risk profiling” used in a multitude of immigration procedures to the rapidly expanding tech surveillance at Europe’s borders, AI systems are increasingly a feature of the EU’s approach to migration.

AI is used to make predictions and assessments about people in their migration claims based on opaque criteria that are hard to know and harder to challenge. 

In Europe, there are already plans to use algorithms to assess the “risk profiles” of all visitors – in a context where there is evidence that visa decisions reflect histories of discrimination rooted in colonialism.

AI systems are also part of an ever-expanding, generalised surveillance apparatus. This includes AI for surveillance at the border and predictive analytic