A 17-year-old applies for a part-time weekend job. They complete an online numerical test and a round of screen assessments, and then hear nothing. When they ask why, nobody can tell them.
A few years ago, a person would have read their application and offered some advice. Today, the first reader is likely to be an algorithm. In fact, the government-commissioned review of young people and work, led by Alan Milburn, suggests this is now the norm.
Its , published in May, describes how the young person who might once have walked into a shop and been given a chance “is now screened by a portal, a test, a recorded interview or an algorithm before anyone has looked them in the eye�.
On 13 July, Lord Clement-Jones in the House of Lords, warning that “a growing share of entry-level roles are now filtered by algorithmic screening before a human has even seen an application�.
When he asked whether employers would be required to test their recruitment algorithms for bias, the minister pointed to existing guidance.
Increasing automated decisions
Most of the debate about AI in schools concerns pupils using it, from homework help to cheating. Far less attention goes to AI being used on young people, including in school.
Some of this relates to their studying, chiefly checking for plagiarism – an area that is already highly contentious.
For example, a found that widely used AI detectors flagged, on average, 61 per cent of essays by non-native English speakers as AI-generated.
For a pupil learning English as an additional language, that can mean being accused of cheating by a tool that has simply misread their writing.
It speaks to the same issue as the job application situation – young people are entering a world in which they will be at the mercy of automated decisions.
What every pupil should learn about AI
This is why I think it is vital every pupil should leave school with an understanding of how automated decisions are made, where they go wrong and what they can do about it. That comes down to four ideas, and none of them needs advanced maths.
The first is to help them understand that algorithms rely on proxies. A system cannot see potential. It sees things such as a postcode, a school or a gap in a CV. Any of these can stand in for something an employer would never ask about directly.
Amazon scrapped a CV-ranking tool after it learned to .
Some proxies make no sense at all. When the German broadcaster BR , putting on glasses or adding a bookshelf to the background was enough to change a candidate’s scores.
The second is to understand that a system can be right on average and still make far more mistakes for some groups than for others. So “the computer said no� is not an explanation. For some pupils, it may be wrong more often than it is right.
The third is to explain that “fair� can mean different things. It might mean treating everyone the same, making outcomes equal across groups or making sure mistakes do not keep landing on the same people. These are different goals, and one model usually cannot meet all of them at once.
Fourth, and linking these together, is the knowing around how to challenge a decision. that if an organisation makes a significant decision about you with no human involved, it must tell you.
It must also let you put your side, ask for a human to step in and challenge the result. A right is not much use if you do not know you have it, though.
A new era
All of this fits naturally into the new computing GCSE. It will replace computer science and is meant to cover how technology and data are used, not just how computers work.
Of course, teaching pupils to use AI matters. But teaching them how AI judges them may matter more. The 17-year-old who heard nothing back deserved to know that the machine can be wrong, and that they had every right to ask for a person to look again.
Kesser Karim is a computer science teacher with 10 years’ experience and a whole-school AI lead. He is also a part-time PhD researcher in algorithmic fairness
To get essential briefings from ÌÇÐÄVlog magazine delivered directly to your inbox, sign up to our newsletters. And stay up to date with the latest education news via the ÌÇÐÄVlog magazine mobile app – available on and on