����Vlog

There is a perception that the��algorithm initially used in Scotland to determine students’ grades this summer��was malign��and that all the problems with the grading stemmed from the statistical model.

But to blame a statistical model is misguided, an academic has told MSPs - she said it would be better to focus attention on the decisions that led to the creation of the statistical model.

“Statistical algorithms do what they are told to do,” she said.

�������ٱ��:��Decision to cancel exams could come as late as February

Background:��National 5 exams cancelled amid coronavirus fears

Opinion:��How do we avoid an avalanche of SQA assessments?

News:��Call for Scotland to follow Wales and cancel exams

That was the argument put forward by Dr Marina Shapira, of the University of Stirling, at a meeting of the Scottish Parliament’s Education and Skills Committee, where she and Professor Mark Priestley were discussing the findings of their review of this year’s��national qualifications fiasco��- as well as what needs to happen to ensure that the awarding of qualifications runs smoothly next year.

Coronavirus: The fallout from the SQA results U-turn

Professor Priestley, also from the University of Stirling,��said there was ample evidence that the SQA had acted with “complete integrity and professionalism” but that the body lacked the “resources and, in some ways, the experience” to deal with the changed situation brought about by the coronavirus.

This year, he said, his “fear” was that the solutions put in place would “be bureaucratic, will involve vast amounts of teacher workload [and]��will be based around quite narrow assessment approaches”.

He said he hoped there would be a move away from “a reliance on pencil and paper testing” and that “a mixed economy of assessments and methods” would be used.

Here are the key points raised by Professor Priestley and Dr Shapira:

• Independent researchers should look in detail at the algorithm used by the SQA and explore alternatives because there is “some merit” in taking a statistical approach to grading and we need to find out how to do it better, said Professor Priestley.
• The SQA acted with “complete integrity and professionalism” but lacked the “resources and experience” to deal with the changed situation, said Professor Priestley.
• The SQA put in place “a pretty good technical solution” but it failed to consider the social impact - if it had taken a multi-agency approach that might not have happened, said Professor��Priestley.
• The SQA gives the impression of being “a hierarchical organisation” that has “a lot of technical expertise and sees itself as somehow not needing to work with other people”, said Professor Priestley. This organisational culture needed to change “to make some progress in the future”, added Dr Shapira.
• There has been an erosion of trust in the SQA and work needs to be done to restore that, said Professor Priestley.
• The SQA was relying on the final stage of its process - the appeals stage - to give those students disadvantaged by its alternative grading model a means of redress, and was expecting a large number of appeals. However, this came as “a complete shock” to young people who expected to receive their final results on 4 August. The psychological impact of that was “significant”, said Professor Priestley.
• A small number of students are still “fighting” to challenge their teacher estimates. Professor Priestley said his personal view was that “it would not hurt to have another look at these cases”, adding “we are talking about the life chances of individual young people”.
• Over the summer of 2020 the SQA should have shared the outcomes of its statistical moderation with councils to allow them to look at the “cohorts and subject level anomalies and outliers” to try to understand discrepancies, but this did not happen. However, Professor Priestley and Dr Shapira acknowledged that not all councils had the capacity to engage in such a process.
• Worries about bias led to SQA results data being anonymised - but that prevented the SQA from assessing the impact of its moderation on disadvantaged students until the results were released
• A lot of literature suggests that��teacher estimates are inaccurate - but another possibility is that a teacher making a judgement based on coursework is measuring something quite different to what is measured in an exam. Teacher judgements might actually provide “a more valid measure of student performance over time”, said Professor Priestley. He said that’s why he would like to see a more “mixed economy” of assessment in the future.
• The review found no evidence that grades were inflated by teachers and schools due to accountability pressures - where inaccuracies came it was because of the difficulty of collating enough evidence, said Professor��Priestley.
• If you have a system that assumes qualifications stay the same year-on-year that denies the possibility that good teaching and policies like the Attainment Challenge can improve attainment.