Last week’s EPI report about this year’s GCSE results revealed a deep and continuing disadvantage gaps, underscoring the importance of addressing inequalities in education. But are we really able to do so with our current blunt measures of school effectiveness?
Published last week, the Northern Powerhouse Partnership’s Fairer Schools Index (FSI) raised questions about the usefulness and dangers of using school-level data, notably Progress 8 (P8).
P8 compares a student’s progress across eight qualifications from the end of key stage 2 to the end of key stage 4 against other students with similar starting points. Since its introduction in 2016, various governments have considered this approach to be fair because it is believed to encourage all schools to achieve the highest outcomes.
Unfortunately, as FSI makes clear, the P8 measure is flawed. Because it has no regard for contextual factors, it favours schools serving the least disadvantaged areas and disfavours those that are achieving notable improvements in student outcomes for the most disadvantaged communities.
Some schools do achieve amazing progress outcomes across both the P8 measure and the FSI measures. This is a notable achievement and worthy of significant independent analysis to better understand how this is being achieved.
However, some (such as a large secondary in the north east with multiple deprivation and cohort factors) jump from a category of ‘Well Below Average’ on the P8 measure to ‘Average’ using the FSI algorithm. Meanwhile others (like one large secondary in London) drop from ‘Well Above Average’ P8 to ‘Average’ FSI.
P8’s importance to school accountability is profound. It has become a shorthand for conveying a school’s effectiveness, been cited as a reason for an academy transferring to another trust, been used as damning evidence within Ofsted inspections – even at a time when curriculum quality is the inspectorate’s main focus.
There are signs that the new chief inspector is keen to strengthen the place of progress measures in a future inspection framework. Indeed, many think it is likely to appear as an indicator in future report cards.
But if this isn’t managed skilfully, it could undermine efforts to improve the accountability system.
This should be a warning to those developing the report card
Stories abound among headteachers and trust CEOs of less scrupulous players inflating their P8 scores through actions that, if not downright unethical, at least undermine the government’s mission to deliver a more inclusive education system.
These include removing lower-performing pupils into alternative provision or encouraging them into elective home education, discouraging children with additional needs from attending the school or designing the curriculum to maximise the P8 score.
These actions are evidence of a system that relies too heavily on one particular measure, and a flawed one at that. They should act as a warning to those developing the report card and any future inspection framework to be alert and sensitive to such unscrupulous actions and their systemic drivers.
Whatever approach is adopted for the future, it is imperative that school leaders, governors, trustees and inspectors understand the limitations of school data.
Progress scores are not measured with complete certainty. If a school had taught a different cohort of students, the score would have been different. The 95-per cent confidence interval shows how much uncertainty surrounds the published score.
A narrow confidence interval means a fair level of confidence that other cohorts of students would have resulted in similar reported values, while a wider interval suggests greater uncertainty. The smaller the school cohort, the more its reported score would have differed across other samples of pupils
These factors must be fully understood by those undertaking the evaluation. Yet, from our experience, the level of awareness is weak, leading to ill-informed assertions about school performance.
We are advocates for the careful and sensitive use of school data. We need much better understanding of what it is reporting and its limitations.
Without significant training and development for all those who create, use and analyse school performance and progress data, we face a future where it continues to be not just misunderstood but occasionally misused for unscrupulous ends.
This is unfair to many hard-working schools, lets questionable performance go unchallenged, and hampers any efforts to really get under the bonnet of school improvement.
