Do equality and high average school performance really go hand in hand?
10 January 2014
SUGGESTED
Uncategorized
Uncategorized
Trade, Development, and Immigration
4 November 2024
Note: the graphs in this post display correlations only. They are used to discuss relationships some argue exist, not to draw causal conclusions.
We are often told that school systems with the highest performance are also the most equitable. For example, when the PISA 2012 results were released, Janet Downs, of the Local Schools Network, argued that ‘PISA found that the most successful school systems tend to be those which are the most equitable. These systems don’t just concentrate on excellence for some but provide excellence for all’. But what do we mean by equity? And is the ‘virtuous cycle’ hypothesis true?
Here, I sidestep the issue of causality, which Downs and the OECD completely ignore, and rather focus on the relationship that we are told exist. The OECD defines a system combining quality and equality as (1) having high performance and (2) low impact of pupil background on achievement, and as a system where (3) all individuals reach a basic minimum level of skills. But while (2) might be an appropriate definition of equality, (3) has nothing to do with it properly speaking. In traditional terms, if country A has a lower mean score than country B, but smaller differences between pupils/schools, country A is still more equal. A high performance floor may very well be the result of a strong focus on efficiency instead of equality. Indeed, a trade-off between the two is often assumed in economic theory.
To investigate the relationship between equality and efficiency in international surveys, I instead look directly at the relationship between performance and inequities in outcomes within education systems. That can give us an indication of whether equality in results does indeed go hand in hand with higher performance.
I use the latest PISA results, which tend to be considered the ultimate yardstick for whether or not school systems perform well (although I disagree). In the first graph, we find that there is in fact a weak but statistically significant positive correlation between between-school variance and performance. The measure here is the variance between schools as a percentage of the total variance in the country (in itself expressed as the percentage of the average variance in all countries).
In the below graph, I hold constant the performance variance within schools, but there is still no relationship between equality between schools and performance. In fact, the between-school variance is now even more strongly and positively correlated with results. Thus, there is no support for the argument that low between-school variation in performance is related to high average scores.
But what happens if we focus on differences between pupils, which I argue is a better measure of equity in the education system? Well, that doesn’t change the picture. In the graph below, I look at the relationship between within-school variation and performance, holding between-school variation constant. The relationship is even stronger than the positive relationship between between-school variation and mean performance.
Continuing our investigation, we find also find a strong positive relationship between test score inequality, measured as the difference between the 95th and the 5th percentile, and average performance. The higher the difference between pupils, the higher average performance. The same analysis can be found in the PISA report, although it compares the 90th and the 10th percentiles. The results are very similar. I also looked at the absolute standard deviation in scores as a measure of inequality. Again, results are very similar.
Another way to measure the relationship between performance and equality is to look at the relationship between the strength of the impact of pupils’ background on scores and average performance. The below graph shows that while the trend line points downwards, the relationship is not significant. In other words, there is no relationship between the impact of background and performance.
We only find a different picture if we look at the coefficient of variation (sometimes called the relative standard deviation). The coefficient of variation is the ratio of standard deviation divided to the mean. This variable is indeed correlated with average performance in a way that suggests a virtuous equality-efficiency relationship. But the measure is not too interesting if we are interested in inequalities per se, simply because it takes into account (and indeed ‘rewards’) countries’ performance. That is tantamount to mixing apples and oranges, performance and equality, which is untenable. Either we focus on variation between pupils and/or schools in the different systems, or we focus on their achievement.
To sum up, therefore, it does not seem to be the case that high equality and high average performance walk hand in hand. If anything, we find a negative correlation. Again, I want to emphasise that the above has nothing to do with causality. I have also ignored the (different) question of how changes in equality are related to changes in performance, to which I will come back in a future piece. But I do question the argument that the highest performing education systems are those that combine quality with equity, as is often contended in the debate. This contention is simply not supported by the data.
We are often told that school systems with the highest performance are also the most equitable. For example, when the PISA 2012 results were released, Janet Downs, of the Local Schools Network, argued that ‘PISA found that the most successful school systems tend to be those which are the most equitable. These systems don’t just concentrate on excellence for some but provide excellence for all’. But what do we mean by equity? And is the ‘virtuous cycle’ hypothesis true?
Here, I sidestep the issue of causality, which Downs and the OECD completely ignore, and rather focus on the relationship that we are told exist. The OECD defines a system combining quality and equality as (1) having high performance and (2) low impact of pupil background on achievement, and as a system where (3) all individuals reach a basic minimum level of skills. But while (2) might be an appropriate definition of equality, (3) has nothing to do with it properly speaking. In traditional terms, if country A has a lower mean score than country B, but smaller differences between pupils/schools, country A is still more equal. A high performance floor may very well be the result of a strong focus on efficiency instead of equality. Indeed, a trade-off between the two is often assumed in economic theory.
To investigate the relationship between equality and efficiency in international surveys, I instead look directly at the relationship between performance and inequities in outcomes within education systems. That can give us an indication of whether equality in results does indeed go hand in hand with higher performance.
I use the latest PISA results, which tend to be considered the ultimate yardstick for whether or not school systems perform well (although I disagree). In the first graph, we find that there is in fact a weak but statistically significant positive correlation between between-school variance and performance. The measure here is the variance between schools as a percentage of the total variance in the country (in itself expressed as the percentage of the average variance in all countries).
In the below graph, I hold constant the performance variance within schools, but there is still no relationship between equality between schools and performance. In fact, the between-school variance is now even more strongly and positively correlated with results. Thus, there is no support for the argument that low between-school variation in performance is related to high average scores.
But what happens if we focus on differences between pupils, which I argue is a better measure of equity in the education system? Well, that doesn’t change the picture. In the graph below, I look at the relationship between within-school variation and performance, holding between-school variation constant. The relationship is even stronger than the positive relationship between between-school variation and mean performance.
Continuing our investigation, we find also find a strong positive relationship between test score inequality, measured as the difference between the 95th and the 5th percentile, and average performance. The higher the difference between pupils, the higher average performance. The same analysis can be found in the PISA report, although it compares the 90th and the 10th percentiles. The results are very similar. I also looked at the absolute standard deviation in scores as a measure of inequality. Again, results are very similar.
Another way to measure the relationship between performance and equality is to look at the relationship between the strength of the impact of pupils’ background on scores and average performance. The below graph shows that while the trend line points downwards, the relationship is not significant. In other words, there is no relationship between the impact of background and performance.
We only find a different picture if we look at the coefficient of variation (sometimes called the relative standard deviation). The coefficient of variation is the ratio of standard deviation divided to the mean. This variable is indeed correlated with average performance in a way that suggests a virtuous equality-efficiency relationship. But the measure is not too interesting if we are interested in inequalities per se, simply because it takes into account (and indeed ‘rewards’) countries’ performance. That is tantamount to mixing apples and oranges, performance and equality, which is untenable. Either we focus on variation between pupils and/or schools in the different systems, or we focus on their achievement.
To sum up, therefore, it does not seem to be the case that high equality and high average performance walk hand in hand. If anything, we find a negative correlation. Again, I want to emphasise that the above has nothing to do with causality. I have also ignored the (different) question of how changes in equality are related to changes in performance, to which I will come back in a future piece. But I do question the argument that the highest performing education systems are those that combine quality with equity, as is often contended in the debate. This contention is simply not supported by the data.
2 thoughts on “Do equality and high average school performance really go hand in hand?”
Comments are closed.
SHARE
Erm… so are any of the results statistically significant?
Also, you selectively provide t-stat, coefficients, and SE for some data sets and not for others. I’m sure you did this innocently but it looks suspicious.
P-values and coefficients for all these data sets would make your argument stronger (or would it?!). As it is, these look like damn lies and t-statistics.
The reason not all graphs provide the t-statistics is just that Stata doesn’t automatically do that for two-way graphs, in contrast to added-variable plots. But I note in the text whether the relationships are significant for Graphs 1 and 5. Didn’t spell it out for Graph 4, but I stated that there is ‘a strong positive relationship’, by which I of course meant that the relationship is significant.
But here you go again: Graph 1: statistically significant at 10% level with t-stat of 1.92 (as I state in the text – ‘weakly significant’). Graph 2: you can see the t-values underneath (significant at 1% level). Graph 3: same as Graph 2. Graph 4: significant (at 1% level – t-stat: 6.7). Graph 5: not significant (as I state in the text).
Again, the point about the post was not to provide evidence of causal relationships, but to question whether equality is related to achievement. It’s not, which would have been the case even if none of the results had been significant.