Episode 160: SATs Data Revealed! Here’s how Maths — No Problem! schools performed in comparison to the national average

Andy Psarianos

Hi, I'm Andy Psarianos.

Robin Potter

Hi, I'm Robin Potter.

Adam Gifford

Hi, I'm Adam Gifford.

Andy Psarianos

This is the School of School podcast. Welcome to the Scho ol of School podcast.

Welcome back everyone to another super exciting episode of the School of School podcast. We're here with Robin Potter, as per normal.

Robin Potter

Hi.

Andy Psarianos

And me and Robin are actually in the same room for a change. Poor little Adam, still by himself in old Manchester England, is with us as well.

Adam Gifford

Yeah, hi, team.

Andy Psarianos

Yeah, hi. And special guest today, Louise Hoskyns-Staples. Very exciting to have her here with us to talk about some research that she's done around the use of Maths -- No Problem! in England. And before we dive into that, Louise, can you tell us a little bit about yourself for our guests?

Louise Hoskyns-Staples

Yeah. Hi, I'm Louise Hoskyns-Staples. I've been a friend of Maths — No Problem! since January 2014. And I currently work as a teacher-educator, working with teachers who are in school. I evaluate some national projects for maths for the National Centre, and I do a little bit of work for the DFE occasionally. And I'm an Associate Lecturer for the University of Worcester.

Andy Psarianos

Very exciting.

Robin Potter

Wow.

Andy Psarianos

So Louise, this is a recent research here that you did, which was commissioned, I think. Can you tell us a little bit about what it was and what you found?

Louise Hoskyns-Staples

Yeah, I was asked if I would have a look at the data from the national key stage 2 tests and compare how the Maths — No Problem! schools had performed with the national data set. We had a sample of schools, 10% of the Maths — No Problem! schools responded to a request. So they were self-selecting in a sample in that way. However, I've spoken to colleagues who use Maths — No Problem! and are very fond of it, who didn't actually get round to filling that in. So I suspect it was people who were organised and we caught them on the right date.

The sample initially, I was a little bit sceptical about what I could see from a small data set as such. So I started to dig into the data and then started to get more and more and more excited. I was looking at the average scores and then looking at the impact that Maths — No Problem! had, the percentage of people getting higher scores, the percentage of people meeting the national standards. And also what really stood out was the fact that our pupils who were eligible for free school meals, so disadvantaged pupils, were performing really, really highly. And nationally there's a huge disparity.

So I had a look through, and I'm just going to call up a little bit of data for this one because I don't keep numbers in my head. But the national... So looking at the free school meals, the average performance, there wasn't any significant difference for children from disadvantaged backgrounds to children from normal backgrounds. Whereas we'd expect there to be significant difference. And it's one of the things when I'm working with schools, I always ask them, when we look at their data, "What about your disadvantaged pupils? How are they doing?" They're usually significantly lower because of a range of factors.

And what I noticed in the Maths — No Problem! schools was there wasn't this disparity that you would get elsewhere. Nationally, 57% of pupils you'd expect to meet the expected standard, and seeing 79% with Maths — No Problem!, the difference between 50...

Andy Psarianos

Wow.

Louise Hoskyns-Staples

Yeah, so it's over 20% higher. It really is... Yeah, it's a real wow. And it was that that I started looking at.

And then the Maths — No Problem! schools tended to be schools with more pupils on free school meals. So then I started looking and saying, "Well actually the Maths — No Problem!'s data, these schools are all coming up higher than the national average anyway, and they've got a much higher percentage of pupils who are in disadvantaged circumstances." So you'd expect the Maths — No Problem! school data to be lower than the national data, if you were going purely on the pupil profiles of those schools. And yet it's higher than the national data. And not hugely higher but higher nonetheless.

The higher scores, this is where our disadvantaged pupils fall down again, and nationally, 24% of disadvantaged pupils get a higher score, it's 28% with Maths — No Problem!. And yet that's 4% higher. But I was starting to look and thinking, this is really, really quite unusual. That was the bit that got me most excited, if I'm honest. The rest of the data, my analysis was looking at where the average scaled score was, but the picture across the board was, if you're a pupil in a Maths — No Problem! school, no matter what your background, no matter the difficulties that you face, you are going to get a good mathematics education.

Adam Gifford

Can I just jump in and ask something just to clarify this? 'Cause it is wow, but I think, and I'm assuming that a lot of our listeners will know this, but that disparity between two, this isn't just a recent thing. This is a very well-established disparity. So to see it, to see that that closed, this has been a perpetual problem. This has been happening in schools for a long time. Am I right in saying that? I mean, you evaluated at a national level, so...

Louise Hoskyns-Staples

Absolutely, yeah. It's historic. We've talked about closing the disadvantage gap for a long, long time. And schools using Maths — No Problem! are closing the disadvantage gap. That for me was sitting there and realising that I needed to just stop the jaw dropping when I was analysing the data.

Andy Psarianos

And I think for those, I suppose... So this is for our listeners who maybe are not so familiar with the significance, I suppose, of what we're talking about. Just in simplistic terms, I think it's worth stating that in education there's a lot of perpetual change all the time, right? It's easy to change things in education, in the sense of... It's not easy, but it happens all the time. Changing the national curriculum or changing a particular thing about the education, whatever it is, the way the funding is done, all these things are actually reasonably easy to do. And there's always a lot of change in education.

Almost none of it ever results in changes in performance. So you can spend a tremendous amount of money in education and change all kinds of things, but very rarely do the results ever change. And that's why this is really significant, because we're not talking about massive amounts of investments.

So if you take a look at a programme like Maths — No Problem!, 'cause we're talking about Maths — No Problem! here. But we are an outlier. Maths — No Problem! is an outlier in the sense that we do actually generate results. I guess the important thing to understand is you can spend oodles of money in education and make absolutely no change. We know that and we've done it a million times. This is a relatively small amount of money. You're talking about less than 1% of the amount of money a school receives for a pupil. You're talking about less than half of 1% of what a school gets for a pupil. Actually you're talking about less than a quarter of 1% of what a school receives for a pupil. That's the investment we're talking about. Okay?

So if you didn't buy Maths — No Problem!, or you did buy Maths — No Problem!, just as an example, it makes absolutely zero... It's not a noticeable thing on your budget. Everybody talks about how expensive it is, it's not expensive. We spend oodles of money in education, and this is the only thing that I've ever heard of that has a measurable impact of this magnitude, in education, for those who need it the most. So I think this is really, really important, it's really, really profound. Everyone needs to look at this.

Louise Hoskyns-Staples

It's the fact that if... I agree with you entirely. There are noticeable changes, if you look at data over time you can see the impact of the national numeracy strategy, which I'd say is when we first really started focusing on how we teach maths. And so you do get blips, and we see a blip again later with the mastery approach being used in the UK. This is a big blip, looking at this data.

The fact that it was consistent across the schools and the percentage of pupils who are achieving the expected standard is... It's higher than average, despite the fact that the profile of a Maths — No Problem! school tends to be a greater number of disadvantaged pupils. The percentage of pupils who achieved a higher score was higher in Maths — No Problem! schools than it was in the national average. The average performance data of the sample was higher. So across the board, whatever aspect I looked at, it was higher.

And there were schools, there was some schools... I counted all the data so there was the odd school, I averaged it out, the odd school where the data was not so good. And so I dug into it. There was one school in particular, I spent quite a long time analysing data in the background. I looked at the school's Ofsted reports and things to see... The school where the data wasn't reflective of the other schools I looked at, because they'd had an Ofsted, their data was poor. And they'd had Ofsted the year before and their Ofsted had been good.

So I was trying to work out how they'd had Ofsted that was good around in a year, when their data for year six was really quite low. In their Ofsted report it said they'd had an awful lot of new students join in year six. So the year that they would sit their assessment. So those children hadn't experienced Maths — No Problem! throughout their school career. And so yes, there was an impact on the school data that was totally explainable.

And I went through each of the Ofsted reports, looking. And I think that's part of this as well. In these schools, you've got engaged and interested head teachers, the Ofsted grades are generally higher. They're not all in green leafy suburbs, by any way at all. But these are head teachers who've made a conscious decision to put Maths — No Problem! into their schools, and they've done that for the benefit of their pupils. And I think that's what this is reflective of. You've got strong heads who are able to lead schools and make strong decisions.

Andy Psarianos

Yeah. And I think that's really... One of the things that you said there, which I think is really important, and we know this, but the profile of Maths — No Problem! schools, if you look at, it's very representative of what's out there as far as schools go. I mean, some of the schools that we have are some of the largest schools in the country, that are inner city schools that have a particular profile. And some of the schools are some of the smallest schools in the country that are in the most remote locations. And we do have that... And everything in between, representative of all kinds of different socioeconomic profiles as well.

So I know you mentioned at first you thought there was maybe a bit of, I suppose, concern about the sample size. I mean, can you talk a little bit about that?

Louise Hoskyns-Staples

Yeah. 10% of the Maths — No Problem! schools... We took schools, I exclude... Obviously international schools don't take the tests, et cetera. And the independent schools that use Maths — No Problem!, We didn't count any of their data. So it's 10% of the schools. So it was a relatively small sample size. That does mean that you can have a bias towards something.

However I was quite confident that... I think if the school profiles had been more mixed, if there'd been... It was the percentage of disadvantaged pupils for me that really had the impact, because it was a big enough sample size, there was 60 schools so it was an awful lot of students. For me, that sample size was large enough when we multiply that out by... As you said, some of these schools are huge. The tiny schools won't have been included because they don't have to present their national data, so I won't have had their data. So we are only looking at schools of a reasonable size.

And the fact that, yes, that profile of this sample and their data being higher than the national average, when looking at the profile of the school, I'd have said their data should be significantly lower, 20% lower really

Andy Psarianos

Interesting.

Robin Potter

And sorry, just wondered the period of time that you were looking at.

Louise Hoskyns-Staples

So this is a snapshot in time, and there's a lot more work that we can do here and more things we could look at. But this snapshot in time, this is last summer's national tests for eleven-year-olds. So the key stage two tests in May last year, and the national data and the Maths — No Problem! data that came from that.

Adam Gifford

I just wanted to say, and I know this is stating the obvious because we're talking a lot about statistics and those sorts of things. But these are real kids, and this is a difference to life opportunities, right? This is a cycle that happens all the time and there are so many predictors out there, where mathematics is a key predictor. You look at the OECD data and you think about health, happiness, wellness, all of those sorts of things.

So the reality is is that that changes life expectations, so many different things for these children. These are real children who have certain characteristics that allow us to categorise them in schools. We have to do that, it's just part and parcel of what we do. But the outcomes, to break that cycle and what that means to real life humans, that's pretty profound, man. I mean, we talk about that, we say one child that makes a difference. If you're doing that to more, that's really important. That's really important.

Louise Hoskyns-Staples

We also know that mathematics grades, particularly at GCSE because that's what it's been analysed on, have the biggest impact on life earnings. So setting these children up really, really well in maths in primary school means they're going to be entering secondary school with a very solid basis. So my assumption is we know if children have a good basis in maths, and that's then maintained, these children are going to have better life outcomes. And primary maths is really, really, really important. And these children have had a fantastic start, and they're leaving primary school ready, and extremely ready, for their secondary school learning.

Andy Psarianos

Yeah, absolutely. And again, I will talk about Eric Hanushek again. I've been fascinated with his work for the last year and a half, and I mean, the arguments that he makes... So for those of you who don't know, Eric Hanushek is a economist out of Stanford University, and has devoted his life on... Well not his life, but at least his most recent work on the impacts of mathematical education on the economies of the world, and has mapped out effectively the impact to be trillions, trillions of dollars worldwide.

If you could get those at the bottom, just to meet a benchmark of what he calls... I wish I could remember what he called it, but it's basically qualified to be a member of the modern economy. And the best benchmark is math scores at the age of 15. And he benchmarked it against one question, which is a really straightforward question, which everyone leaving primary, by the way, should be able to do.

The impact on economy is tremendous. The impact on the individuals is tremendous if you can just get their maths levels to a certain level. I mean, this is such an important thing, especially for the underprivileged, because if you're coming from... Underprivileged? That's not the right word. You know what I mean, those at a disadvantage. If you're looking at children who are coming from very stable socioeconomic, and richer and whatever, I mean, the odds are stacked in their favour. We know that.

But those kids who don't come from those backgrounds... And this is a global thing. I'm not even just talking about England here. If you look at places like Sub-Saharan Africa and bits of South America and stuff, if you can get those kids to perform mathematics at a certain level, we know that that is what is going to change a nation, never mind the individuals. It's so important. And that's exactly what we're doing here, it's exactly what we're talking about, is taking those who are likely to be unemployable in a modern economy and giving them a chance, giving them an opportunity. Right?

Louise Hoskyns-Staples

Yeah, absolutely.

Andy Psarianos

That's what we're talking about.

Louise Hoskyns-Staples

Yeah, we are. And yeah, fundamentally that's why I got so excited by this data, because this is really important.

Andy Psarianos

And us too, because this is what we set out to do. This is our mission statement at Maths — No Problem!, is giving every child the opportunity they deserve. And this is exactly what this is about, 'cause the reality is Maths — No Problem!'s now been used by millions of children, and in those millions of children, many of them fall in that category. And many of them now will have the opportunities that they wouldn't otherwise have had, and that's what this is all about.

Louise Hoskyns-Staples

And looking at the sample as well, I want to see how long the schools have been using Maths — No Problem!. Because if it was a school that's just put in Maths — No Problem! six months ago I don't think we can claim that it's had the impact on the data. But it was five and a half years that these schools have been using, and that corresponded to the national funding, when there was match funding for primary schools and they bought the scheme, the government funded half. So five and a half years. Actually there's years one to six, they sit this in year six. So they've used Maths — No Problem! for their primary education. So I think we really definitely can say that it's the impact of Maths — No Problem! on their learning.

Robin Potter

I can't wait for the next study she does.

Andy Psarianos

Yeah, what's the next study, Louise? What should we look at next?

Louise Hoskyns-Staples

I'd quite like, funnily enough, to look at gender. That's on my list, actually, 'cause I want to dig into the insights data. So yes, that's also on my list of things to really get... The nerd in me is very happy when I'm analysing the data.

Robin Potter

Well we're so happy the nerd is happy.

Andy Psarianos

We're happy to know nerds like you, Louise.

Louise Hoskyns-Staples

Yeah, yes.

Robin Potter

Hope you come back again. Thanks so much for joining us.

Louise Hoskyns-Staples

I'd love to come back again. Thank you for having me.

Andy Psarianos

Thank you for joining us on the School of School podcast.