Episode 219: Analysing PISA Results and trends in Canada

Andy Psarianos
Welcome back everyone to another episode of the school of school podcast. We've got our regular pundits here. So we've got Robin and Adam. You can both say hi either synchronized or separately up to you guys.

Adam Gifford
How are you doing? I got on this straight away. Well, hello. Good morning, good afternoon.

Robin Potter
Yes, I'll just stick with hi.

Andy Psarianos
So we've got a guest today, a very exciting guest, actually Jerry and Jerry is sorry. Adam asked you how to pronounce your name and I didn't really pay attention, but why don't you say it is Jerry. Mussio. Yeah, that's what I thought it was, but okay. Good. And, yeah, definitely sounds Irish. Yeah.

Jerry Mussio
Mussio. Yeah. Good Irish name,

Robin Potter
Yeah.

Adam Gifford
Okay.

Andy Psarianos
I'm sure it's got something to do with growing grapes, but we might come to that later. Anyway, Jerry's, Jerry's joining us from, from Vancouver Island in British Columbia and Canada. And Jerry's worked in education quite a lot in Canada. So Jerry, why don't you tell us a little bit about yourself and some of the things you've done and feel free to just jump right into the educational system in Canada, if you want.

Jerry Mussio
Sure. I grew up in British Columbia here in Canada and went to school in the US and also grad studies at the University of Toronto and done a fair amount of international work following about 25 years or so working with the Ministry of Education here in British Columbia.

And one of the things I did toward the end of my stay with the education ministry, I ended up being counted as a representative to the OECD, advising the overall design of the program for international student assessment, commonly known as PISA.

And around that time also, I was seconded to Stats Canada, where I got involved in a fair amount of both national work here in Canada focusing on education indicators, but also doing a fair amount of international work.

Anyway, more recently, I was involved in a couple of studies here in British Columbia, including one where I prepared a case study for British Columbia, which is part of an international comparative study that included South Korea, Hong Kong, Finland, Estonia, and British Columbia.

So that gave me an opportunity to reflect a bit on some of our history in BC and more recently performance of British Columbia internationally and more specifically on with respect to the PISA results.

So in the around 2000 when PISA was first administered and by the way, Canada and British Columbia particularly been very active in international work starting off with the TIMSS 30 International math and science assessments in the, I think late 70s, early 80s.

But by the 2000s, Canada and all the provinces decided to participate fully in PISA. So we have very good data for the 10 provinces in the country as a whole around 2000, starting 2000.

And in the early assessments, to my surprise and also to many Canadians, that Canada did very well in terms of the rankings, up close to the top in the 2000 assessment, and performed at a relatively high level among the top performers through to about 2018 and around in that assessment 2018, the scores declined and fairly substantially and mathematics was the most significant one in my look at the data and not just the mean scores.

I'm particularly interested in the percentage of students that meet standards. And as I think you may know, level two in the assessments, PISA assessments, represents the standard that they define as the types of knowledge skills required to function in a knowledge society. between 2000, I guess 2003, when the benchmark year for mathematics, that was the first full PISA assessment.

In that year, 9% of British Columbia students did not meet that level two standard. In 2018, that number increased to 19% of students not meeting the level two standard. So that was a substantial decline. Now there were modest declines in reading and science literacy.

There's still declines though, but the more, from what I'd looked at, the more significant one was in mathematics. And if you looked at the 2021 or 2022 PISA data, the most recent, there was a further drop, I think, like in mathematics, that number not meeting the standard bumped up to 22%.

Robin Potter
So Jerry, Jerry, Jerry, this isn't sounding positive. You're not providing all kinds of wonderful information for us. You're telling us the truth, the cold hard facts. What's going on? Why is this happening?

Jerry Mussio
Well, yeah, so that's a big question. And I should mention that in that comparative study with the other four jurisdictions, which collectively, the five were defined and selected by the Australian Council of Education Research who are leading the study as five high performing jurisdictions.

And when you look at the other four as well, there were also, if you look at the patterned results over the 20 year period, they're very similar. So they all experienced declines. And if you look at mathematics, the one exception was in Estonia where their results, in fact, I believe increased toward 2018. So, and then you look at PISA overall, the scores have dipped over that period.

Nevertheless, I get if you look at mean scores, but I like looking, breaking that down and look at what percentage of kids are meeting benchmarks or exceeding and so on. And historically, I've always tried to do that even though say we have been performing at high levels, it doesn't mean when you look at mean scores, it doesn't mean all students are excelling and they're not.

And so I keep flagging that regardless of where the mean scores are at. so yeah, so during this period, I'm not sure what's happened internationally in terms of where there has been a dip overall in the PISA scores.

But in British Columbia during the 2000s, starting around 2010, discussions began regarding the need for a reform to the provincial curriculum in the province. And those discussions carried on for a number of years and in 2016 began that year was the first year of the implementation of the new redesigned curriculum.

Now, some people argue that because of discussions during that period that people perhaps took their eye off the ball and that was part of the reason why scores have declined. I think there's some truth to that, but also there may be something else happening. Because if you look across Canada, there's also a dip and also internationally.

Andy Psarianos
But not everyone's dipping though, Jerry, right? So some countries manage to buck the trend on that one and, you know, continue to climb. In particular, if you look at those, you know, South Korea's and Singapore and Japan, for example, you know, they're, it's an opposite story, right? So they started at the top and they're going, you know, the gap between those...

call them the highest performing nations and Canada or Finland or some of these other countries that were in your study, there is a difference, right? Something is happening.

Jerry Mussio
Yeah, China, Singapore. Well, you look at South Korea and say versus British Columbia or Canada, and also some of the others, the patterns are very similar. But you're correct that if you look at the four jurisdictions in China that participated in PISA, they are out of sight, right, in terms of their performance. I mean, historically, Singapore, obviously, and that grouping of nations.

Andy Psarianos
Yeah. And, and so what's interesting. So you said something you said, people think that maybe, maybe one of the things that happened was that people took their eye off the ball from that period of, of when the curriculum review started till 2016, when it was actually launched and then, of course, we don't know that it's impossible. If you look at PISA, it's impossible to tell what the effect of the changes in 2016 are until really the children who were affected by 2016 actually end up writing the PISA and they don't write it till they're 15.

So it's until 2031, we won't really know what the effect of the new curriculum in British Columbia, is going to have on PISA. And that's a, I think that's the thing that catches people out quite often because they'll, they'll look at, um, they'll look at a score in PISA and, know, if you look at Finland as a great example, British Columbia is kind of a similar story in some respects. when the original PISA came out for, and Finland was on top, people thought, wow, what's happening in Finland?

Finland is the greatest, you know, country as far as education goes. And, and you still hear that even though that's based off a, you know, 25 year old research now. the question to ask is not what is, what are they doing right now? But the question is, did they do 15 years ago?

And that's something that people often get caught up by. I say, well, look, you know, they're doing great. Let's look at what they're doing. But education is constantly changing in countries. So it's just a, it's just some couple of interesting things there.

Jerry Mussio
Yeah, no, I agree with what you're saying. Absolutely.

Andy Psarianos
What can you talk us through the changes the the BC curriculum changes a little bit?

Jerry Mussio
Sure.

And I should back up a bit, just Robin's saying, you know, this negative information, it's, I mean, the broader context is that in BC, we have a very strong, what I call vibrant professional culture.

A lot of teachers, like to innovate, change, look at doing things differently. So it's real, I think, strength in the system. The biggest challenge from my perspective is the issue of coherence. In other words, the alignment between sort of the mission of the system, the legislation that governs the system, how schools are organized and so on, and what teachers are teaching, what materials teachers are using, there's a real coherence issue here.

When the new curriculum came out, part of the reason for, or main reason that triggered the redesign of the curriculum was a new minister coming in and with senior staff looked at, say the PISA results, which were very positive.

And their question was, is our system actually preparing kids for the future? And so that triggered a review. And some of the shifts in terms of the former curriculum and newer curriculum centered in on issues around how the curriculum was laid out and there were concerns that there was too much of a focus on content and criticism that exam systems, assessment systems were focusing. And some of these questions, by the way, I think were not accurate, but the criticisms were that students were continuing to focus on facts and regurgitating the facts and there needed to be more of a focus on looking at sort of broader picture and some of the broader skills that kids need.

Particularly in using content and applying that content in other areas of their school studies but also in real life. So there want to be a shift between focusing on detailed content more on the application of that content say to other areas of study and across the curriculum and in daily life.

And I would say that that was sort of the major thrust. And so the implementation began in the early grades, and then by 2000, it was into the senior grades, secondary schools. And one of the problems there was that that's when COVID hit, year 2000.

And so teachers were expected to implement this at a time when schools were closed for about a month or so, and there was a bit of that chaos surrounding the COVID crisis. I'm not sure if I answered your question there, Andy.

Andy Psarianos
Yeah, it's okay. don't remember what my question was. But COVID was 2020, not 2000, just to be. Yeah, that's okay. That's all good. Yeah. Well, so, but interesting is taking up on this thread of coherence and the lack of coherence. So, if I understand correctly, I'll just regurgitate what you said.

So, the general feeling which launched this review was that the math curriculum was too focused on, let's say, know, procedures and memorization, right, to simplify it, and that we needed to look at, you know, sort of 21st century skills.

What do the kids need moving into the future? And it was a feeling that maybe that focus was a little bit misguided and that they needed to get this more into what I guess they call “big ideas” in the curriculum.

And with that seemed like it gave teachers a lot more freedom as to how they could teach and what they could teach and that kind of thing. So it was sort of slimmed down a little bit on the content side and, and more call it altruistic. I don't know if that's the right word.I don't want it to sound condescending, it's like altruistic sort of like higher, higher principle goals. But then that was given to the teachers. And then now you feel, you feel that maybe that's not enough. Maybe there's something missing. Maybe we need to do something more in BC.

Jerry Mussio
Well, the, and by the way, it wasn't just math, was right across the curriculum, right, okay. But certainly in math, say the whole issue, know, are we teaching procedures and we've done a good job at that, but are kids in fact being able to take those procedures and knowledge of mathematics and actually apply it in solving problems and dealing with, current issues that pop up in daily life, et cetera.

But the major difference in terms of the implementation of this redesigned curriculum, major difference has been that the ministry simply issued the new curriculum without providing a listing of teaching and learning resources that had been reviewed, curated by experienced teachers, which has been the case historically in the province.

This is the first time that a new curriculum has been issued without supporting resources that have been identified for teachers. So that's been my major criticism and criticism, well, coming from teachers themselves saying, well, this looks good.

We like how things are organized. got the big idea so we can understand the big picture here. We're not buried in details all the time without kids being clear on, where are we headed here on the big picture? We like that. translating that into practices in the classroom has been a huge issue, as you can imagine.

So that's, so when I talk about coherence here, that's one key element here in our experience here over the past six or seven years that the intents are honorable, look great, but in terms of actually delivery in classrooms, big challenge. And teachers have been scrambling.

What we have is sort of use of metaphor where we introduce a new electric car and arrives in the local dealership and here are mechanics that have been traditionally working gas engines and the dealership says that you're working with a new electric cars here but you've got to bring in your own tools. Figure that out. And that's what we have in the province right now in my view. That's the big challenge.

Adam Gifford
I think these challenges exist. no, but I think there's a couple of things that come into play. When you look at the English results, and there's that international decline post-COVID, but England wasn't quite as badly hit. But in reaction to that, and 2021 over here, or 2021, 22, we introduced a times tables test, Jerry.

So one of the arguments is that we weren't hit quite as badly in something in the latest round of PISA results, simply because of the approach that we've taken to teaching. We go into COVID, things drop. We then introduced something that is about the successes around instant recall, which to me is a huge backward step.

Because it's that same messaging to teachers. And I think this is what gets lost is what's the importance in the curriculum and how are you skilled to deal with it? if we give our teachers, right, part of your success story is asking your eight and nine year olds to recall multiplication facts as fast as you can. So it's a statutory test.

How are you going to deal with that whilst maintaining pedagogical integrity with the the idea behind the 2014 curriculum and then coming into what you're saying about if you're asking for these competencies to be developed but then asking like your analogy about the cars, I just really feel for teachers because they're being asked to do these things and A, it takes a long time to really embed new practice because you have to understand what outcome you're going for.

And I just find it remarkable and often it's a knee-jerk reaction. And I think more and more so, I think, I'm sort of going all over the place here with terms in PISA, where we react to these international tables and then think, right, we've got to do this, we've got to do this. And it's expected to be implemented very quickly, but at the same time, you know, not taking your workforce with you. And I think it's a huge issue.

Because you don't know where's the importance. What part am I focusing on my job to make sure I'm getting the important stuff right? And what even is the important stuff? That can get lost in amongst it.

Jerry Mussio
Yeah, yeah, was, and all the people, I don't know, quite a few of them who worked on this were very good. And there was a lot of consultations across the province in terms of the curriculum design, a lot of teacher input and so on.

So when I was doing my interviews and I was looking at what's going on, I asked the question, where are the learning resources for teachers? And the response was, well, that's a local responsibility now. In fact, there was a memo or policy memo went out to the system basically saying, we've decentralized this to the local level.

So it's up to school districts and we have about 60 school districts. So that's the second layer in the province. Each district then is responsible number of schools. So the responsibility goes down to districts, most of whom don't have the capacity to review materials. And so a lot of this gets basically moved on to the classroom teachers themselves.

So when I sort of started to learn about this, I asked the question from some of the leaders of the whole initiative about the resources. And they said, well, they can just find them on the internet. I said, well, have you been on the internet lately?

I mean, it's the Wild West, right? And so that's, to me, the major gap that exists in British Columbia right now between this curriculum, which in itself needs a lot of refinement.

So in terms of how content, for example, is laid out, been criticism saying mathematics, a lot of this assumes that some of these topics can be taught in any order you want, when in fact, and as we know, in some of those areas, you need prerequisite skills and a continuum of progress here to deal with them. I mean, some of that has to be cleaned up.

But the main gap here is the provision of materials for teachers so they can focus on time working with individual children, not staying up all night trying to figure out the new lesson plans for the next two weeks or whatever.

Andy Psarianos
It takes a long time. mean, you know, just, you can just, you know, mean, Adam and I were just talking about some mapping work that we're doing for New Zealand just before this. And yeah, I don't think, I don't think, you know, from helicopter, you, you kind of get a sense that, wow, you know, teachers know what they need to do and come on, you know, they're professionals and that's the bit of passing the buck really.

It's, it's difficult. It's really hard to draw a coherent, execution of a curriculum. it's really difficult. There's a lot of things to consider. And, and the reality is we know, we know teachers, they don't have the time. They just don't have the time, you know, because they've got to deal with the logistics of just

you know, being in the classroom in front of 30 children every day or whatever the number is, and, and, and just managing that whole process. Like that in itself is enough, right? Even if they know what they're supposed to teach and they have all the teaching materials with them, the planning to do the lesson is already more than enough work for a teacher.

And, and the other thing is we forget that this is a specialist job. It's not...

shouldn't be left to chance. Teachers don't, they're not trained on how to develop a curriculum or the execution of a curriculum, the sequencing of lessons.

You know, do you teach this first or that first? And for some topics and mathematics in particular, it's really important that you get that right because you, everything builds on previous, you know, it's a very complicated web of understanding and knowledge mathematics.

And, and you can't just jump in wherever you like and start from there because there's a good chance that it's just not gonna work. The children are not gonna learn. You could teach it. You can teach everything in the curriculum and nobody understands anything, right?

Because you did it in the wrong order. I mean, that's entirely possible with mathematics. It's not true necessarily of all topics, but for mathematics, that's definitely the case. So why are we leaving these things to chance? It's a good question.

But I think for politicians or for people who are too far separated from the day-to-day realities of actually teaching children so that they learn things, you know, it may look a lot easier than it actually is.

Jerry, thank you so much for joining us. That’s really insightful, so thank you.

Jerry Mussio
Okay, thank you.