Sue Thomson
The data released every three years from the OECD on results from the Programme for International Student Assessment (PISA) is mostly reported in the press as rankings and ‘who beat who’. However, it is not particularly helpful to use an overall country mean to determine where we need to direct our attention to improve learning, particularly in a country like Australia where the primary responsibility for education lies at the state and territory level. Instead, we need to carefully disaggregate the data and consider, among other things, the social and economic factors that influence performance across states and between schools.
Socioeconomic background
The relationship between socioeconomic background (SES) and student achievement is well-established, with a vast body of literature showing that more advantaged students tend to do better in school than disadvantaged students. Also well-established is the relationship between a school’s socioeconomic background, defined as the aggregated measure of the socioeconomic backgrounds of the students within the school. In all PISA studies, in most countries, academic outcomes tend to be more strongly associated with school SES than with individual students’ SES, although both are important. Figure 1 shows the relationship between science achievement and student socioeconomic background (SES) for Australian students in PISA 2015. The difference between the average low SES student and the average high SES student is 91 score points. This represents almost three years of learning.
Figure 1. Relationship between student socioeconomic background and science achievement, Australia, PISA 2015
However, we know that not all schools are the same – and that not all children attend schools that reflect their own socioeconomic background. Higher SES students still come to school with a higher level of resources than low SES students do, both in terms of physical resources and in less tangible resources such as parental support and involvement. Thus it would be helpful to look at the achievement of students in three broad groups of schools: low, average and high average SES1.
Figure 2 shows the relationship between science achievement and socioeconomic background at both student and school level. Clearly the school a student attends matters. For an average disadvantaged student, the difference between attending a disadvantaged school versus an advantaged school is 56 score points. For the typical average achieving student, the difference between attending a disadvantaged school versus an advantaged school is 64 score points. For both groups, this average is statistically and substantially below the OECD average, and would place these students’ achievement with that of students from far less developed countries.
For advantaged students there is a 71 point penalty for attending a disadvantaged school, bringing average achievement down from around the average of the highest performing country, Singapore, to that of the OECD average.
Figure 2. Relationship between student and school socioeconomic background and science achievement, Australia, PISA 2015
The message from this analysis of the PISA 2015 data for Australia seems clear and consistent. For Australian students both student- and school-level SES matter consistently and substantially in the academic performance of students in science. The disaggregation of the PISA 2015 data for Australia shows unequivocally that increasing student and school group SES are strongly associated with more positive academic outcomes.
Are there differences between disadvantaged students and advantaged students, and between those in advantaged and disadvantaged schools, on other outcomes of school, non-cognitive areas such as enjoyment of science, for example?
Students’ level of enjoyment of learning science was derived from their level of agreement with the following five statements, measured on a four-point scale (strongly disagree; disagree; agree; strongly agree):
- I generally have fun when I am learning science topics
- I like reading about science topics
- I am happy working on science topics
- I enjoy acquiring new knowledge in science
- I am interested in learning about science.
The five items were standardised to create an index of enjoyment of learning science. Positive values on this index indicated higher levels of student enjoyment of science. Students in Australia had a mean index score of 0.12, which was significantly higher than the OECD average of 0.02. However, averages can mask big differences. Figure 3 shows the index of Enjoyment of science, by student and school socioeconomic background.
As in Figure 2, Figure 3 shows that there are substantial differences on this one measure between students in different schools. Disadvantaged students in disadvantaged schools have the lowest levels of enjoyment of science. Their average score was significantly lower than the OECD average, and significantly lower than that of advantaged students in similar disadvantaged schools. However, disadvantaged students in advantaged schools had a level of enjoyment of science that was not only significantly higher than the OECD average, but was not significantly different to that of the advantaged students within the same type of school. Interestingly for socioeconomically advantaged students, there was no significant difference between their score on this scale between those in disadvantaged schools and those in advantaged schools.
Figure 3. Relationship between student and school socioeconomic background and enjoyment of science, Australia, PISA 2015
School climate
From the previous analyses, it seems clear that schools matter. What is it about disadvantaged schools that make it difficult for students to learn? PISA can also provide some insights into this.
Student and teacher behaviour hindering learning
To examine the impact of student behaviour factors on school climate, principals were asked to report the extent to which the learning of students was hindered by a series of 10 items using a four-point scale (not at all; very little; to some extent; a lot). Using these items, two indices were constructed, the first was an index of student behaviour hindering learning and the second an index of teacher behaviour hindering learning.
The index of student behaviour hindering learning comprised the following five items:
- student truancy
- students skipping classes
- students lacking respect for teachers
- student use of alcohol or illegal drugs
- students intimidating or bullying other students.
Principals’ responses to these questions were likely to reflect both how frequently these phenomena happen in their schools and, when they do occur, how much they affect student learning. Positive values on this index reflect principals’ perceptions that student behaviour hinders learning to a greater extent, and negative values reflect principals’ perceptions that student behaviour hinders learning to a lesser extent than, on average across the OECD. In Australia, there was a moderate negative relationship between student behaviour hindering learning and scientific literacy performance (r = –0.28). Higher scores on the index were reflected in lower scores on science achievement.
Figure 4 shows the proportion of students in disadvantaged and advantaged schools whose principals responded ‘to some extent’ or ‘a lot’ to each of the items.
It is evident from this figure that there are massive differences in the extent that principals perceive there to be student behavioural issues within disadvantaged schools that compromise student learning. Student truancy was reported by the principal as a problem for about 45 per cent of the students who attended disadvantaged schools, compared to just 3 per cent of those who attended advantaged schools. Similarly, students skipping classes is an issue for principals in about half of the disadvantaged schools, but in just 2 per cent of advantaged schools. The proportion of students lacking respect for teachers is also of concern, with this being identified as an issue in 43 per cent of disadvantaged schools but just 2 per cent of advantaged schools. The use of alcohol and drugs, and the amount of bullying at a school, were more prevalent than the other student behaviours at advantaged schools, but the differences were still substantial.
Figure 4. Student behaviours hindering learning
Teacher behaviour hindering learning
School principals were also asked to report the extent to which they believed that student learning in their schools was hindered by teacher behaviours. The index of teacher behaviour hindering learning was constructed with the following five items:
- teachers not meeting individual students’ needs
- teacher absenteeism
- staff resisting change
- teachers being too strict with students
- teachers not being well prepared for classes.
Again, items comprising this index were standardised to have a mean of 0 and a standard deviation of 1, with positive values reflecting principals’ perceptions that these teacher- related behaviours hinder learning to a greater extent compared to the OECD average. In Australia, there was a weak negative relationship between teacher behaviour hindering learning and science performance (r = –0.11). Higher scores on the teacher-behaviour index were reflected in lower average science scores.
Figure 5 shows the proportion of principals in disadvantaged and advantaged schools who responded ‘to some extent’ or ‘a lot’ to each of the items.
The differences between advantaged and disadvantaged schools are not quite so stark on this scale, however on all but one, a higher proportion of principals report that the behaviour hinders learning ‘to some extent’ or ‘a lot’.
The perception of teachers not meeting individual student needs is seen as an issue by about 25 per cent of principals at disadvantaged schools and 17 per cent of principals of advantaged schools. Staff resisting change is an interesting point of difference: in this case principals at advantaged schools see this as a problem to a greater extent than do principals at disadvantaged schools. Teacher absenteeism at disadvantaged schools is potentially a major issue. If teachers are frequently absent then continuity of teaching is compromised, particularly in subjects like mathematics and science. This makes it very difficult for students to learn, and more so if they are frequently absent as well.
Figure 5. Teacher behaviours hindering learning
School resources
Shortage of teaching staff and assistants
Access to school resources was examined by asking principals to report whether their schools’ capacity to provide instruction was hindered by a shortage of resources across eight items, with responses on a four-point scale (not at all; very little; to some extent; a lot). Using these items, two indices were constructed, the first an index of shortage of educational staff and the second an index of shortage of educational materials including physical infrastructure. The index of shortage of educational staff comprised the following four items:
- a lack of teaching staff
- inadequate or poorly qualified teaching staff
- a lack of assisting staff
- inadequate or poorly qualified assisting staff.
When interpreting these findings, it should be kept in mind that school principals did not provide an objective measure of the condition of educational resources, but rather they provided their perceptions of whether a shortage or inadequacy of educational resources hindered the capacity to provide lessons in their schools. Therefore caution is needed when comparing responses across countries and schools. In Australia, there was a moderate negative relationship between shortage of educational staff and scientific literacy performance (r = –0.18). A greater reported shortage of educational staff was reflected in lower levels of science performance.
Figure 6 provides the responses from principals on these items. On every item comprising this index, students who attended disadvantaged schools were more likely to have principals who reported that the statement was an issue compared to those from more advantaged schools.
Figure 6. Shortage of staff
Shortage of educational materials including physical infrastructure
A further aspect of school resourcing specifically related to physical infrastructure and supply of educational resources was explored, as the absence of such resources could negatively affect student learning. The second index in this section, the index of shortage of educational material including physical infrastructure, was constructed using the following items:
- lack of educational material (e.g. textbooks, IT equipment, library or lab material)
- inadequate or poor quality educational material (e.g. textbooks, IT equipment)
- lack of physical infrastructure (building, grounds, heating/cooling, lighting)
- inadequate/poor quality physical infrastructure (building, grounds, heating/cooling).
Positive values on the index reflected principals’ perceptions that the shortage of educational material hindered learning to a greater extent than the OECD average. In
Australia, there was a weak negative relationship between a shortage of educational materials and science performance (r = –0.14). A greater reported shortage of
educational materials including physical infrastructure was reflected in the lower levels of science performance.
Figure 7 shows the proportion of principals in disadvantaged and advantaged schools who responded ‘to some extent’ or ‘a lot’ to each of the items.
Figure 7. Shortage of materials and infrastructure
As may be expected, resources were a key issue for students in disadvantaged schools – the principals of 41per cent of these students reported that a lack of, or inadequate, educational material limited their capacity to provide instruction, while principals of 25 per cent of students in disadvantaged schools reported that poor quality or a complete lack of physical infrastructure hindered their capacity to provide instruction.
Conclusions
These findings show that where one goes to school in Australia makes a significant difference for students’ science performance. This is inequitable because it means that a student’s achievement is heavily influenced by his or her family’s ability to afford a good school. If Australia is to turn around the decline in PISA scores, it has an opportunity to do so by lifting the performance of our lowest achieving students. To do so makes sense on more grounds than equity, or it being ‘the right thing to do’. Economists have argued that increasing Australia’s average PISA score by 25 points, for example, would provide an increase of more than $3 million to its GDP (OECD, 2010, for example).
We need to ensure that education provides a level playing field for all students. While schools cannot compensate for a disadvantaged background, we can ensure that all schools have adequate resources – both physical and in terms of teaching staff to teach properly. Students cannot feel motivated to learn if they feel they are not valued by the system.
Notes
1 The school-level SES is calculated as the average SES of all PISA students attending the school. Using the data from all participating schools, quartiles were formed. The lowest quartile corresponds to ‘low SES’ or disadvantaged schools, the highest to ‘high SES’ or advantaged schools, the remaining middle quartiles were labelled as ‘average’.
Sue Thomson is the Director of the Educational Monitoring and Research Division and Research Director of the Australian Surveys research program at the Australian Council for Educational Research. She is also the National Research Coordinator for Australia in the Trends in International Mathematics and Science Study (TIMSS), the Progress in International Reading Literacy Study (PIRLS), and the National Project Manager for Australia for the OECD Programme for International Student Assessment (PISA). Prior to joining ACER Dr Thomson worked as a Mathematics and IT teacher in government secondary schools in Victoria.
This article appears in Professional Voice 12.1 Professional learning.