Professional Voice 13.3.4

The decline and fall of NAPLAN’s objectives

John Graham

Prelude

In 2008, in an attempt to highlight the virtues of a new Australian national testing regime known as NAPLAN, and a new consumer market mechanism for schools, known as My School, the New York City schools Chancellor Joel Klein paid a visit to Australia sponsored by the Swiss multinational investment bank UBS.

The Minister for Education at the time, Julia Gillard, had invited Joel Klein to Australia to “share his reform experience” with the Australian public. Whether Ms Gillard was fully aware of Klein’s record during his eight years in charge of New York’s education system, or was bamboozled by his PR on her visit there, is unclear. However, by the time he arrived in Australia his claims to having achieved huge gains in achievement and equity in New York schools were subject to growing scepticism.

At the centre of the Klein reform recipe was high stakes standardised population testing. This was used to grade schools, teachers and principals. The consequences of low scores included school closings and the replacement of principals and other school staff and performance-based pay. As a result, schools narrowed their curriculum, spent more and more time on test preparation (including scripted curricula) and in some cases manipulated student scores.

Klein’s assertion that his reform strategy was a roaring success and had led to record gains in his administration’s standardised population literacy and numeracy tests was shown to lack any real substance when the American nationally-administered sample-based tests showed that little or no progress had been made between 2002, when Klein came into office, and 2009.[1] Achievement was static or declining and the equity gap for disadvantaged students showed little change.

In other words, the high stakes decisions made by the Klein administration in rewarding or punishing schools and school staff using standardised test results were not only flawed in theory and research, but were also shown to be based on defective standardised test data. One year later in 2010 Klein left his position as Chancellor and went to work for Rupert Murdoch selling schools tablets and a tablet-based approach to curriculum and instruction.

Achievement, equity and visibility

NAPLAN was introduced into Australia in 2008 as a cornerstone of the Rudd-Gillard ‘Education Revolution’. This linked increases in school funding for the states and territories and non-government authorities to education ‘reform’ through a National Education Agreement and National Partnerships in teacher quality, literacy and numeracy, and low SES communities. The additional investment was wrapped around standardised testing and the publication of its outcomes on a consumer website comparing school results. it was claimed that these carrot and stick measures would improve student achievement and equity.

The idea behind this claim, borrowed from New York and other American education jurisdictions, was that greater transparency and accountability would kickstart improvement in a complacent education system. Schools, classrooms and teachers would become more ‘visible’, and improvement would be achieved through “…practices of ‘auditing’ schools and teachers through the production of (largely) quantitative data, and to the creation of systems that use data to steer or manage institutions, individuals and practices at a distance.”[3] Using NAPLAN as the school quality metric on My School would force ‘underperforming’ schools to face a market mechanism (‘… if some (parents) walk with their feet that’s exactly what the system is designed to do …’Kevin Rudd, 2008[4]) with consequences for the viability of the school, its teachers and its principal. The new ‘NAPLAN culture’ would galvanise schools to try harder to improve their achievement.

Achievement outcomes

There is little evidence that the introduction of NAPLAN, and the publication of its results on My School, have in any way fulfilled the stated original purpose of improving student achievement and equity across Australia.

One of the ways of calculating the achievement progress of Australian students during the NAPLAN years is to examine the attainment of the country’s 15 year-olds in the sample-based international PISA testing program held every three years. Geoff Masters, the CEO of ACER, has contrasted the emphasis on low level basic skills in NAPLAN testing with PISA’s assessment of the ability to transfer and apply learning to new situations and unseen problems. “This requires an understanding of fundamental concepts and principles, as well as the ability to think. It is in these areas that Australian 15-year-olds’ performances are declining.”[5] The central role of NAPLAN, according to Masters, promotes an undesirable focus in schools on basic skills rather than the high-level capacities which PISA evaluates and which students will increasingly require in their post-school lives.

Between 2009 and 2015 the mean performance score in reading for Australian students in PISA steadily declined from 515 in 2009, to 512 in 2012, to 503 in 2015. In 2018 it remained at 503. In contrast, the mean performance score in reading for the OECD as a whole between 2009 (491) and 2018 (487) saw very little change.

In mathematics the mean performance score for Australian students steadily declined from 514 in 2009, to 5004 in 2012, to 494 in 2015 and to 491 in 2018. As with reading, the mean performance score in mathematics for the OECD as a whole between 2009 (496) and 2018 (494) saw very little change.[6]

The results also show between 2009 and 2018 an increase in both reading and mathematics in the proportion of Australian students who were classified as ‘low performers’ - this means they did not reach Level 2 out of the six PISA proficiency levels. In 2009 there were 14 per cent of low performers in reading. This rose to 20 per cent in 2018. In 2009 in mathematics 16 per cent of students were classified as low performers. This rose to 22 per cent in 2018.

The student achievement gains in NAPLAN itself have been inconsistent, insignificant and limited to certain areas. Many of the assessment domains show no improvement. As the Gonski Institute reported in its submission to the 2019 review of aspects of NAPLAN:

Significant gains are only seen in Years 5 and 9 numeracy, Years 3 and 5 reading, Years 3 and 5 spelling, and Years 3 and 7 grammar, all of which are significantly above the NAPLAN 2008 average. Thus, many year groups and domains show no significant progress. In particular, significant and consistent declines are evident in writing test results across Years 5, 7 and 9, since the first year writing was assessed in 2011. It is fair to say that the learning outcomes measured by NAPLAN have not improved over time. In some areas, like writing, they are disturbing.[7]

 Equity Outcomes

The flaws in the idea that a NAPLAN – My School strategy could address equity were obvious to all but its backers from day one. Alan Reid has pointed out that extending the education market and improving equity are incompatible polices.

Education markets lead to greater segregation and exacerbate achievement gaps in schooling. They provide an illusory choice for many, and inevitably residualise public education by leaving public schools with the largest numbers of students in need of special attention, and thus to do the heavy lifting on behalf of all schools. This is not an argument against accountability. It is an argument against forms of accountability that reduce quality and widen inequality.[8]

The cumulative evidence from more than a decade of NAPLAN testing shows no improvement in equity outcomes for Australian school students. Students who have been through four cycles of NAPLAN over time and participate as 15-year-olds in the OECD PISA sample-based testing program in reading, maths and science, display the same poor equity outcomes which existed ten years earlier.

The 2018 PISA mean score differences between high and low SES students and between Indigenous and non-Indigenous students represent differences of two to three and a half years of schooling, roughly the same as in 2009. The OECD labels students who have not reached the baseline level of proficiency required to participate fully in modern society as “low performers” (below Level 2 of the six PISA proficiency levels). The proportion of low performers in each category - SES and Indigenous/non-Indigenous – also display a lack of any change over ten years of NAPLAN. (see Table 1).

Table 1: Australian students’ PISA performance 2009 to 2018: SES quartiles, Indigenous/non-Indigenous[9]

When the Grattan Institute analysed NAPLAN results in 2016 it found that the gap between the performance of students from disadvantaged backgrounds and those from more advantaged circumstances significantly widened as students proceeded through school: when capabilities are similar in Year 3, disadvantaged students fall between 12 months and 21 months behind more advantaged students by Year 9; many regional and rural students make up to two years less progress than students in inner city areas between Years 3 and 9; students in disadvantaged schools make around two years less progress between Year 3 and Year 9 than similarly capable students in high advantage schools; high achievers in Year 3 make about two-and-a half years less progress by Year 9 if they attend a disadvantaged school rather than a high advantage school; high achievers in disadvantaged schools make less progress than low achievers in high advantage schools over the six years.[10]

Visibility outcomes

NAPLAN, like its New York template, has become the major yardstick to measure the quality of Australian schools. The media (publicly) and education departments (under the cloak of school improvement strategies) have used NAPLAN results to create school league tables. High performing schools are those that do well in NAPLAN and low performing schools are those that do poorly on this same measure, and the ‘hard evidence’ of school improvement is increases in NAPLAN scores. For central education bureaucracies, NAPLAN scores become the light switch that can illuminate the quality of learning at each school. The performance of every school can be cut and sliced and summed up under the same performance microscope and through the same data analysis software. Schools, and by implication their principals, teaching staff and students, become ‘visible’ locally, regionally, at a statewide level and nationally through a single annual literacy and numeracy basic skills testing program.

Outside of the political and bureaucratic bubble, the shortcomings of this form of educational ‘visibility’ have been shown to be many and varied. Margaret Wu has written extensively about the misuse of NAPLAN as a valid and reliable assessment of student and school performance. Statistically NAPLAN scores contain large margins of error and as such “do not provide sufficiently accurate information on student performance, student progress or school performance”.[11] There is also a substantial body of research identifying the curriculum and pedagogical distortions created in schools and classrooms by high stakes standardised population testing such as NAPLAN. When the rise and fall of basic skills test scores become the official metric for school quality, more educationally important measures such as a balanced curriculum and the motivation and wellbeing of students as learners lose their precedence. Teaching-to-the-test has been rife, and often implicitly sanctioned by authorities, so any improvements in NAPLAN scores are as likely to be due to an increase in test-taking capacity as an accurate picture of student gains in literacy and numeracy.

Warnings about teaching to the test often go unheeded because the stakes associated with NAPLAN’s status as the fundamental measure of school quality remain and encourage an excessive focus on the tests, which in turn distorts the measure. “Put another way, the higher the stakes, the more likely it is that the construct being measured has somehow been changed. High stakes, therefore, lead inexorably to invalidity” (Nichols & Berliner, 2007, p. 1)[12]. Hargreaves and Braun identify risks of an obsession with test-based data giving rise to Campbell’s Law, where test results themselves become the goal of education rather than acting as indicators of educational progress.[13]

Campbell’s Law’ was created by Donald T. Campbell the American social scientist and states:

The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

In 1976 Campbell applied the ‘law’ to standardised testing and stated:

Achievement tests may well be valuable indicators of general school achievement under conditions of normal teaching aimed at general competence. But when test scores become the goal of the teaching process, they both lose their value as indicators of educational status and distort the educational process in undesirable ways.[14]

The NAPLAN spotlight makes the standardised test results of schools and their students visible at the expense of leaving other more important measures of school quality and learning progress in the shadows. In 2020 when COVID switched off the NAPLAN light the various education bureaucracies, rather than using these circumstances to re-evaluate their addiction to standardised population testing, began scrabbling around to find a NAPLAN-lite alternative.

The alternative to NAPLAN

In September 2020 the AEU published the results of a national survey of over 12,000 teachers across Australia. The views of the teaching profession about the value of NAPLAN were overwhelmingly negative. 75 per cent of teachers did not believe NAPLAN is effective for measuring school performance or for school comparisons. 85 per cent did not believe that NAPLAN improves student outcomes and 94 per cent indicated that they believe it contributes to student stress and anxiety. NAPLAN was not only seen by the teaching profession as not contributing to student learning progress and affecting the wellbeing of students, but there was a frustration that its political and bureaucratic primacy was diverting attention away from what is able to do this and wasting valuable classroom teaching time.

The alternative to NAPLAN is a fit-for-purpose assessment system which has as its first design principle not market-based data or bureaucratic visibility but the improvement of education outcomes for all students. Research indicates that such a system should be curriculum-based, incorporate the professional expertise of the classroom teacher and have a diagnostic and formative orientation - in other words the antithesis to NAPLAN. The basis of a valid and reliable classroom-based system of assessment is already there, corresponding to the quality practice that can be found in schools around Australia outside of the NAPLAN spotlight. The new system would provide school communities and education authorities with authentic and quality information about student achievement and progress and, at the same time, the legitimate needs of system self-monitoring for equity and policy purposes could be met by scientific sampling methods which can provide accurate and useful information without any of the negative outcomes of mass standardised testing.

The new post-NAPLAN assessment system designed to improve achievement, equity and wellbeing outcomes for students would be supported by:

• Technology-based resources to: make ‘visible’ and disseminate through the public system quality school assessment practices; provide free and accessible online assessment materials linked to the curriculum for classroom use; improve reporting systems for school community and departmental use;
• Conditions of teaching and staffing resources providing the time for teachers to address the broad spectrum of individual needs in government schools and meet community expectations about school outcomes for all students;
• Restoring trust in the capacity of the teaching profession as experts in developing student learning and the reliable and valid assessment of that learning;
• A new funding deal for public schools from the federal government to energise school improvement in the wake of the failed NAPLAN culture. Australia is presently one of the most segregated education systems in the world. According to the OECD, Australia has the biggest proportion of disadvantaged children going to disadvantaged schools compared to any other country.[15]

What the OECD is now saying is that when equity doesn’t improve, improving the quality of learning outcomes becomes very difficult. That’s why I say that, for Australia, investing heavily in improving equity probably will be the best way to improve the learning outcomes for everyone in the system and make the country better as an education nation.[16]

References

[1] Medina, Jennifer (2009), US math tests find scant gains across New York, New York Times, October 14. 

[3] Thompson, Greg and Harbaugh, Alan (2013), The Effects of NAPLAN: Teacher perceptions of the impact of NAPLAN on pedagogy and curriculum, AARE

[4] https://pmtranscripts.pmc.gov.au/release/transcript-16090

[5] Masters Geoff (2019), Focus on basics leaves schoolkids short in essential deep thinking, Research Developments, ACER, 4 December 

[6] PISA statistics from: Thomson Sue, De Bortoli Lisa, Underwood Catherine, Schmid Marina (2019), PISA 2018: Reporting Australia’s Results, Volume I Student Performance, ACER 

[7] Gonski Institute for Education: Submission to the Education Council of the Council of Australian Governments (COAG) NAPLAN Reporting Review 2019, Uni NSW 

[8] Reid Alan (2009), Is this a revolution?: A critical analysis of the Rudd government’s national education agenda, ACSA 

[9] Ibid ACER PISA (2018) and Thomson Sue, De Bortoli Lisa, Nicholas Marina, Hillman Kylie, and Buckley Sarah (2011), Challenges for Australian Education: Results from PISA 2009, ACER 

[10] Goss P, Sonnemann J, Hunter J, Chisholm C and Nelson L (2016), Widening gaps: What NAPLAN tells us about student progress. Grattan Institute Report No. 2016-3, Grattan Institute, p.2 

[11] Wu, M (2010), Inadequacies of NAPLAN results for measuring schools performance. Submission to NAPLAN Senate Enquiry. Submission 208  For further critiques of NAPLAN BY Professor Wu: Wu M. (2010) The inappropriate use of NAPLAN data. Professional Voice 8: 21-25. Wu M. (2016) What can national testing data tell us? In: Lingard B, Thompson G and Sellar S (eds) National Testing in Schools: An Australian Assessment. Abingdon: Routledge, 19-29

[12] Nichols, S. L., & Berliner, D. C. (2007). Collateral damage: How high-stakes testing corrupts America's schools. Harvard Education Press.

[13] Hargreaves, A. & Braun, H. (2017). Data-driven improvement and accountability, National Education Policy Center.

[14] See the two quotes and an extended argument about the relevance of Campbell’s Law to standardised testing 

[15] Pasi Sahlberg, (2019) Interview, Professional Voice V.13 N.1, AEU(Vic)

[16] Ibid

John Graham is editor of Professional Voice and works as a research officer at the Australian Education Union (Vic). He has been a secondary teacher, worked on national and state-based education programs and in the policy division of the Victorian Education Department. He has carried out research in a wide range of areas related to education and training. He has had particular responsibility for the many issues impacting on teachers and teaching as a profession, teacher education, curriculum change, and the politics, organisation and funding of public education.

This article appears in Professional Voice 13.3 The new basics.