‘5 weeks….5 books’ Part 3: Measuring up!

Teaching to the test can be good or bad: good if it means teaching a focused and aligned curriculum; bad if it reduces instruction to the memorisation of test items. (Baker 2004)

Slightly lagging behind with the old ‘5 weeks 5 books’ challenge! BUSY end to the term, recently I have written about some of the books that have been most influential on my educational thinking, to include the topics, Part 1: Leadership, Part 2: Teaching for effective learning, Assessment, Learning and Curriculum.

In this post I shall consider Assessment and Measuring Up: What educational testing really tells us by Daniel Koretz, Professor at Harvard Graduate School of Education. Koretz gives an excellent summary of the problems associated with testing, his work is largely applicable to the US context and the ‘No child left behind’ (NCLB), but could pertain to any education system. Chief among the disputes that interest Koretz are those that come from NCLB’s use of accountability, including the relationship between high-stakes testing and inflated score gains.

Within the US NCLB has been criticised as “a failed policy initiative” because higher scores on state tests aren’t transferring to other tests such as the SAT or National Assessment of Education Progress. The same could be said of the UK with large gains made in GCSE grades A*-C but this success not relayed in PISA scores, where the UK fails to make the top 20 in Maths, Reading and Science.

Three myths

Koretz attempts to debunk three common misconceptions when it comes to educational testing.

That test scores alone are sufficient to evaluate a teacher, a school, or an educational programme.

Not everything teachers do is measurable right away, and teacher quality is not necessarily causally related to improved student learning. Some of the seeds a good teacher plants may not germinate for years. A teacher can teach to the highest level of capability, and a particular student still not get it on test day.

That you can trust the often very large gains in scores we are seeing on tests used to hold students accountable.

That alignment is a cure-all – that more alignment is always better, and that alignment is enough to take care of problems like inflated scores.

Koretz explains that to a point, alignment is a good thing: we want clarity about goals, and we want both instruction and assessment to focus on the goals for our curriculum standards.

However, no matter how well aligned, tests are small samples from large domains of performance. That means that most of the domain, including much of the content and skills relevant to the standards, is necessarily omitted from the test. Under high-stakes conditions, there is a strong incentive to focus on the sampled content at the expense of the omitted material, which causes score inflation.

A further problem is predictability. To prepare students in a way that inflates scores, you have to know something about the test that is coming this year, not just the ones you have seen in the past. The content, format, style, or scoring of the test has to be somewhat predictable. And, of course, it usually is, as anyone who has looked at tests and test preparation materials should know. Carried too far, alignment actually makes this problem worse, by focusing attention on the particular way that knowledge and skills are presented in a given set of standards.

Why is this bad? Because many of those specifics are not relevant to the students’ broader competencies, long-term well-being and the overall goals of education. “Scores on a test are a means to an end, not properly an end in themselves.” (Koretz) Education should provide students knowledge and skills that they can use in later study and in the real world. Real gains in achievement require that students can perform well when confronted with “unfamiliar particulars.” Improving performance on the familiar but not the unfamiliar is score inflation.

Summary-Problems with testing.

Test scores reflect a small sample of behaviour and are valuable in so far as they support conclusions about the larger domains of interest
High stakes testing (league tables, targets, reliance on SAT scores, GCSE A*-C as indicators of school quality) leads to discrepancies in results and are a misleading indicator of what students have actually learned. Teachers ‘teach to the test’, which leads to score inflation, focusing their instruction on material emphasised by the test at the expense of other important aspects of the curriculum.
Performance will be weaker when students take another test that places emphasis on different parts of the domain.
Higher test scores do not reflect real improvements in achievement. Scores on a new test start out low, but show rapid gains over the next several years, eventually levelling out. If a different test is now introduced, the pattern repeats, and over time the scores seesaw up and down. When improvements in teaching and learning are real, and test questions are representative of the curriculum, scores should go up at a reasonable rate and remain high when tests change.

Is ‘teaching to the test’ always wrong?

I think the answer to this question depends on your definition of ‘teaching to the test’. If tests are written with our curriculum in mind, then teaching to the test means teaching students the knowledge and skills we agree they ought to learn, exactly what teachers are legally and ethically/morally obliged to do. Using exam board past papers and examiner’s reports as a guide to learning. Teachers are unaware of external exam questions, so this approach is not teaching to the test but teaching to agreed curriculum priorities.

Teaching to the test is only harmful if through excessive preparation, the focus is more on the format of the test and test-taking techniques than on the subject matter.

When students spend some time preparing for tests, they learn valuable skills. Time management, reading comprehension, following directions, knowing when certain answers can be eliminated, these are all important test-taking skills that students need to know as they progress through school and their future career paths.

There’s obviously a need for both teaching the skills necessary to succeed on tests, and preparing students to think and engage in learning.

Further reading:

I can highly recommend the recent blog posts of Daisy Christodoulou @daisychristo on Assessment:

Assessment is difficult but it is not mysterious

Assessment alternatives 1: Using questions instead of criteria

Assessment alternatives 2: Using pupil work instead of criteria

Also this post by Ruth Powley @powley_r, is an excellent summary of recent posts on the issue:

Ronseal Asssessment Part 2: Does what it says on the tin

Here is an archive I have put together of Assessment posts:

Assessment after levels: In search of the holy grail

Next 5 weeks, 5 books post: Learning

Seven myths of education: Daisy Christodoulou