Part 1: 1st Generation: The Product Approach

Assessing the effectiveness of educational programs is a complex and elusive task. As a Teacher Educator I have considered many ways to answer the question: How effective are our courses? In this four-part series I am going to explore the evolution of educational evaluation. Understanding this evolution involves exploring various perspectives that have shaped evaluation practices within the broader context of education. Key distinctions emerge between quantitative and qualitative approaches, democratic and non-democratic processes, as well as scientific and naturalistic methodologies. Additionally, evaluations can be oriented towards action or understanding. This historical trajectory reveals four distinct generations of evaluation.

1st Generation: The Product Approach

If we wanted to measure how effective archers are at shooting an arrow, we would measure the distance from the arrow to its intended target. This is essentially the basic premise behind a product (or goal driven) approach to evaluation, comparing intended outcomes ‘goals’ with actual outcomes: this simple logic has had a tremendous influence on educational and curriculum evaluation.

This approach historically aligns evaluation with the scientific, or positivist paradigm, particularly in measuring the effectiveness of educational curricula in achieving stated objectives. Many evaluation studies in English Language Teaching (ELT) over recent decades have adhered to this goal-driven approach. The current emphasis on identifying teacher competencies and measuring development based on a range of skills, abilities, and knowledge represents an evolution of this approach. Typically conducted by external evaluators according to predefined criteria, these studies primarily aim at ensuring accountability.

However, a significant limitation of the goal-driven approach is its inability to provide insights into the reasons behind the outcomes. If we go back to our archer’s analogy, supposing all the arrows missed their target. Well, we would know that the archers were not meeting their goals but not why. Similarly, if all the arrows were hitting the bull’s eye does this mean the archers are exceptional or that the target should be set at a more ambitious distance further away. In other words, there could be a strong temptation to make the targets more achievable than to improve overall programme effectiveness.

 Critics of the product approach to evaluation argue that in the realm of education, the focus should be on people rather than parameters. While positivist methods excel at quantifying and measuring achievement, they often fall short in explaining the underlying dynamics of educational processes. In short, the product approach fails to provide a holistic understanding of educational practices.