Rethinking Evaluation of Student Learning

 

Patricia Cranton

University of New Brunswick

 

Abstract

 

This paper presents a critical examination of strategies for the evaluation of student learning. A perspective on evaluation is proposed which takes into account Habermas’s three kinds of knowledge: instrumental, practical or communicative, and emancipatory. The research paradigms corresponding to each kind of knowledge are used as a starting point for thinking about the types of evaluation of learning strategies that might be appropriate. It is suggested that objectively-scored techniques are suited to the assessment of instrumental knowledge, interpretive techniques to the evaluation of communicative knowledge, and subjective self-evaluation to the understanding of emancipatory knowledge.

 

Introduction

here may be no other everyday issue with which faculty struggle as much as evaluation of student learning. We want to be fair to students. We try to be objective. We want grades to accurately reflect what students know. We understand that students compete for grades and are motivated by marks. We realize that students rely on grades as an entry to further studies, jobs, or career advancement. Our classes are often large, and we feel forced to use quick-to-score multiple choice tests even while we doubt their validity.

There are no clear-cut solutions to these issues. Evaluation of learning is a complex process that takes place within many different contexts and is conducted by individuals with widely varying values and philosophies of teaching. However, I believe many of us have fallen into certain mindsets regarding evaluation, limiting our ability to see alternatives and fresh perspectives.

In this paper, I suggest we rethink and critically examine some of our taken-for- granted assumptions about evaluation of student learning. I propose a perspective on evaluation that takes into account the nature of the knowledge we want our students to attain. Although this sounds quite ordinary, and like something we already do, I show that such a rethinking leads to reflection on several of our assimilated norms about evaluation. Perhaps objective is not always better than subjective. Perhaps there are even no truly objective ways to evaluate learning. Perhaps teachers are not always the best judge of what students have learned.

First, I use Habermas’s (1971) three kinds of knowledge as a starting point for this discussion. Habermas’s classic work on human interests and how knowledge about them is acquired has spawned a rich literature in education. I suspect his ideas can also guide us in reexamining such practical issues as how to evaluate learning. Second, I draw a parallel between evaluating learning and educational research methodologies as a way to spark new thoughts about evaluation. Third, I contemplate how these perspectives might translate into strategies we can use in higher education. This leads me to three kinds of evaluation: objectively-scored, interpretive, and subjective or self-evaluation.

 

Three Domains of Knowledge

 

abermas (1971) describes three basic human interests, each of which leads people to acquire a different kind of knowledge. One of his theses is that instrumental knowledge has become all-pervasive in modern society, thereby enticing us to look at everything through this lens. I think this may be true of how we view and value student learning. But before I discuss this possibility, I will briefly review the three domains of knowledge.

Our technical interests lead us to instrumental knowledge. Technical interests come from the need to control and manipulate the external environment so as to obtain, for example, shelter, food, transportation, and fuel. Instrumental knowledge is based on invariant causeand-effect principles. It allows us to predict observable events in the world. The knowledge is objective and empirically derived. Instrumental knowledge falls into the philosophical realm of positivism. Habermas’s distinction between invariant cause-and-effect relationships that are true in all situations and those that vary with social factors and changing conditions is important to note here. Because of the invariant laws of gravity, for example, a stone thrown in the air will always fall to the earth. Human learning, thinking, development, and emotions cannot be predicted in the same way, however much we would like it to be so.

Our practical interests lead us to practical knowledge or communicative knowledge, the latter term being one employed by Mezirow (1997, for example) in his work on adult learning. Mutual understanding of individual interests and needs is required for people to live together in a society and to coordinate social actions to satisfy those interests and needs. We have to live together and work together. In order to do so, we need to understand each other on both a simple personal level and on a larger social and political level. Practical knowledge is acquired through language, through communication with each other. It is a knowledge of the norms that underlie the society we live in, whether this be interpersonal relationships, groups, communities, organizations, cultures, nations, or the global society. Rather than truth in terms of invariant principles, as exist in instrumental knowledge, validity is determined by consensus within a group and a sense of “rightness” or morality. What is agreed-upon knowledge in one culture may not be valid in another culture. The philosophical foundation, therefore, for communicative knowledge lies in hermeneutics. Our justice systems, social systems, and political systems are based on communicative knowledge. Values related to family, work, individualism, and spiritualism are communicative in nature.

Our emancipatory interests are reflected in our desire to grow and develop, to obtain self-knowledge, and to experience freedom and relational autonomy. Emancipatory knowledge is acquired through critical reflection and critical self- reflection (Mezirow, 1991). Philosophically, the underpinnings lie in critical theory. We are constrained by uncritically assimilated norms, beliefs, and values. Social norms and social systems become reified, unquestioned, and unquestionable. When we absorb these norms and systems, we are oppressed or constrained in that we are not aware of alternatives. For example, as a young girl growing up in the 1950s in rural Western Canada, I assimilated the social norm that girls could become wives, mothers, teachers, or nurses. No other alternatives existed in the community nor in my mind. I was constrained by that norm. Our basic human drive for growth  can lead us to critically question such systems and thereby become free from the constraint of not knowing there are alternative perspectives.

In higher education, the acquisition of each of these kinds of knowledge are valid and accepted goals, though there is a tendency to value instrumental over communicative and emancipatory knowledge (for example, see Cranton, 1998). This is a remnant of the Age of Enlightenment when instrumental rationality was seen to be the solution to the world’s ills. A reverence for objective science remains with us still. It is in the sciences, of course, where we find the acquisition of instrumental knowledge to be a primary goal of higher education. The trades and technologies also focus mainly on instrumental knowledge in their programs. Communicative knowledge is the purview of the social sciences, humanities, and arts. As well, the service industries and helping professions seek to foster communicative learning to a large extent. Unfortunately, it is often the case that in these disciplines, researchers, theorists, and teachers strive to force knowledge into the instrumental domain in an attempt to give it greater value (for example, see Kincheloe, 1991). In the social sciences (note the assumption underlying the terminology we use), the search for invariant relationships still dominates a good portion of academic research and writing. Emancipatory knowledge essentially crosses disciplines. Most faculty advocate critical thinking, for example, which is the basis for questioning assumptions and beliefs. And, at least in our rhetoric, we aspire to student self-determination and autonomy. Self-determination includes both being reflective about cultural contexts and traditions and being articulate about our own values. Autonomy or empowerment is achieved through self-reflection on the subjective conditions of knowledge (Kant’s critique of knowledge) and reflection that frees the individual from the hidden constraints inherent in social structures (Marx’s critique of ideology). Self-determination and self-reflection at least touch all disciplines and may be seen as the primary goal in some programs.

 

Research Methodologies

 

he impetus for this essay came from comments I received from a reviewer of a chapter on transformative learning. He or she asked for empirical evidence (in the narrower empirical-analytical sense rather than the broader experiential sense) regarding transformative or emancipatory learning. This led me to reflect on how we think of evidence of student learning. It is quite clear that different research methodologies are required for obtaining instrumental, communicative, and emancipatory knowledge. It may seem to be an aside to bring research methodologies into this discussion when rethinking evaluation of student learning is the focus. However, higher education faculty are in tune with the way research strategies vary according to the question being addressed, and I see the acquisition of knowledge through research as parallel to the acquisition of knowledge we expect of our students. A brief examination of research paradigms may stimulate our thinking about evaluating learning.

Each scientific methodology is a valid means for knowing within its domain of knowledge. Instrumental knowledge is acquired through the empirical-analytical or natural sciences (Ewert, 1991). Quantitative measurement, experimental design, and the traditional scientific method for addressing research questions are employed. Research questions are generally of the forms, “Does A cause B?” “Is there a significant difference between X and Y?” “How do A and B vary together (or correlate)?” and “How frequently or where does X occur?” The goal of empirical- analytical research is to discover invariant relationships. We strive for correct or true answers to questions. Social factors do not influence the outcome. The researcher is objective; her opinions do not have an impact on results.

Communicative or practical knowledge is obtained through the hermeneutic or interpre- tive sciences (Ewert, 1991). Qualitative data are used to interpret and understand intersubjective meaning. Through interviews, observations, immersion in a context or situation, written materials, and conversations with others, researchers strive to understand human and social processes (for example, see Creswell, 1998). Research questions are generally of the form, “What is the nature of X?” “How do people understand Y?” and “How do individuals learn about X?” The goal is not to establish causality but to inquire into the meaning of things in the world. The researcher is an interpreter rather than an objective observer.

Emancipatory knowledge is obtained through the critical sciences (Ewert, 1991). Unlike

the empirical and interpretive sciences that describe the world as it is, the critical sciences critique the world with a goal of knowing how it should be. The purpose is enlightenment and emancipation. Self-reflection and self-development are explicit aims for the researcher working in this paradigm. The researcher therefore works with participants who are seen as coresearchers in a transformative process (for example, see Merrian and Simpson, 1995). Research questions take the form, “What is happening here?” “How did we come to this state?” and “How can we open up alternative perspectives and ways of acting?” Participants not only seek an understanding of a situation, but they also jointly make decisions about possible courses of action.

 

Parallels Between Research Paradigms and Evaluation Strategies

 

here is, of course, a fundamental difference between conducting research and evaluating student learning. Regardless of what is being learned, learning is influenced by social and psychological factors. Therefore it never falls entirely into the instrumental domain. If we understand this, though, I believe we can still use the parallels between research and evaluation to spark ideas about evaluation.

When our goal is to evaluate student learning in the instrumental domain, the nature of the knowledge allows us to employ quantitative, objective strategies, as it does when we engage in empirical-analytical research. When there is one correct answer to a problem or clearly correct procedures for completing a task, we can judge whether or not students have acquired the knowledge by their ability to produce the correct answers or demonstrate the correct procedures. We could argue that though students may learn in different ways or at varying speeds all that matters in the end is the quantifiable and observable learning outcome. This is a commonly-held view in many higher education programs. Psychological factors such as learning styles and social factors such as classroom dynamics are things to be dealt with in our teaching methods, not our evaluation strategies. I am not sure I would actually advocate this position, but it is one that follows logically from focusing on instrumental learning outcomes. It is worth considering, even as we also critically question it.

When our goal is to evaluate learning in the communicative domain, we are looking at quite a different picture. Using the hermeneutic or interpretive research paradigms as a model, we are still interested in describing what happens in the objective world, but we are now concerned with an interpretive understanding of what we see. There are no objectively verifiable truths. Validity is determined by consensus within a social group of informed persons. In this domain, student learning should then be evaluated by methods which allow freedom of expression and interpretation (essays, oral presentations, role plays, creative and artistic productions), interpersonal interactions (discussion, conversation, interviews), and flexibility in form and content. Attempts to quantify and objectify learning in this domain deny its very essence. This is a key point and one that follows naturally from thinking about the parallel research paradigm. In evaluating learning, we have traditionally striven for objectivity.  Yet, the knowledge itself is such that it needs to be interpreted so as to make meaning of it. As faculty, we are informed experts in our field, as is a researcher interpreting qualitative data. We can validly judge the meaning of students’ work.

When it is our goal to evaluate learning in the emancipatory domain, we are interested in how self-reflection and self-development are taking place. In the critical sciences research paradigm, participants (those we are striving to understand) are fully engaged as coresearchers. It is the person who undertakes self-reflection and self- development who can say that he or she is doing so. The researcher helps participants to understand meaning, interprets actions, challenges assumptions, and asks good questions. But participants must be involved for the research to take place as the process is wholly subjective. If we take apply these notions to evaluating students’ emancipatory learning, we are led to student self-evaluation. This is not to say that students can assign themselves marks or grades without dialogue and guidance, any more than research participants can address the research questions without the researcher, but they must be actively involved in the evaluation process in a way that promotes self-determination. If they are not, the goal of emancipatory learning is defeated.

 

Implications for Selecting Evaluation Strategies

 

raditional instructional design procedures include clarifying the learning objectives along with their domain and level of learning in order to best select testing methods. In one way, what I am suggesting here is similar, except in traditional instructional design the goal is always to find observable (quantifiable, empirical) indicators of learning. I recall some of the texts from the late 1960s and early 1970s that advised us to list observable indicators of students’ motives, attitudes, emotions, and values. This led to such bizarre statements as, “The student will voluntarily attend art gallery showings eight times out of ten and stay for at least one hour per visit,” as an indication of student interest in art. The word ‘subjective’ was a derogatory term. Although instructional designers had the right idea when they advocated matching evaluation strategies to the nature of learning, they then unfortunately forced everything into the instrumental mould. Under the influence of behaviourism, those were also the days when writers in education advised the use of task and procedural analyses of learning objectives, competency-based modules, standardized curriculum packages, and standardized test scores.

When instrumental knowledge is being evaluated objectively-scored assessment techniques are generally appropriate. By objectively-scored techniques, I mean those for which two people using a scoring key would come up with the same result. Multiple-choice, truefalse, matching, and some short answer formats are included. I deliberately say objectivelyscored rather than objective since the selection and formulation of the questions themselves involves subjective judgement. Some objectively-scored tests are very subjective indeed, reflecting the teacher’s perception of what is important in the course and his or her biases in terms of wording, weights on questions, and so forth. Unfortunately, when we quantify things such as student learning, we tend to give the numbers, the results, more power and precision than they should have simply because they are numbers and we have learned to value numbers.

The evaluation of communicative learning must be explicitly and openly interpretive, if we follow through from our earlier analogy of research paradigms. This is not a label found in the literature on evaluation of learning. I use the term interpretive evaluation techniques to include the familiar written essay, term paper, or report. Case studies and problem solving exercises are relevant in some disciplines. Student presentations, discussions, debates, and other forms of oral communication also fall into this category. The traditional thesis defence is an example of an oral evaluation strategy upon which we place great value in higher education. In some disciplines, student performance (drama, music, sports) or products created by students (art, architecture, engineering) are interpreted in order to evaluate learning.

This may seem to be nothing new. Do we not already use these methods regularly? I am suggesting that we use such techniques in a different way, or perhaps that the philosophy underlying their use be rethought. Many faculty using, for example, essays or term papers try to make the scoring as objective as possible. They count up points, subtract points for spelling errors, follow predetermined criteria, and still feel vaguely uneasy about the scoring. What about the original essay that meets few of the predetermined criteria? What about the student who has the courage to challenge existing viewpoints? What about the student who has made enormous strides in her thinking but has still listed all the main points? In nearly 25 years of academic life, I do not think I have met a faculty member who enjoys marking essays. This could be because we are forcing an interpretive procedure into an objective model. Don’t art and film critics enjoy judging the quality of the work they review? Perhaps this is due to the freedom they have to be openly interpretive.

Good interpretive evaluations are trustworthy and credible. In accepting this form of evaluation, we rely on the expertise, professionalism, and credibility of the teacher or any other judges who might be involved. We rely on negotiation, agreement, and consensus among individuals. Talk of “bias” is irrelevant. Our inclinations, beliefs, and values become a part of the evaluation process. Evaluations are based on our interpretations rather than detached from them. The trustworthiness of the evaluation is founded on the professionalism, experience, knowledge, authentic, and ethical behaviour of the educator.

To value interpretation in evaluation of learning challenges our assumptions. The behaviourists and instructional designers, and the pervasiveness of instrumental knowledge have been strong influences in education. To interpret does not mean to be random, unfair, or discriminatory against those students who have perspectives different from our own. In interpretive evaluation procedures, there should be agreement between faculty and students that the nature of the learning is communicative. There should be agreement as to what is being evaluated. The person doing the evaluation should be a subject expert with established and respected credentials. Evaluators must be ethical, caring, responsible, and open-minded. Most of us would agree that these conditions are not onerous. Most of us would also agree that interpretive evaluation is valid and credible in many other aspects of life. When we cannot accept it in the evaluation of communicative learning, we need to step back and question our basic assumptions.

The evaluation of emancipatory learning may require an even bigger stretch. In writing about transformative (emancipatory) learning, Mezirow (1991, pp. 219-220) makes a strong and challenging statement: “dogmatic insistence that learning outcomes be specified in advance of the educational experience in terms of observable changes in behavior or ‘competencies’ that are used as benchmarks against which to measure learning gains will result in a reductive distortion and serve merely as a device of indoctrination.” As we saw earlier, based on the parallel between research paradigms and evaluation models, student self-assessment needs to be central to the evaluation of emancipatory learning. Evaluation is subjective, of the self. If our goal is truly emancipatory, it clearly makes no sense to not allow student involvement in one of the most powerful aspects of the learning experience – judging its quality. Empowerment on a short leash is not empowerment at all. We cannot say, “Our goal is self-direction,

self-determination, and self-reflection, but I will judge how well you did.” Even though it may be well-meaning and most often based on our uncritically assimilated assumptions about the role of teacher, I would argue that this stance is hypocritical.

How can self-evaluation be incorporated into the higher education setting where students are often motivated by grades and competitive in attaining them? Several points can be made in response to this question. Perhaps we need to question the systems that have led to grade motivation and competition. But that is a topic for another time. For our purposes here, it is important to realize that self-evaluation is critical when the learning goal is emancipatory. I believe that subjective self-evaluation should be encouraged in all learning, but there are many occasions where this does not work, given the nature of higher education institutions. So, first, we need to be clear about when and how we are promoting emancipatory learning and promote self-evaluation in relation to those goals.

Good self-evaluation focuses on self-reflection, openness to alternative points of view, an increased awareness of underlying assumptions, participation in reflective discourse, and changes in long-established patterns of expectations and behaviours. Students know when they have learned in these ways. We may not.

But even so, it is not so simple as saying, “Student, evaluate thyself.” Students need to learn how to engage in self-evaluation. They may need help clarifying goals and how their learning relates to those goals. Students can help each other in this process, and faculty should also try to meet individually and privately with students to discuss their self-evaluation. Examples of how to judge emancipatory learning can be given. Students should be encouraged to validate their perceptions with another person—the teacher, a peer, a professional in the field. Students have well-established habits of mind about the teacher’s role in evaluating their learning. It may be as difficult for them to participate in evaluation as it is for many faculty to invite it. Students will say, “You’re the teacher – you tell me,” reflecting a whole web of assumptions about what teachers and students should do. It is our responsibility to challenge those assumptions.

 

Critical Questions for Reflection

 

 have suggested that an understanding of Habermas’s three kinds of knowledge and the research paradigms that are used to obtain those kinds of knowledge leads us to three quite distinct approaches to evaluation of students’ acquisition of knowledge. Objectively-scored techniques can be used to assess students’ instrumental knowledge. Interpretive strategies can be used to evaluate students’ communicative knowledge. Subjective self-evaluation is appropriate for understanding students’ emancipatory learning. Rather than formulating a conclusion, I would prefer to end this discussion with some questions that we may all take away and ponder as we go about our practice.

Is it possible to be truly objective in the evaluation of student learning which is, by definition, a psychological and social process? Even when the knowledge attained is instrumental in nature, should we take into account how students learn? How can we account for, in striving for objectivity, the personality, preferences, and values of the person who designs the evaluation method?

In the interpretive model of evaluation, can we still compare students? How do we translate evaluation results into the numerical or letter grades acquired by most institutions? Can we be sure that all faculty interpreting student work are indeed professional, responsible, ethical, and caring? What arguments can we present to our colleagues to persuade them that it is valid to interpret students’ work? Are interpretive methods appropriate only in the arts and social sciences, or are they also relevant at advanced levels in the sciences?  How can subjective self-evaluation be incorporated into institutional requirements? Can we expect honest and valid subjective self-evaluation from students when they are also competing for grades? How can we best prepare students to engage in self-evaluation? Do undergraduate students have the maturity and self- awareness to judge their progress toward emancipatory knowledge? If we advocate self-determination and critical reflection in higher education, how can we deny students the opportunity of self-evaluation?

 

References

 

Cranton, P. (1998). No One Way: Teaching and Learning in Higher Education. Toronto, Canada: Wall & Emerson.

Creswell, J.W. (1998). Qualitative Inquiry and Research Design. Thousand Oaks CA: Sage Publications.

Ewert, G. (1991). Habermas and education: A comprehensive overview of the influence of Habermas in educational literature. Review of Educational Research, 61, 345-378.

Habermas, J. (1971). Knowledge and Human Interests. Boston: Beacon Press.

Kincheloe, J. (1991). Teachers as Researchers: Qualitative Inquiry as a Path to Empowerment. London: Falmer Press.

Merriam, S.B., & Simpson, E.W. (1995). A Guide to Research for Educators and Trainers of Adults. Malabar, FL: Krieger Publishing Company.

Mezirow, J. (1991). Transformative dimensions of adult learning. San Francisco:

Jossey-Bass.

Mezirow, J. (1997). Transformative learning: Theory to practice. In P. Cranton (Ed.), Transformative learning in action: Insights from practice. New Directions for Adult and Continuing Education, no. 74. San Francisco: Jossey-Bass.