Revista Electrónica de Investigación Educativa

Vol. 6, No. 2, 2004

Evaluation of the Educational Relationship
in the University

Mario Rueda Beltrán
mariorb@unam.mx

Instituto de Investigaciones Sobre la Universidad y la Educación
Universidad Nacional Autónoma de México

Privada Cariaco #28 Casuarina No. 6
Fuentes de Tepepan
Delegación Tlalpan, 14648
México, D. F., México

Abstract

Educational evaluation processes appeared in Mexico in the late eighties, associated with financing and accreditation to ensure the quality of educational services, according to the official discourse. This article presents a characterization of the processes of evaluation of teaching in universities and of student-opinion questionnaires. It stresses the use of this instrument in the decision to allow academics access to financial incentive programs, and technical carelessness in the instrument’s preparation. We point out the paltry amount of connection between evaluation and other activities such as ongoing training, planning and the continuing review of institutional conditions. It suggests developing a critical attitude toward current practices to enable to resolution of problems detected, and to encourage the positive aspects also present. The evaluation of teaching is a social activity imbued with value judgments—in which the actors do not participate with the desired equity.

Key words: Evaluation of teaching, evaluation of teachers, evaluation policies.

Introduction

Educational evaluation processes in our country, as in many nations of the world, have increased in importance during recent decades. Regular assessment has been a fundamental part of educational planning; however, it was only at the end of the eighties in Mexico that it acquired dimensions of special relevance because it arose as a tool for improving the quality of the school system, as a means of access to different workers’ compensation programs for academic staff, and as a condition of obtaining additional financial resources at the institutional level.

Since 1989, there have been established planning agencies, programs and individuals dedicated to the implementation of institutional evaluation systems. Among these are: the National Commission for the Evaluation of Education (CONAEVA),1 the Inter-Institutional Committees for the Evaluation of Higher Education (CIEES),2 the National System of Researchers (SNI),3 the Scholarship Program for the Performance of Academic Personnel,4 the Bachelor’s Degree Program in Education,5 the National Center for the Evaluation of Higher Education (CENEVAL),6 the Council for the Accreditation of Higher Education (COPAES),7 and the National Institute for the Evaluation of Educational (INEE).8

From the moment it was stipulated that the processes of evaluation would have the goal of differentiating and recognizing the work done by the institutions and the academics, the relationship took on a new dimension. The strategies of financial negotiation which the public universities and the government had sustained, underwent a change, at least discursively. Since then, the institutions and individuals had access to additional income, depending on the results of the evaluation processes. In addition to the evaluation/financing trend, accreditation was added as a strategy for ensuring the quality of educational services, according to the content of official documents. For this reason, among others, assessment has become in a comparatively short time, an essential part of the daily activities of the universities.

Another factor, pointed out by some authors, for the rapid expansion of the assessment processes associated with the discussion on quality and funding in higher-education institutions, is the influence of international agencies such as the Organization for Cooperation and Economic Development (OECD), the World Bank, the United Nations Educational, Scientific and Cultural Organization (UNESCO, for its acronym in English), the Regional Centre for Higher Education in Latin America and the Caribbean (CRESALC) and the Economic Commission for Latin America and the Caribbean (ECLAC), among others. This situation manifests itself in different countries such as the United States, Great Britain, Canada and France; including Latin American countries such as Brazil, Argentina, Colombia and Chile (Castillo, 2004 and Villaseñor, 2003).

Contexts and practices

Educational evaluation is considered a pillar of current higher education, now that it is part of the planning process powered by the programs of the federal government agencies related to education, with the aforementioned bias resulting from its association with financial aspects. In the case of public universities, evaluation has been implemented across the board starting from the nineties and continuing into the present, based on the policies promoted by official bodies like the Secretariat of Public Education and the National Association of Universities and Institutions of Higher Education (Rueda, Elizalde and Torquemada 2003). In recent years the educational sector’s policies have tended to favor the accreditation of professional training programs, derived from meticulous scrutiny of academic peers recognized by the official body responsible for carrying out this function, the COPAES; so that the actions of institutional assessment of plans and programs and those directed toward the academics are part of this new effort to achieve an improvement in the quality of the sector.

The context described above is part of the environment of the evaluation of teaching in Mexico, particularly relating to classroom teacher/student interaction as part of a professional training program offered by an institution. This activity is the object of evaluation with a differential treatment regarding other academics’ performance of other functions such as research, extension or dissemination of culture.

Based on the analysis done in a previous work on the processes of the evaluation of teaching in six Mexican public universities (Rueda, et al., 2003), we present here a general characterization of the state in which such activity is found in Mexico; however, we recognize the desirability of performing systematic studies on all or on a chosen sample of the public universities in order to reach more substantiated conclusions.

The first confirmation is the pervasiveness of various teacher-evaluation practices in their concept, age, type and uses; not only among universities but also among the processes followed in each program of professional training within the same institution. For example, in one university there can be an undergraduate program that has for three decades applied a system of teacher evaluation; another where the system has been suspended; and yet another that does not yet have an institutional program to carry out. However, it can be confirmed that in the group of public universities the impact of the teacher-evaluation policies, in the sense of their generalized application, is successful.

Generally speaking, agents that promote the initiation of the process of evaluating teaching are directors or people close to the incumbent administration, who receive the commission to design and manage these processes, with the justification that these activities are part of the institutional plans, or that the central governing body has determined it to be thus. So determined, they are even said to be required by external regulatory bodies. In the profile of those in charge of the evaluation of teaching, what predominates is a disciplinary training with a sense of belonging to the professional course of study concerned; the persons involved have limited work experience in evaluation. In all cases, evaluations of teachers are associated with universities’ wage-compensation programs put into place in times of widespread low pay for the academic staff of higher-education institutions.

In approaching the assessment experiences it was confirmed that the establishment of evaluation procedures is difficult. Some initiatives are even terminated in extreme cases because they are considered punitive toward teachers. In some facilities the teachers’ initial rejection ends up being diluted, since participation allows access to wage-compensation programs generally accepted by the academics, although many problems are pointed out in their everyday application.

Opinion questionnaires directed toward students are the only way for universities to comply with the policies of teacher evaluation, but there can be found isolated attempts to diversify or supplement this form of assessment with other sources. The use of questionnaires, employed in most universities in other countries, has experienced recent, strong momentum in Mexico. However, it is known that questionnaires began to be used toward the end of the sixties in a private university, and in the seventies in some schools of the National Autonomous University of Mexico (UNAM) and another private institution. It was not until the late eighties that their use became widespread in most public universities (Garcia Garduño, 2001).

The evaluation of teaching through the use of student-opinion questionnaires is recognized as the most widely used and controversial strategy in the university. In great measure this resource is employed by those who have accumulated experience in their adoption as a privileged instrument for the evaluation of teaching, and who have devoted part of their efforts to researching the questionnaires’ methodological and calculatory aspects, as well as to the validity and reliability of the instruments. Recently, in the debate on the subject, there are beginning to be recognized the effects of political and administrative factors inherent in the instruments’ application and the use of their results (Luna, Valle and Tinajero, 2004). But there is still a pressure to find out what biases in obtaining the results may be caused by a multitude of factors such as the time of application, group size, nature of the subject taught, number of response options used—to name some of those already documented in other contexts, but not explored in most of the evaluation experiences taken as reference.

Although the traditional use of questionnaires is relatively accessible, there are aspects that cannot be disregarded in the definition of a teacher-evaluation system, such as reflection on its purpose, congruence between the categories and indicators, criteria for interpretation of results, and clarity of the management and use of results (Loredo and Rigo, 2001). This situation has not usually been reported in the experiences reviewed. Similarly, in the daily practices of teacher evaluation in universities, there is an absence of the recommendations made by various authors regarding the usefulness of scores obtained on the questionnaires, the diagnosis and feedback of teachers for improving their classes; among administrators to determine the effectiveness of teaching derived from administrative decisions; or the ability to provide students with information for use in selecting courses (Luna et al., 2004).

If the impact of the feedback provided by the questionnaires on teaching has been demonstrated, it has also been pointed out that a written communication of the results is not sufficient, above all, when compared with the effects of personalized consultation (Luna et al., 2004).

Pervasive in the questionnaires are the dimensions relating to the planning, development and coverage of the program, as well as ways of evaluating learning. To a lesser degree are considered dimensions on the mastery of the subject, the management of teaching techniques, the style of teacher-student interaction, attendance and punctuality. Usually there is present in the processes of teacher evaluation neither a concern for identifying the dimensions associated with effective learning, nor a careful analysis derived from an orientation of the learning theories or a consideration of the results of research on university teaching practice, as a backdrop that would compel the recognition of the complexity of the activity evaluated. This can mean a risk related to giving exaggerated importance to the questionnaire, or attempting to ignore the limitations of any form of assessment.

In some universities, as well as the student-opinion questionnaire there are utilized self-assessment questionnaires for teachers, as well as assessment by academic peers and by supervisors (Rueda et al., 2003). Only two of the institutions analyzed, used different questionnaires based on educational level (undergraduate or postgraduate) (Luna et al., 2003). There is usually just one questionnaire for all the disciplinary areas and scholastic levels. We must remember that the existence of a single evaluation questionnaire leads to the implicit assumption that teaching skills are generic. However, the discussion over the possibility of determining aspects of effective, general teaching for all types of subjects is far from closed. Likewise, it is striking that every day there is more evidence for the existence of specific needs for the teaching of the different disciplines. To this it is also necessary to distinguish between the nature of the discipline, and the teaching of it in a professional-training context.

In most universities no participation by teachers and students has been reported in the production and implementation of assessment instruments. Only two institutions used the experiences of some teachers in making up the final version of the questionnaire, and one also considered the students’ point of view (Rueda et al., 2003). A review of the literature makes clear not only the strategic need for teachers, students and student bodies to be involved in the evaluation processes for improving the activity (Garcia Garduño, 2001), but also the advisability of knowing the actors’ point of view, above all since they may differ among themselves, or their perspective may change over time.

In some universities, the bodies in charge of evaluation process inform administrators and teachers of the results, while in others, the results are made available to the entire school community. Various strategies coexist within and between universities; in some academic programs all teachers are evaluated, and in others only some are selected. This situation allows us to reflect on the intention of those who promote the evaluation process, since if it has to do with influencing the quality of teaching, it is not justifiable to evaluate samples only, or to fail to disclose the results to those tested. The perspective of making available the results of the teachers’ evaluations to the entire community is acceptable only when it is guaranteed that the design of the evaluation process, the application of the instruments and the interpretation of the results have universally met accepted professional standards.

The most relevant characteristics found in nearly all cases are: the use of the evaluation for access to financial incentive programs, and technical carelessness in the drafting and implementation of teacher-evaluation instruments expected to improve their performance; in this sense, there has been observed little connection between the evaluation exercise and other institutional activities with which it should be naturally attached, such as ongoing training, planning, and continuous review of the institutional conditions under which teaching takes place.

The location of the groups and the academics in change of designing and implementing the teacher-evaluation processes, reveals the origin of the initiative and the import of the assessment activity. That is, there is revealed a clear concern with administrative control, and a preferential use of the evaluation as a requirement for access to monetary-incentive programs. In this regard, it would be useful to clarify the purpose of the evaluation, above all, to reorient it toward performing as an instrument for perfecting the activity of teaching. It is possible to use the assessment to “reward”, but both the evaluators and the evaluated must be aware of this. The same clarification would have to be made when the evaluation is employed on a more inclusive basis; such as is the case, for example, when the teacher is promoted, or is not hired, based on the results he/she obtained. The argument about the essential quality of the assessment remains valid for this case, but it is also necessary to add that a single measurement would otherwise be unfair, and we would have to prefer a series of assessments that would provide a better panorama of the activity evaluated, above all, for use in making a judgment that could have hazardous consequences for those who have been the object of the evaluation.

Conclusions

Our first comment is directed toward recognizing that the evaluation of teaching is an integral part of the universities’ daily actions. It represents a positive fact in the perspective of constructing a culture of evaluation within the set of social actors of the educational sector. However, we must develop a critical attitude toward current practices, so as to be able to resolve the problems detected and encourage the positive aspects also present.

In view of the aforementioned, it can be interpreted that we are facing further evidence of the impact of the general policies of individual-performance evaluation on university academics, associated with monetary-incentive programs which it will be necessary to scrutinize in greater detail so as to assess the effects formally predicted, such as the unexpected ones that have occurred in each of the institutions. The impact of teacher evaluation on increasing the quality of teaching and learning in higher educational institutions remains to be demonstrated.

It is necessary to encourage case studies to determine the effect of policies after more than two decades of their implementation. With the partial information available, serious deficiencies have been noted in the design and implementation of the processes through which an effort is made to evaluate the teaching profession in institutions of higher education. While some exemplary cases can be identified, the suspicion is that these are the exception rather than the rule.

Because of the environment of multiple assessment initiatives within the institutions of higher education, it is necessary to make an analysis of each one, so as to confirm their complementarity or to detect their contradictions. Already in international forums (Ardoino and Berger, 2003), it has been pointed out that in the evaluation processes there can be distinguished at least three planes: practices, devices and systems. When evaluation is analyzed on the plane of making a comparison among the experiences in different countries, light is thrown on the evidence for the role of culture in the orientation and implementation of these processes.

There is a lack of institutional initiatives to offer options for specialized training to professionalize the conduct of those in charge of the design and implementation of teacher-evaluation processes. Without a doubt, the accumulated experience of these teams created to respond to the orientation of policies, can be a source of information for the exchange of views between them, or for providing professional support to institutions that are in the beginning stages of developing processes of teacher evaluation.

The design of public policies for strengthening teaching must contemplate the diversity of institutional settings and teaching situations for each subject area, so as to encourage research and activation of effective assessment processes that would effect the improvement of one of the university’s principal functions.

It is necessary to encourage a discussion of the interpretation and use of the results yielded by the assessment processes and the analysis of the each player’s role throughout the process. How can they benefit from assessment by the students, teachers and directors of an institution? What conditions must prevail in order for the evaluation to become a fruitful dialogue between the evaluators and the evaluated? How to guarantee the ongoing review of the assessment system itself so as to ensure its perfection?

Finally we should not lose sight of the fact that evaluation of teaching is a social activity, which, as such, is full of value judgments, not equally present in the consciousness of all those involved. Furthermore, not all the actors necessarily participate in these processes with the desired equity, because of which there must be a guarantee of channels of dialogue that would actually lead to an improvement in the evaluation processes so as to achieve effective teaching/learning practices.

References

Ardoino J. & Berger, G. (Eds.). (2003). Former les enseignants et les éducateurs, une priorité pour l’enseignement supérieur: Actes du colloque. Paris: Association Francophone Internationale de Recherche Scientifique en Education.

Castillo, O. E. (2004). Sociedad del conocimiento y políticas públicas de evaluación del trabajo académico en educación superior: Programa de estímulos al desempeño docente en la universidad de Sonora, periodo 1993-2002. Unpublished doctoral thesis, Universidad Nacional Autónoma de México.

Cetina, E. (2004). Estímulos al desempeño del personal docente en las universidades públicas estatales. In M. Rueda (Coord.), ¿Es posible evaluar la docencia en la universidad? Experiencias en México, Canadá, Francia, España y Brasil (Proceedings), (pp. 77-82, Col. Biblioteca de la Educación Superior). Mexico: Asociación Nacional de Universidades e Instituciones de Educación Superior-Universidad Autónoma Benito Juárez de Oaxaca.

Garcia Garduño, J. M. (2001). Diez consejos para hacer fracasar la implantación de un sistema de evaluación de la docencia. In M. Rueda, A. Díaz Barriga, & M. Díaz, Evaluar para comprender y mejorar la docencia en la educación superior (pp. 104-111). Mexico: Universidad Autónoma Metropolitana-Universidad Nacional Autónoma de México-Universidad Autónoma Benito Juárez de Oaxaca.

Loredo, J. & Rigo, M. A. (2001). La evaluación docente en un contexto universitario. Una propuesta formativa y humanista. In M. Rueda, A. Díaz-Barriga, & M. Díaz (Eds.), Evaluar para comprender y mejorar la docencia en la educación superior (pp. 55-68). Mexico: Universidad Autónoma Metropolitana-Universidad Nacional Autónoma de México-Universidad Autónoma de Baja California.

Luna, E., Valle, C., & Tinajero, G. (2003). Evaluación de la docencia: Paradojas de un proceso institucional. Revista de la Educación Superior, 32 (127), 89-100.

Luna, E., Valle, C., & Tinajero, G. (2004). Evaluación de la docencia. Vicisitudes de un proceso institucional. In M. Rueda (Coord.), ¿Es posible evaluar la docencia en la universidad? Experiencias en México, Canadá, Francia, España y Brasil (Proceedings), (pp. 169-183, Col. Biblioteca de la Educación Superior). Mexico: Asociación Nacional de Universidades e Instituciones de Educación Superior-Universidad Autónoma Benito Juárez de Oaxaca.

Rueda, M., Elizalde, L., & Torquemada, A. D. (2003). La evaluación de la docencia en las universidades mexicanas. Revista de la Educación Superior, 32I (127), 71-77.

Villaseñor, G. (2003). La evaluación de la educación superior: su función social. Serie Cuadernos de Reencuentro, Análisis de Problemas Universitarios, 36, 21-29.

Translator: Lessie Evona York-Weatherman

UABC Mexicali

1 In this commission, created in November of 1989, participate representatives of the institutions of higher learning, from the National Association of Universities and Institutions of Higher Learning (ANUIES) and the Mexican Federal Government.

2These organizations were created based on the development of policies that tend to construct a national system of evaluation independent of the universities themselves; principally made up of colleagues from the same discipline as the program that will be evaluated.

Please cite the source as:

Rueda, B. (2004). Evaluation of the educational relationship in the university. Revista Electrónica de Investigación Educativa, 6 (2). Retrieved month day, year, from: http://redie.ens.uabc.mx/vol6no2/contents-rueda.html