The evaluation of schools is a very complex task, above all, in practice, due to technical reasons, environmental pressures, lack of qualified assessors, and difficulties in obtaining the participation and collaboration needed (Escudero, 1997). Evaluating a school is supposed to answer three fundamental questions, namely: “What do we want to evaluate? How are we going to do it?” and “Why are we are going to evaluate?” (De Miguel, 1997).

The following is the model known as NETWORKS, which has been set in action in the Basque Country for the external evaluation of compulsory high schools (CHS).

In 1999 the evaluation team, with funding from the Santa Maria Foundation, created the Institute of Educational Evaluation and Consultation (IDEA, for its initials in Spanish), organized in a decentralized manner and having several offices in different autonomous communities, including those of the Basque Country.

The evaluation began in the academic year 1999-2000 in the Basque Country, and has since involved 55 compulsory high schools (for young people 12-16 years of age). As noted previously, it was promoted and funded by the Department of Education of the Basque Government, and the faculty of the University of the Basque Country is in charge of putting it into practice.

In the agreement signed between the IDEA and the University of the Basque Country there was contemplated an evaluation plan for four consecutive courses. At the same time in other Spanish autonomous communities (Catalonia, Madrid, Andalusia, etc.), there was being implemented the same of evaluation plan. This would give the opportunity to obtain benchmark scores for different regions.

The role of educational administrator, in this case the Department of Education of the Basque Country government, was, as well as the promoter, the facilitator and controller-supervisor. No accountability was contemplated by the schools evaluated, and indeed, one of the characteristics that it guaranteed was the confidentiality of the data. The information the different schools received was confidential, and could be known only by the school and the external evaluation team members, so that even the administration did not have access to it.

g) Formative Evaluation

The ultimate objective of this evaluation was to work together with schools so they could know themselves better and design strategies for change. From this point of view, we can speak of a formative evaluation. The functions of inspection and control are a long way from this assessment plan.

h) Interpreted by the schools themselves

The information provided to schools was descriptive, without interpretations. The analysis and interpretation of information had to be made by the schools themselves, since they better understood the scope of the results, and because they had a more accurate knowledge of the school’s history, culture, problems, and decisions that have been adopted.

i) Able to understand the factors that influence the quality of teaching

To the extent that the evaluation applied to a representative number of schools, it provided more scientific information on the relative influence of the different levels and dimensions that made up the evaluation model.

Each of the four levels identified comprises several dimensions. Table I shows the dimensions selected in each of the levels.

Table I. Selected levels and dimensions in the NETWORKS evaluation design

At the first level, the entry level, there were collected the preliminary results for students entering high school. This level was very important for knowing the changes that could take place over time, and for determining most reliably the added value of the school. The added value is the measurement of the improvement occurring in students’ performance in a school.

The level of context has to do with cultural and socioeconomic status. As mentioned on several occasions, all studies emphasize the influence of the socioeconomic and cultural context on the results obtained by students.

The process level includes the specific processes of the classroom and those related to the school’s organization and functioning. Some of the processes studied in this dimension of the school were the following: knowledge of how the school functions, participation, work environment, assessment of the management team’s functioning; functioning of seminars or teaching departments; expectations regarding the school and its students; and working conditions. In terms of classroom processes there were taken into account, inter alia, the planning of teaching, innovation in evaluation, the teacher’s ability to maintain an orderly work environment, etc.

The products level mainly emphasizes the students, but also incorporates the views of teachers and family. With regard to students, Marchesi (2002) notes that there is advocated a balanced and comprehensive concept of academic performance. Accordingly, in this model, students evaluated the learning of most curricular areas (Mathematics, Social studies, Natural Sciences, Technology, Spanish Language, the language of the country (Basque), and English, but also included metacognitive skills, their own learning strategies, and their attitudes. They also completed a questionnaire on how the school was functioning; in this questionnaire they assessed their relationship with the tutor, teachers and other classmates; extracurricular activities; skills and training received, etc. Parents, in turn, answered a questionnaire evaluating the operation of the school: the attention they received; the relationship with the tutor; the school’s order and discipline; the grades received by their children, etc. Finally, teachers expressed their level of satisfaction with the school’s operation, and the conditions in which they did their work.

An operational-type evaluation was carried out in the following five stages:

1. The school’s commitment. The headship team collected information about the characteristics of the evaluation program. After the governing bodies of the school discussed and analyzed the program, they made the decision as to whether or not to participate in it.
2. Data collection.
3. Interview with the headship team about the report from its school.
4. Interpretation of information by the school.
5. Proposed change and evaluation of subsequent impact.

Those responsible for evaluation were aware that an external and quantitative evaluation, which seeks to provide relevant and contextualized information to schools by applying the same tests in all of them, allows for greater objectivity in the analysis of the results, and gives them a wider perspective of their situation, although this procedure has its limitations (Marchesi, 2002). First, the tests are shorter, and cannot replace the information that each teacher has about her? learners, and second, the excessive emphasis on the quantitative data can diminish the perception of the importance of the qualitative and internal evaluation done by the school. Therefore, we insisted that this external evaluation was only an aid to the evaluation necessary within each school, stressing that it must be they themselves that would interpret the information provided them. On the other hand, this quantitative information was complemented with the interview maintained with the headship teams.

The questionnaires applied were mostly Likert scales. Curriculum examinations were objective test that attempted to maintain a balance between the conceptual and the procedural.

One aspect that merits attention is the preparation of tests and questionnaires. The data collection tools were the same for all state schools, except the tests on Social Studies and the Language of the Community, for obvious reasons. In the first stage, these questionnaires and tests were prepared in the IDEA by test-construction experts and high school teachers who piloted the tests and performed the analysis of items and test, as well as matching, following the models of classical test theory and Item Response Theory or IRT.

According to the classification of Jornet and Suárez (1996), we can define the tests used as standardized tests of performance indicators, since they attempted to translate the levels of competence, which an institution, a school or educational system assumes to be goals to achieve in the educational process in the various disciplines and subjects.

In test production there is a very important component, educational mastery, to which they hoped to respond. Understood as educational mastery are the dimensions one attempts to measure within a subject or discipline. The term mastery was produced by experts in the subject matter in the field of evaluation, and has the support of specialists in measurement and evaluation. The role of these professionals is essential, since the work they do in the configuration and development of the tests comes to compensate, somehow, for the difficulties encountered because of the weak theoretical constructs often supporting them, and by the lack of proof for their unidimensionality. This is because the specific methodological proposals for empirical analysis of the results using dichotomous variables have yielded unsatisfactory results in identifying the dimensions, in many cases, based on the difficulty of the items and not on the contents (Jornet and Suárez, 1996).

Besides the work of experts in determining the theoretical importance and difficulty of the items, another key element was the technical quality of the tests. To ensure such quality, it was important to perform a psychometric analysis of the test, so as to guarantee its reliability and that of its items one by one. In this regard, and based on data provided by the pilot test, analysis of distractors constituted the central work in this area, because it allowed us to anticipate and manage any difficulties that might arise in the final application of the tests. Along with this, analysis of potential bias was also a key element for ensuring the validity of the tests.

Finally it could be concluded that the standardized tests being applied were based on a correct approach, since being planned for extensive and diverse populations they were not interpreted individually; on the other hand, we considered the selection of indicators, the analysis of items, and the proof of reliability and validity checks to be sufficient. To this we should add another element we consider very important—that which has to do with the proper use of the evidence, by which we attempt to have a more formative than summative dimension on which to base decisions about the group, and not about individuals.

The information that until then had been offered to the institutions participating in the evaluation was regulatory, in the sense that each knew its position in relation to the rest of the participating institutions.

In order to complete the normative character of the test interpretations, there were also established standards and cutoff scores on the performance tests, as is done on criterion-referenced tests, so that using the standards established we could determine the level at which we could classify the students as competent or incompetent in the performance variable considered.

In this context, and taking into account the importance possessed by the variable sociocultural context of families, work is currently being done with the aim of being able to offer different standards for different levels. The process followed in establishing those standards and the influence of sociocultural context on them, are specified in detail in an article by Lizasoain et al. (in press) which will shortly be published.

Steps followed in the collection of data:

The tests and questionnaires relating to first-level high school education were applied in October, i.e. the beginning of school at this stage. Along with these, data was collected to determine the socio-cultural context by means of a questionnaire applied to all students of 1st, 2nd and 4th levels of high school. The application of these tests at the beginning of the course was aimed at making an initial assessment of students; this would be valid for the teachers of that course (in the sense that these first data could adjust the programs to the real situation), and for the school itself, so that over a period of four scholastic years the progress of those students could be appreciated.

Before the data was collected, a meeting was held in each of the schools; involved were the school principals, high school teachers and the evaluation team. In that meeting the school was informed about everything related to the evaluation plan that would be implemented: objectives, characteristics, types of tests and questionnaires, dates and phases of data collection, and so on. It was at this time that various aspects of the evaluation could be negotiated. Thus, the meeting was very important because it laid the foundation for the climate of collaboration (between the schools, and in this case the evaluators) needed to meet the established objectives.

During the months of January and February the students of 2nd and 4th levels filled out the questionnaires on attitudes and opinion; the faculty, the questionnaires about the processes and the classroom; and the families, the questionnaires regarding their assessment of the school. Finally, in May, performance tests were applied to students in the 2nd and 4th year of high school.

• Include all information needed by the different audiences.
• Try to be pertinent and relevant.
• Return the reports as soon as possible (in the same quarter in which the tests are applied)
• Try to be brief and concise.
• Use accessible language.
• Use various communication resources such as tables, graphs and so forth.
• Do not include recommendations, since that is considered to be a task for the recipients of the report.
• Use an expository style.

For this, two meetings of the evaluator with the school director were held during the school year, one in November and another in June. In these meetings the results were delivered.

Each year two reports were delivered to each school: in November, one with student results of Level 1, and another in June with the results of Level 2 and 4 tests applied by teachers; evaluations made by students and parents. They were simple reports, essentially describing the data presented, without any evaluation. Possible interpretations were left for the meeting of the persons in charge. The structure of the reports was as follows:

• Explanation of the data on the school’s sociocultural context.
• Individual scores for each student in the different areas evaluated. In addition to the total scores, subscores were collected in each of the subareas.
• Scores on the students’ attitudes, and evaluations of the school made by the students, teachers and families, both overall and sectorial.
• Average scores, and histogram for the average scores in each of the areas, and for each group/class of each level (1st, 2nd and 4th.)
• Table showing the school’s points in each of the areas and sub-areas by group/class, along with the average scores of the schools classified in the same sociocultural context.
• Polygon of frequencies in which are reflected graphically and numerically the mean score of the specific school, schools of the same sociocultural context within the autonomous community, the total of those of the community, of schools of the same social context within the Spanish State, and the total of those that participated in the evaluation within the State.
• Description and schematic of each of the tests, to be used in interpreting the results.
• Explanation of the report and of the most important findings.

As we have pointed out above, it is the schools themselves that had to make timely interpretations and evaluations based on their own criteria. Therefore, one of the most important objectives of the meetings held for delivering and presenting the results was concerned with orientation in this interpretation and helping to establish projects for change and improvement.

Below are examples of two graphs that appeared in the reports (see Figures 2 and 3). The first shows the results of the questionnaire “School Processes” filled out by teachers, compared with the results of the schools of their own contexts, and with those of all the schools of the Basque Country (BC) and the State of Spain.

Figure 2. Results obtained for the school by means of the questionnaire
“School processes” (dummy data)

Figure 3. Results obtained by the school on the Social Studies test
of Level I compulsory high school education (dummy data)

Figure 3 shows the results obtained by the school on the Social Studies test for CHS Level 1 students, compared with the results obtained in the establishments of the same context and with those of all the schools of the Basque Country (BC) and the State of Spain.

However, when a school completed the full four years of the evaluation program, it received a final report describing what happened during that period. Basically, the structure of the report was similar to that of other progress reports, except that it offered a series of graphs that summarized the results obtained during these four scholastic courses in 1st, 2nd and 4th year of CHS (Figures 4, 5 and 6) in each of the dimensions evaluated. Also included was a final graphic that showed the route followed by the students to whom the evaluation program was applied in the first year, CHS Level 1, and at the end of the evaluation program had completed Level 4 (see Figure 7.) By means of that figure, one could get an idea, though not entirely accurate, of the added value of the school in question.

Figure 4: Mathematics CHS Level 1

Figure 5. Mathematics CHS Level 2

Figure 6. Mathematics CHS Level 4

Figure 7. Development of mathematics
(the same group of students in Levels 1, 2, and 4)

• Direct or indirect information from various sources and with varying degrees of relevance and reliability in each case;
• Normative, institutional or personnel factors which determine the existing margin for maneuvering;
• Information about the context of interpersonal relationships, anticipating the reactions that may occur;
• Multilateral negotiations so that all the actors involved feel themselves to be co-participants in the decision adopted.

That is, we move in a reality where information and knowledge are necessary but not sufficient, since the ability to negotiate and mobilize the will, the sense of opportunity and even the intuition are also very important. In order to facilitate this work, the evaluation model NETWORKS proposes a series of guidelines for making evaluation a tool for improvement. These are summarized below.

Evaluation is only a first step, an initial photograph that reveals the whole and each of the parts with some precision. It displays the strong contours and the blurred ones.

But this photograph is unable to speak for itself; we must learn to see, understand and interpret it, which requires the participation of the headship and the faculty of the school. Therefore, as mentioned on several occasions, the confluence of internal and external evaluation is a guarantee for a better understanding of the school.

However, this step is not the last. So that evaluation can help to generate processes of change and improvement, it is necessary to take at least two steps. The second is to interpret and use properly the information obtained. The third is to promote projects or initiatives for change. A final step that completes the circle is to re-evaluate, in order to confirm the dynamic being produced (see Figure 8).

Figure 8. Process of evaluation and change

Ensuring that information from the evaluation be made available to most of the people involved in the school’s operation will enable the promotion of change processes—provided, of course, that the people receiving the information are able to interpret it correctly so as to benefit the work of teachers and the school.

Proper use of the information is already an important part of the change process. Reflection shared between the various members of the educational community, especially the headship teams and those of teachers, will help to arouse awareness of the situation and to change attitudes, teaching styles and ways of working. Similarly, parents and students modify their attitudes and expectations toward the school when they feel part of the evaluation process.

However, while we acknowledge its undoubted interest and advantage, we must assert that this collective analysis must go one step further, lead to reinforcing the positive aspects of the school, and to resolving the problems diagnosed. As noted in the previous paragraph, it is the work of the headship team to determine the possible projects for improving the school and to establish the objectives, periods, conditions, responsibilities and actions to accomplish them. But we must never forget that changes in education are slow and laborious, and there are many conditions which must be taken into account (the school’s culture, the attitudes of its teachers, the time available, human relations, professional expectations, etc.), since on them will depend the eventual success or failure of the plan of action. On them depend the scholastic community’s increase of satisfaction and morale.

Jencks, C. S., Smith, M., Acland, H., Bane, M. J., Cohen, D., Ginits et al. (1972). Inequality: A reassessment of the effect of family and schooling in America. New York: Basic Books.

Jornet, J. M. & Suárez, J. M. (1996). Pruebas estandarizadas y evaluación del rendimiento: usos y características métricas. Revista de Investigación Educativa, 14 (2), 141-163.

Lizasoain, L., Joaristi, L., Santiago, C., Lukas, J. F., Moyano, N., & Sedano, M (in press). Establecimiento de estándares y puntos de corte en pruebas de rendimiento. Una aplicación en el área de Matemáticas de 2º y 4º de ESO. Bordón, 56 (2).

Lukas, J. F., Santiago, C., Munarriz, B., Moyano, N., & Sedano, M. (2000) Euskal Autonomia Erkidegoko ikastetxeen kanpo-ebaluazioa. Tantak, 24, 39-63.

Marchesi, A. (2002). El proyecto REDES. In A. Marchesi & E. Martín (Comps.), Evaluación de la educación secundaria. Fotografía de una etapa polémica (pp. 21-33). Madrid: Fundación Santa María.

Marchesi, A. & Martín, E. (1999a). Tendencias actuales en la evaluación de los centros docentes. Infancia y Aprendizaje, 85, 5 18.

Marchesi, A. & Martín, E. (Coords.). (1999b). Una red de evaluación de centros de educación secundaria. Infancia y Aprendizaje, 85, 59 73.

Marchesi, A. & Martín, E. (Comps.). (2002). Evaluación de la educación secundaria. Fotografía de una etapa polémica. Madrid: Fundación Santa María.

Mateo, J. (2000). La evaluación educativa, su práctica y otras metáforas. Barcelona: ICE Horsori.

Pérez Juste, R. (1999). Nuevas perspectivas de gestión, control y evaluación. Bordón, 51 (4), 449 477.

Reynolds, D., Teddlie, C., Creemers, B. P. M., Scheerens, J., & Townsend, T. (2000). An introduction to school effectiveness research. In C. Teddlie & D. Reynolds (Eds.), The international handbook of school effectiveness research (pp. 3-25). London: Falmer Press.

Sáez, M. J. (1995). Caracterización y problemática de los informes de evaluación. In M. J. Sáez (Ed.), Conceptualizando la evaluación en España (pp. 83 93). Madrid: Servicio de Publicaciones de la Universidad de Alcalá.

Tiana, A. (1997). Tratamiento y usos de la información en evaluación. Retrieved January 20, 2004, from: http://www.campus-oei.org/calidad/tiana.htm

Tiana, A. (1999). Ikastetxeen kanpo ebaluazioa Aukerak eta mugak. Tantak, 21, 7 32.

Webster, W. J., Mendro, R. L., & Almaguer, T. O. (1994). Effectiveness indices: A “value added” approach to measuring school effect. Studies in Educational Evaluation, 20, 113 145.

Translator: Lessie Evona York-Weatherman

UABC Mexicali