• The dubious executive capacity of the directive bodies of public universities with respect to their peers in the private sector.
• The disconnection between the central government, the university and the regional governments in the determination and formulation of common goals.

1.1 The dubious executive capacity of the directive bodies of public universities with respect to their peers in the private sector.

This aspect refers to the diffuse and weak capacity for autonomy and leadership which prevails in many Spanish universities because of statutory developments that ignore or superficially address these issues. The entrenched corporatist trend of the university structure, on the one hand, and the complex network of colleges, on the other,2 contribute to the arrival of a shock absorber/filter that slows down decision-making processes, makes them more complex and expanding them to such an extent as to impede the practical and operational execution of any order extending it indefinitely. Vicente says (2001, p.1):

The audits were limited to autoevaluations and external evaluations carried out by “friends” from other “branches”; but the truth is that nobody knows exactly if there are “losses” or “benefits” and what is worse, nobody demands them; nobody knows of any case in which an academic authority has resigned or has been penalized because of a university’s low performance.

Thus, contrary to one of the objectives of the NPUQA:

To provide educational authorities and the Council of Universities objective information on the quality level achieved by the universities to serve as a basis for decision-making in the field of the respective competence (Royal Decree 1947/95, Article 1, paragraph 3, p. 35.473).

We affirm that neither has information been transmitted with the desirable fluency (De Miguel, 1999a, p.110), nor have reports issued served as a basis for decisions aimed at accountability, improving the situation or making commitments to social change (De Miguel, 1999c).

In addition to the network of colleges constituting the structure of governance and representation of the Spanish university, despite the implementation of the Organic Law of Universities (OLU)3 in 2001, there is another set of conditions and circumstances contributing to the development and maintenance of a weak executive capacity. In this sense, we emphasize the presence of three different levels participating, sometimes with little coordination, in the process of quality evaluation: Level 1, which consists of the Council of Universities and the General Secretariat of the Council of Universities, agencies of the Ministry of Education, Culture and Sports; Level 2, consisting of the Autonomous Agencies for University Quality which oversee the Autonomous Communities; Level 3, structured by the Evaluation Committees, Autoevaluation Committees and External Evaluation Committees. The first two are agencies of the unit evaluated, and the third, of the Council of Universities.

This structure has brought about the sluggishness of the administrative bureaucracy and various disturbances derived from the lack of organization and coordination between the levels of performance described, as well as the insufficient means of carrying out the tasks needed to be done in the time expected. The consequence of this accumulation of malfunctions has been the dearth of decision-making, once the final distortions contained in evaluation reports were known. In this regard, Coba (2001, p. 386) has commented that “the limitations of NPUQA, to summarize them in a sentence, have been the scant involvement and impact which the results of evaluation have had beyond the evaluated units themselves.” Finally, we emphasize the difficulty of extrapolating the business sector’s quality management techniques to whatever service organization—in our case, the university. In this sense Quintanilla (1999, p. 88) distinguishes between:

a) Conceptual problems: referring to the difficulties in transferring the concepts of quality and execution of corporate control to the university, as well as the difficulties in identifying them clearly to the clients of the higher education service.
b) Methodological problems associated with the need to articulate different dimensions and levels in the evaluation of the quality of universities, and to fit the methodology of quality evaluation together with other standard practices within the academic culture.

1.2. The disconnection between the State, the university and regional governments in the identification and formulation of common objectives

In the first place, this aspect may contribute to the dissolution of the rights and duties existing in relation to and for universities. This is because neither are the universities capable of taking the initiative on their own, nor do the regional governments immediately come to terms with the guidelines of the central government. Thus, in many cases, the disconnection is due to strong discrepancies between the central government’s underlying political approaches and those of the regional administrations.

The administrative organization of Spain is geographically distributed over regional communities with ample competencies in the various areas of social, political and economic reality (health, social services, education, etc.) It forms a heterogeneous mosaic where understanding is not always easy.4 That, although desirable in a democratic state, creates some mismatches—the result of misunderstanding between the State, the autonomous communities and the universities, for lack of coordination between the public powers that be, and the universities’ goals. Although these clashes have been particularly virulent—at least individually—when universities get together and express their problems collectively to the group of public authorities, within the Council of Universities, the situation becomes saturated with frictions evinced by endless debates and violent verbal confrontations (Perez Rubalcaba, 1997). In this sense, a rapprochement seems reasonable, aseptic with regard to all ideology, in the joint project of articulating a collection of goals, strategies and activities for evaluating the university’s activities. This would be fully consensual on the part of the academic institutions, the regional governments and even the central government.5 Some of the conclusions reached in the work group dedicated to the university and the organizers of the seminar “The Objectives of the University Facing the New Century” (Conference of Rectors of Spanish Universities [CRUE], 1997, p.1) have already pointed in this direction:

“It is necessary to find, with mutual loyalty and service to the public interest, a satisfactory balance between the autonomy of the universities, recognized constitutionally, and the powers of the autonomous parliaments and governments”.There must also reviewed as well the framework of relations between the universities and public authorities through coordinating bodies, so as to ensure a satisfactory relationship between academic and political economic aspects of the Spanish university system. It is urgent to have a stable but flexible framework which would avoid continuous and costly reforms of the rules relating to curriculums and other aspects of university education.

However, in spite of all this, we cannot overlook the encouraging measures that have been taken in connection with the OLU after the creation of new management and coordination organizations between the three administrations involved in the university’s future. Thus, one can read the following in paragraph IV of the OLU preamble:

The Council for University Coordination will be the maximum consulting and advisory body of the university system, and is configured as a forum for encounter and debate between the three administrations that converge in the university system: State, Regional and University (Organic Law of Universities [OLU], paragraph IV, p. 49.402).

• The absence of a systematic method of data collection to support evaluation.
• Lack of mechanisms for analyzing and validating the information collected.
• Lack of executive actions for the implementation of immediate improvements, and lack of financial support to make them operational.
• Imbalances that affect the process of selecting and training assessors.
• Inadequate functioning of the evaluation committees.
• Questioning performance indicators in terms of how they are drawn up, applied and interpreted.

2.1. The absence of a systematic method of data collection to support evaluation

Concerning this weakness, we mean the lack of information available regarding information-gathering instruments which should be used in the process of evaluating universities. While the NPUQA guides and those of the current UQP make it clear which indicators should be measured, it is uncertain what instruments should be used, and by means of what methodological strategies this should be done. The indications are too generic,6 and of course, provide little clarification for a group of agents who are usually inadequately trained in research. Should standardized instruments be used or developed ad hoc? The agents participating in the self-directed study centers are lost, and this undoubtedly slows down the evaluation process, except in cases where these agents have consistently participated as external assessors and have developed a mechanism of self-report.

Recently initiated have been systematic work processes centered on some specific aspect of the university system (e.g., evaluation of qualifications and teaching). However, we are far from having an integrated system of indicators with which to addresses the complexity of the system. This is compounded by the need to make explicit the theoretical reference models that would support such systems of indicators.

2.2. Lack of mechanisms for analyzing and validating the information collected

Even if we can overcome this first hurdle (that of the systematic collection of information), we find ourselves facing others:

a) The management, tabulation and analysis of data—laborious tasks, especially when most of the data except some specific ones, are qualitative, based on individual interviews with different actors: teachers, students, administrative and service personnel, and the governing team of the school or university.
b) Lack of criteria for validating the information gathered. The qualitative data require systematic triangulation strategies that would allow one to compare the information of more than one type of agent. This may not be feasible because it often generates more information than it is possible to process. In addition, teams often have no specific training in analyzing qualitative data and triangulation.

Both aspects have much to do with methodological training in which the internal and external evaluators have not necessarily been involved in the evaluation process—training that involves knowing about the technical criteria concerning the quality of instruments used for the collection of information, as well as the nature and type of sample used in the explorations carried out, the parameters of validity and reliability, either from the perspective of classical test theory or qualitative criteria (credibility, transferability).

2.3. Lack of executive actions for the implementation of immediate improvements, and lack of financial support that would make them operational.

The publication of the evaluation report and the obtaining of a number of conclusions could trigger different types of actions: accountability, improvement processes (development), to serve as a source of knowledge and commitment to social change.

Unfortunately, reports of results presented to date7 do not take a serious or decided view of any of the specified functions; this indicates the important executive deficit suffered by the Spanish university. Much is known about the disruptions that afflict our university, but little or nothing is done to solve them. In this sense, Perez Garcia (1998, pp. 123-124) indicates that for evaluation to bear fruit, it must first be possible, and also it must contribute to the improvement of information and to the production of strategic planning that would serve as a basis for executive actions based on results obtained through the evaluation process. Finally, Harman (1999) expresses the opinion that the activities of reporting and monitoring are vital elements of any valuable program of quality assurance, although the biggest change is in designing methods effective and capable of being performed to achieve improvements. Some regions are experiencing operational models of contracts-programs for the implementation of progressive improvements in different universities,8 but until these are structured and a consensus is reached on global and local strategic plans to assess the quality of higher education, all these measures will continue to be specific actions of little importance.

Nothing is more illustrative of this argument than the processes of the institutional evaluation of teaching,9 by students using more or less standardized Likert10 evaluation scales to assess overall compliance with teachers’ obligations, work method, motivational skills, relevance of subject programs, and adequacy of evaluation models. Even when a teacher gets a low rating on these scales, the results do not affect payment of the teachers’ five-year raises (granted automatically), nor their future promotion and stabilization. So, the question is what reasons led a university to implement practices of teaching-quality evaluation if the results will have no impact. As noted by Tejedor (2000), from the results obtained there should be derived decision-making processes and patterns of renewal when necessary. The formative sense of the evaluation lies in the assumption of the virtuality that the information provided would encourage the group of professionals (teachers, researchers and technicians) to make the relevant changes.

In this regard, various factors have been considered in the evaluation of teachers which De Miguel (1998, pp. 70-72) systematized in three criteria: productivity; teaching competence; and excellence or professional development. Into the first, the criterion of productivity, we can put the merit pay models and pay for performance; in both cases the salaries of teachers depend on teacher productivity according to students’ academic performance. These models of remuneration enjoyed a resurgence in the U.S. during the Reagan administration; however, due to the disappointing results achieved by many programs, these models have been modified or eliminated.

Thus, there are experiences that have empirically demonstrated their inefficiency. In this regard, Clofelter and Ladd (1996, quoted by Muir, n.d.) confirmed that in a number of Dallas schools where the merit pay model was applied, the students had increased their academic performance before the model was put into practice. Thus, the merit pay model could not be associated with increased academic performance. Another example found by Muir (n.d.) speaks of the experimental implementation of pay for performance during the Nixon administration in the United States. In this experience teachers were paid based on students’ reading achievement; so, to maximize their pay, some teachers focused their interest on students of average performance, ignored the better students because they considered them skilled on their own, and also ignored the worst ones, because they required too much attention and contributed little profitability.

Consequently, these and other experiences and studies discouraged the implementation of such models, which nonetheless are effected still in some countries. In short, there is no empirical evidence showing that merit pay models improve student achievement, and yet they can generate perverse side effects, of which Kirkpatrick (2001) and Muir (n.d.) reviewed some:

• Merit pay models can promote competition among teachers who, by virtue of their roles, must often work in teams.
• Merit pay models can “punish” good teachers who have students that are poorly motivated, poorly skilled, etc.; in short, bad students.
• Research on the implementation of merit pay models shows that they lower the morale of teachers.
• Merit pay models can sour relations between teachers and administrators.

Suffice it to recall the words of the famous liberal economist Milton Friedman, cited by Kirkpatrick (2001, p. 1) with respect to merit pay models: “Merit pay works in a competitive marketplace, not in a socialistic enterprise such as the public school system”.

2.4. Imbalances that affect the process of selecting and training assessors

The working methodology of the NPUQA was based on a model combining self-evaluation (Maassen, 1987; Vroeijenstijn and Acherman, 1990, Van Vught and Westerheijden, 1995) with peer review (Kells, 1992; Vroeijenstijn, 1995) and performance indicators recommended by the evaluating agency (Frazer, 1997).

In line with this, Harman (1999, p. 1) believes that quality assurance in the evaluation of the university depends on:

A combination of a limited number of key methodologies, the most important of which are: self-studies; peer review by panels of experts, at least usually involving external some members; the use of relevant statistical information and performance indicators; and surveys of key groups, such as students, graduates and employers.

Thus, this work structure generated the selection and training of internal evaluation committees (IEC) and external ones (EEC). In both cases the selection process was not without problems. De Miguel (1999a, p. 107) notes the following:

• In the IECs it has been noted that the appointments of the members have too often gone to the authorities of their own units and persons closely related to the governing teams. Nor does it seem desirable that the composition of these IECs be established in a totally participatory way, since this entails the risk of establishing one more committee with little operational power.
• In relation to the external evaluators, the lack of a tradition of institutional evaluation in Spain has made it difficult to set up EECs with participants who meet the established conditions, and who also maintain a positive attitude toward participating in these processes. To this we should add the question of the availability of participants: duties, agenda, etc.
• On the other hand, lack of experience in evaluating each other (IEC and EEC) required the establishment of training programs with the aim of informing the members of these committees about the purpose of the evaluation processes, and orienting them on the use of guides and resources that can be used to carry out the task. Unfortunately, experience tells us that future committee members, at least on the subject of autoevaluation, are barely trained—and improperly. Their contact with the evaluation process can be categorized as a challenge in facing the unknown.

2.5. Inadequate functioning of the evaluation committees

In general terms the performance of the IEC can be defined as unsatisfactory. To the non-disclosure of the reports, an aspect alien to the members of these committees, should be added their merely descriptive and politically correct nature. De Miguel (1999a, p. 108) highlights some problems which affect the work of the IECs:

• In many Spanish universities there are no technical units of evaluation (and if there are, they work in precarious conditions).
• Many institutions suffer from the lack of a reliable, updated and computerized database, because of which there are often produced reports of little value, far from today’s reality.
• The coordination and integration of information from different units to be evaluated becomes a difficult task due to the lack of understanding, skepticism and lack of cooperation from those who have to provide it.
• Lack of time and practical help has meant that most members of these IECs have lived this experience as a work overload, which has prevented them from making the effort the task requires.

We, for our part, would add a problem common to both committees: the dilemma of the voluntary versus the mandatory status of these committee members. If participation is voluntary, we run the risk of low involvement, lack of motivation in the work, and the inclusion of biased data with little or no verification. On the other hand, if participation is mandatory, we infringe upon the principle of the professional autonomy of some university agents. If there are not also established adequate compensation mechanisms (economic or academic), the activity of the committee members may be affected by incompetence, lack of motivation, and a lack of coordination. In short, the dialectic on the type of participation in the evaluation process is evident, and the solution is not at all easy. In fact, Harman (1999, p. 1) believes that:

An important variation between quality assurance systems is whether participation is voluntary or compulsory. Many countries began with institutional audits on a voluntary basis ?...?. Generally, however, with national reviews of disciplines participation is compulsory. Even when participation in such reviews is voluntary, strong moral and professional pressures often operate in institutions.

2.6. Questioning performance indicators in terms of how they are drawn up, applied and interpreted

The first important idea in this regard is that performance indicators, although they constitute one of the most popular tools for evaluating teaching, may involve a certain danger in three differential aspects (De Miguel, 1999b, pp. 1-4, Mora, 1999, pp. 2-3):

a) Quality of preparation.
b) More or less correct use made of them.
c) Sphere to which the evaluation makes reference.

The first two aspects would be what we call technical difficulties. Thus, in speaking about the quality of the indicators, we mean problems focused on questions relating to how they are constructed, what criteria should be selected and what is the theoretical foundation that justifies their production and application. Based on to these aspects, Wyatt, Ruby Norton, Davies and Shrubb (1989, p. 65) and Osoro and Salvador (1994, p. 279) established a set of criteria, still valid today, in the selection of indicators.11 These are:

1. Importance and use: value of information for policy development, audience interest and accountability.
2. Technical quality: based on the validity of the context and the reliability of the information collected.
3. Reliability: on the basis of careful data collection, its cost, analysis and report, and simplicity of the information.

On their use and application, most of the criticisms focus on aspects of their policy and practices, as regards several issues:

• There is a lack of criteria agreed upon as to pertinent information having to do with the quality of the object to be evaluated (Levesque, Bradby, and Rossi, 1996).
• There exists a question regarding their suitability and opportunity as a tool for decision-making, since indiscriminate use by policy-makers can have deleterious effects on the system (De Miguel, 1999b).
• There is a danger of turning indicators into quality standards. Performance indicators can be converted into a goal in themselves, and not into what they really are: a partial manifestation of a complex reality in a constant state of transformation and change (Vroeijenstijn, 1995).
• Performance indicators are ambiguous by nature, unless they are used in a context of dialogue (Bormans, Brouwer, In’t Veld, and Mertens, 1987; Cuenin, 1987, Frackman, 1989.) In this sense, strict procedures must be articulated so as to anticipate the potential conflicts that may arise between the different actors involved (Grao and Winter, 1999).
• There exists a tendency to obtain easily measurable indicators (Weert, 1990; In’t Veld, 1990; Kells, 1992); the right thing would be to try to measure what should be evaluated, and if necessary, resort to non-numerical systems (Dochy, Segers, and Wijnen, 1990a, 1990b). The set of indicators should include weighted quantitative and qualitative information (Barnetson, 1999).
• Quality, and therefore its indicators, is not an “absolute” of universal validity, and is consequently outside place and time. The specification of the field of quality makes it real and realistic; it keeps it from being submitted to rigid patterns, generally applied, and disconnected from a factual reality which imposes its own configuration (Raga, 1998). Therefore, without denying that rankings are of useful journalistic interest to the media, both their methodological reliability and the validity of the constructs used in their production are questionable (Berry, 1999; Morrison, Magennis and Carey, 1995).
• The use of indicators related to the allocation of funds and assignment of rankings can be especially problematic. If the indicators are bad, resources diminish and indicators will be poor due to a lack of resources—a vicious cycle (Williams, 1986). Funding adjusted to standard indicators that do not address the principle of equity is unfair, unrealistic and wrong. The starting point for each university is crucial. The same thing cannot be demanded for all scenarios evaluated. Equal treatment, for equal improvements in funding and rank would imply that each university would have different requirements, depending where it started from (Villarreal, 1999). Failure to act in that direction causes us much fear that:
  • Funding will be affected by what is called the “San Mateo” effect: more will be given to those who have more, and less to those who have less.
  • The classification of universities will be unreal, like the profit of a group, when an experiment has not been begun with comparable groups.
• Finally, it is advisable to warn about the confusion that can be produced between quality indicators and other aspects such as: information on management and statistics summaries, which are very important for corporate management, but are not true indicators of an institution’s quality (Escudero, 2000). The truth is that often the discernment of each other becomes an arduous and difficult task (Dochy et al., 1990a, 1990b, Segers and Dochy, 1996). However, as indicated by Escudero (1999), the assessor should sharpen his or her thinking about what an indicator of quality really is, and what it only appears to be, but cannot be considered as. It is habitual to confuse indicators with statistics, when in fact the first (indicators) are quantifications of the resources used or the results obtained in areas related to the objectives of a particular organization (Sizer, Spee and Bormans, 1992), and the second (statistics) are numerical characteristics of an empirical description. Because of all that, as Vidal (2001) strongly indicates, indicators should not be taken as absolute measurements, but acquire their meaning in the process of operationalization, and conceivably (we add), of contextualization.

The third aspect is that referring to the question of the scope of the evaluation. In this regard we find that the context evaluated is less controversial in some cases than others. There are disciplines, such as economics, where assessment, per se, is considered something inherent and necessary, but in other disciplines, such as education, evaluation is not always well received. On the other hand, and within this discipline, we find areas where institutional assessment is less controversial (research and management), and others where it is much more problematic (teaching-learning processes). The explanation of such phenomena may perhaps reside in the fact that the teaching-learning processes are difficult to put into operation, and consequently, questions arise about whether it is appropriate to collect information about them using performance indicators constructed with objective approaches.

1) Train staff volunteers participating in the processes of internal and external auto-evaluation (teachers, students and the personnel of administration and services [PAS]) in the process of assessing university quality with a set of ideas and basic research strategies.

The concept of the institutional evaluation of quality proposed by De Miguel, Mora and Rodriguez, 1991; Vroejenstijn, 1995; Rodríguez, 1995 and Van Vught and Westerheijden, 1995) is a model which postulates the integration of different types of evaluation: more external auto-evaluation, i.e. the use of internal indicators proceeding from self-regulation and peer review.

What is really capital about this concept of university quality assessment, according to De Miguel (1997a, p. 172), is its comprehensive dimension, which addresses both processes and products, as well as effectiveness and efficiency, improvement, quality management and the procedures established to assure it. The evaluation assumes, based on these parameters, a three-pronged approach: quality control, quality management and quality assurance, overcoming the exclusivity of the modalities: quality audit and quality assessment of clear external trends (Rodríguez, 1995, De Miguel, 1997b).

To establish various methodological strategies for this evaluation process there has been produced a series of models with different phases and procedures (House, 1993, Van Vught, 1995; Westerheijden, 1996; Rodríguez Espinar, 1997). In our judgment, the Achilles heel of most of these models is the first phase of the sensitization and preparation of this model: the selection of the institution’s internal staff who will be responsible for the evaluation process as well as the appointment of people who will be responsible for directing the process (the evaluation committee). And if we consider it this way, it is because in the selection of such members, some criteria that should be considered are not taken into account:

a) Preliminary criteria for the competency of the participants regarding institutional evaluation.
b) Members’ level of knowledge about research processes.
c) Participants’ level of involvement and personal interest about the evaluation of quality.

In this way, the objectives of the evaluation can be known: the institution’s strengths and weaknesses, and opportunities and threats in relation to its future development; but little or nothing is known about the research sequence needed to achieve these things; the methodologies chosen; the tools for gathering information; data analysis strategies to use; or the quality criteria for the data collection instruments to be employed, for example.

2) Take into account objective indicators so as to be able to measure them, but at the same time solicit observations, opinions and evaluations that have been contrasted (evidence) and apply them equally according to the starting point of each scenario evaluated.

We refer to the desirability of continuing to maintain the objective indicators contemplated by the UQP guide and the model of indicators proposed by De Miguel (1999b), but complementing both aspects by contributions that can generate more qualitative strategies. The emphasis on a greater role for performance indicators (quantitative) in the evaluation process seems more geared toward replacing the assessment processes themselves, as value judgments contextualized and shared by the agents, by bureaucratic and conclusive procedures that would pay little attention to the environment, to the specific characteristics of each unit, and to the opinion and participation of those involved (Apodaca, 2001, p. 371). With this methodological orientation transparency and clarity are gained, although it is possible to confuse statistical data with real performance indicators which should perhaps refer to goals, values, context and specific times (Dochy et al., 1990b). Frackmann (1991) goes further, indicating that there is a risk in thinking that the institutions’ objectives are the same as indicators. It is therefore appropriate to consider the value judgments and the evidence (quantitative data, documents, opinions, etc.) of the agents involved in the evaluation process, as well as the objective indicators it has been decided to contemplate (Shadish, Cook and Levinton, 1991, Apodaca and Grau, 1996).

3) Establish technical criteria for the selection of measurement indicators.

The choice of performance indicators in institutional university evaluation must be supported by a number of technical criteria such as those proposed by Cave, Hanney, Henkel and Kogan (1988) and Tognolini (1991):

• Type of indicator: input-process-output;
• Relevance: objectives and mission of the institution or department;
• Clarity: avoid ambiguity;
• Impossible to manipulate;
• Cost of information collection and ease of comparison;
• Level: department, discipline and teacher;
• Relationship with other indicators.

In addition, Osoro and Salvador (1994) propose the following:

• Consistency: a balance between indicators of teaching, research, and services;
• Reliability: valid and reliable in other institutions or departments;
• Permanence: valid and reliable over time.

4) Provide the evaluatory organizations with executive powers and capacity for liaison (after the results were known), so that they can implement remedial measures, either in the line of accountability, improvement, or both.

The last phase of the evaluation model considered includes the development of an improvement plan which would include goals to achieve, actions that must be taken to achieve them, estimates of control systems, and provision of necessary resources. So far, we do not know if an improvement plan has been either designed or implemented.

Higher education institutions, therefore, must be committed to going through transparent internal and external evaluations, conducted openly by independent specialists (Matthew, 2001, pp. 642-643). In this sense, the possible creation of the National Agency for Quality Assessment, proposed in the 2000 University Report,12 and the recent OLU contribute to the effective implementation of improvements through an accreditation process centralized in a body independent of the university. In this regard, authors such as De la Plaza y Peces-Barba are of the opinion that this independence is insufficient for excessive patronage of the central government, and the opacity of its creation process (“Creation of the National Assessment Agency”, 2002).

In any case it seems reasonable, as Mora (2001, p. 393) indicates, to undertake courses of action in parallel with existing assessment processes:

• Establish an accreditation system that would lead to the classification of degrees;
• Create a system of public information about degrees (indicators);
• Establish a financial mechanism to improve quality;
• Organize a system of institutional certification.

The accreditation model makes a distinction between the terms certification and accreditation. The first (certification) alludes to the process which guarantees that an organization meets the requirements for quality assurance; the second (accreditation) directly affects the validation process of certification agencies.

Contrasting with this model is that of the institutional evaluation, having aims similar to the previous one (accreditation model), but with obvious differences: it is oriented toward making judgments, not verdicts; it is not always determined by standards, and it is basically situated in the framework of self-regulation generated within the organization. Thus, the fundamental difference in the two procedures may be considered in terms of standards as contrasted with purposes.

It is not the same thing to evaluate an institution or academic program in terms of quality standards established by certain agencies, as it is to estimate the adaptation of the processes and the quality of the results obtained by an institution based on the objectives it proposed to achieve. (De Miguel, 2001, p. 399)

The two models are compatible, and can be established in conjunction with the use of performance indicators and by setting up a financial mechanism to improve quality. In both cases, the undesirable effects of a misuse of performance indicators and funding should be considered, with quality as a goal. On the production, use and interpretation of indicators we have previously developed a complete dissertation. With regard to funding as a quality incentive, we find ourselves with the option called strategic use of funding (Villarreal, 1998), which answers to the growing government concern about increasing the effectiveness and efficiency of the universities’ functioning (Wagner, 1996). This concern can be channeled through different strategies. Villarreal (1998, pp.170-171) propose the following:

1. The use of incentives linked to performance level, measured in relation to the achievement of certain objectives (quality indicators) in funding programs accessible to all universities;
2. The use of bilateral programs-contracts in which the administration determines the objectives to be achieved by each individual university, and the funding linked to these objectives;
3. The establishment of competitive funding programs. The complementary resource is linked to competitive processes, with an objective evaluation process, and with a monitoring process that would guide in determining whether to continue allocating such resources or not.

Legislative references

Ley Orgánica de Universidades 6/2001, de 21 de diciembre de 2001, de Universidades. Boletín Oficial del Estado No. 307 (December 24, 2001).

Real Decreto 1947/1995 de 1 de diciembre, por el que se establece el Plan Nacional de Evaluación de la Calidad Universitaria, Boletín Oficial del Estado No. 294. (December 9, 1995).

Real Decreto 408/2001 de 20 de abril, por el que se establece el II Plan de Calidad de las Universidades, Boletín Oficial del Estado No. 96 (April 21, 2001).

1For example, in Mexico through the ANUIES document called “Higher Education in the Twenty-first Century. Strategic lines of development” (Rodríguez Gómez, 2000).

2Faculties, governing boards, other secondary organizations (staff meetings, departmental councils), and finally, the commissions set up under the latter (teaching, economics, etc.)

3Approved January 20, 2002.

4Whether because of each other’s political color, or because of the evident disconnection between the university and the regional and central governments.

5If they had acted thus, it is possible that the OLU might not be so controversial today. In this sense we quote the words of Gurrutxaga Ander Abad, Basque Government Deputy Minister of Universities and Research (Gurrutxaga, 2001):

The ministry has prepared the draft of a law that aims to transform the structure of the university without asking, without listening to, and without consulting the Basque Autonomous Community, which paradoxically must manage and finance the cost of this law.

6Self-directed study and external evaluation based on peer review consistent with the methodology used in the general framework of most of the countries of the European Union.

7On the subject of this aspect there are authors like De Miguel (1999a, p.110), who believes that: Most of the reports prepared have not been disseminated, even though among the recommendations specified in the PNECU there was established the need to publicize all the documentation generated, in order to give credibility to the evaluation process carried out.

8The Unit for the Quality of Andalusian Universities (UCUA) has launched for the academic year 2002-2003 one such initiative that gives priority to a number of degrees in different universities of Andalusia, committing itself to a funding structure that would make viable the implementation of improvements.

9Processes that are currently developing in various specialties of the University of Granada.

10Type of attitude scale consisting of a collection of items or phrases related to a dimension or feature, and about which are emitted judgments of agreement or disagreement. It is usual to contemplate five possible options of evaluation, which are distributed in a bipolar continuum from the least to the greatest amount of agreement or vice versa.

11The interested reader can find a comprehensive and detailed description of the types of indicators referred to in the PNECU (National Plan for Evaluation of Quality) on the website: www.mec.es/consejou/indicadores/index.html.

12In essence, the University Report 2000 is an extensive catalog of recommendations on various aspects of Spanish university life; it has taken as a reference earlier reports such as: France’s Attali reports, Britain’s Dearing report, or the USA’s Boyer report. The newest recommendations articulated in the document are those dealing with a new system of accreditation for universities and degrees; another new, more flexible system of studies; the reinforcement of public funding; increased scholarships; the modification of the system of access for university teachers; government and administration; and the creation of companies that exploit university research.