Revista Electrónica de Investigación Educativa
Vol. 5, No. 1, 2003
Relationships Among Student-body Composition, School
Process, and Mathematics Achievement in
Argentina’s High Schools
Rubén Alberto Cervini Iturre
racervin@infovia.com.ar
Departamento de Ciencias Sociales
Universidad Nacional de Quilmes
Salguero 2460, 5º Piso
Buenos Aires, Argentina
(Received: December 18, 2002; accepted for publishing: March 11, 2003)
This is a study of the relationships between the student-body composition of the school, some characteristics of the scholastic process—culture and school climate—according to learners’ perceptions, and the mathematics achievement of students in the last year of high school in Argentina. The data used came from the 1998 National Census of High School Completion, carried out by the nation’s Ministry of Culture and Education. The file contains data for 135,000 students of 2,708 schools in 25 states. Multilevel linear modeling with three levels (student, school and state) was applied. A strong relationship was detected between mathematics achievement and the variables student-body composition and school process. When both variables acted together, the effect of other variables experienced a pronounced descent. Although reduced, the variables of the process influenced the student’s achievement. There was identified for future works a reference model which would evaluate other institutional learning factors.
Key words: Scholastic achievement, high school education, institutional factors, school climate, scholastic culture.
Recently, several studies have questioned the validity of the conclusions of a great part of the research on scholastic effectivity (Angus, 1993; Coe and Taylor, 1998; Gerwitz, 1998; Gibson and Asthana, 1998; Hatcher, 1998, cited by Thrupp, 2001b; Slee, Weiner and Tomlinson, 1998; Thrupp, 2001a; 2001b), and have generated, in turn, reactions of some of the most visible of this type of study (Daly and Ainley, 2000; Goldstein, 1998; Teddlie and Reynolds, 2001). Up for grabs in the debate is the credibility of the messages that endorse scholastic efficacy and identify factors of institutional effectiveness. In connection with the studies on the “determination of the existence and magnitude of school effects” (Teddlie, 1994b, p. 89) through regression analysis, which are different from the case studies whose interest is to determine the characteristics or processes belonging to singular, effective schools, three methodological topics have acquired relevance in this debate. These are the inclusion of contextual variables, the conceptualization and measurement of the culture and institutional climate as aspects of the scholastic process, and the technique used to analyze the data.
The results of investigations will undoubtedly vary according to the treatment given these three aspects. This study is guided by the idea that to demonstrate the effectiveness of any characteristic of the schooling process (policy, organization, teaching practice), there is required a reference model (control or adjustment). This must include not only measures of individual student background (social origin, gender, academic history), but also and principally, many different indicators of the school’s composition; and measurements of the culture and school climate based on attitudes, expectations and perceptions of the school’s own students. In addition, the data must be analyzed with the technique of hierarchical linear models—the only means available for overcoming the technical problems of traditional methods of regression.
In this paper my aim is to illustrate these criteria and to identify the model of reference for secondary education in Argentina, as based on the 1998 National Census of High School Completion, conducted by the nation’s Ministry of Culture and Education. To this end, we have explored the relationships between the students’ achievement in mathematics, the composition of the student body, and some characteristics of the culture and the school climate, according to the perceptions of the students themselves. The results obtained will serve for future studies oriented toward evaluating the measurements concerning the school’s institutional life, according to the information provided by school principals.
The hypothesis that school climate relates to educational results has a long history (Scheerens and Bosker, 1997). Much research has been concentrated on the effects of the classroom’s social climate on students’ cognitive and affective development (Angell, 1991; Dunn and Harris, 1998; Fraser, 1989, Waxman and Ellett, 1992). Other studies, no less numerous, have focused on the effects of institutional climate or culture (Anderson, 1982; Purkey and Smith, 1983). Recent critiques of the studies on effectiveness have questioned the validity of a great part of this background research. At least three methodological aspects could be considered to assess such validity in the correlational studies: (1) measurement of context or composition of the student body, (2) concept and measurement of culture or institutional climate and (3) technique used to analyze data.
1.1. Composition of the student body
One of the main criticisms of the tradition of scholastic effectiveness studies is that it has not given enough attention to the effect of the school’s social context, giving the impression that the school acts independently of such determination (Slee, Weiner, and Tomlinson, 1998; Thrupp, 2001b). On the other hand, Teddlie and Reynolds (2001), believe that since very early (Coleman Report) school effectiveness studies have addressed the impact of social class on student achievement, distinguishing between “the individual socioeconomic level of the student, and the effect of the group of students” (p. 53), and have never claimed that school effectiveness was independent of the student body’s composition (p. 56). In fact, the hallmark of the authors of school effectiveness is believing that schools can have an impact beyond that of social class, rather than being obsessed with the relationship between social class and student achievement (p. 54).
A decade ago, Kreft (1993) indicated the existence of two alternative theoretical focuses regarding the causes of effective school climate, that is, leading to improved scholastic achievement. The first approach argues that the climate is the result of the institution’s organization and policies (type of school) and that, moreover, it molds the behavior and the attitudes and expectations (scholastic ethos) of the student, immediate conditioners of their level of scholastic achievement (school climate hypothesis); in other words, the joint effort of principal, teachers and parents—who agree on policies and standards of education—creates an effective school climate.
According to the second focus, however, school climate, like other institutional variables, is the result of the composition of the school’s student body, a product of institutional selectivity (school composition hypothesis). The students’ attitudes and individual behavior depend on the type of student attracted to or selected by the school, which molds its composition and determines the student climate. Then, higher expectations of achievement and an environment of ordered learning are the result of the makeup of the student body, and not of polices or directed institutional interventions. In summary, neither of the two theories questions the association between institutional climate and scholastic achievement, but while the first says that the climate models the attitudes and behavior (ethos) of the student, the second considers this to be a result of these last two variables and student-body composition. The students create their own school climate.
This dilemma appears in the analysis of any aggregation of the educational system (classroom, school, district, state, etc.), although its individual components change. In the classroom, the specific element is the instructor’s teaching practice. For the focus of educational effectiveness, “successful schools are those that can achieve effective classrooms (Creemers, 1994, p. 201), and ultimately, teachers are the crucial factor for education in the classroom” (p. 203). Through his/her pedagogical practice, the teacher can generate an atmosphere tranquil, orderly, and oriented toward learning. A prolific history of research has identified many different behaviors of the teacher, both concerning classroom management and teaching itself, supposedly effective in achieving that objective. There have also been proposed theoretical models that incorporate and synthesize part of those empirical findings, frequently based on the key concepts of Carroll’s model (1963) (Creemers, 1994; Slavin, 1996), or that integrate the classroom level with upper strata of grouping (Scheerens y Creemers, 1989; Teddlie, 1994a). Regarding the latter, the review of several studies leads us to conclude that, by means of various mechanisms, “teachers in the most effective schools conduct the classroom with a more positive climate and with higher expectations for learning than do teachers in less-effective schools (Tedlie, 1994b, p.128). A recent study by Yasumoto, Uekawa and Bidwell (2001) confirms the argument for integration of different levels—department (or subject area) and classroom—in the high school. According to their data, the authors conclude that through the interactions of teachers in the same department in each school, there is created a pedagogical culture of “the effective way to teach and manage the classroom” (p. 183), producing greater pedagogical consistency, and intensifying its effect on students’ learning.
In the alternative focus, by contrast the composition of the makeup of the group of students in the classroom influences the teacher’s style, whether because teaching practice (quality and quantity) varies according to his/her expectations about the group’s learning ability, conditioned in turn by its aptitudinal composition (Barr and Dreben, 1983; Gamoran, 1987); because of the processes of group reference; or because of both factors (Hallinan, 1988).
While correlational studies can hardly resolve in conclusive form, the impasse between school climate and student-body composition, at least there is consensus around the idea that the validity of the conclusions of any study depends on the inclusion of good and various measurements of the school’s socio-economic context. In reality, it is of little interest that “for every school-impact study that has indicated weak ‘compositional’, ‘contextual’, or ‘peer group’ effects’ on students’ achievements, there has been another that indicates a much stronger impact of student-body composition” (Thrupp, 2001b, p. 24). Nor is it of interest to scrutinize the impact of the context of scholastic achievement, if this is interpreted by means of an organizational or sociological theory, such as Bourdieu’s theory of cultural reproduction (Thrupp, 2001b, pp. 27). From the methodological point of view, the problem can be reduced to the following question: has the data analysis included good measurements of the school’s social context, in addition to the individual student’s social background? That is, are the estimates of the scholastic effect “adjusted” by the student’s social background and the school’s socioeconomic composition? Do these measurements capture diverse aspects of the students’ socioeconomic background? The theme continues to be valid for the existence of recent studies where this condition is not fulfilled. In brief, all research on scholastic factors should reflect the face-off between the effect of student-body composition and scholastic effect.
Other conclusions related with the above also seem to have a certain consensus: (a) in the more developed countries, the differences in scholastic achievement explainable by the school (interschool variance) is notably less than those attributable to the extra-scholastic factors (intra-school variance) (Teddlie and Reynolds, 2001, p. 54; Thrupp, 2001a, p. 448). (b) At the same time, the effect of context variables is always greater than the individual; and (c) the factors of effectiveness can vary according to the socioeconomic context (Teddlie and Reynolds, 2001, p. 59; Thrupp, 2001b, p. 24). It seems appropriate, therefore, always to include the analysis of interaction with contextual measurements. Finally, there are numerous studies which support the hypothesis of an effect relevant and significant for Intellectual Composition or Composition by Background Achievement (Teddlie and Reynolds, 2000) regarding scholastic achievement in primary-school education (Leiter, 1983) and high school (Resh and Dar, 1992; Opdenakker and Van Damme, 2001).
1.2. Culture and school climate
The classical focus defines school culture as systems of attitudes, values, norms and meaning shared by the members of a school (Tagiuri, 1968). However, the postmodern critique of the idea of a unique culture, static, as opposed to a dynamic and heterogeneous concept with different cultures competing, confronting each other, and negotiating (McLaren, 1991; Quantz, 1988) has revealed the importance of considering all the “voices” existing in the institutional environment, including the student body. In this regard, Thrupp (2001b) has observed that school effectiveness studies tend to use “notions of school culture that emphasize the organizational, administrative (management), and instructional dimensions of the school at the expense of the student and community culture”, when in reality, the school culture should be visualized as the result of a “negotiation with the students on the basis of levels of docility, motivation and aptitude according to social class, which in turn, is related to the vision of the student regarding the school and his/her probable occupational future (Thrupp, 2001b, p. 26). In this perspective, “school cultures are the product of the interaction between the official culture and that of the students” (Hatcher, 1998, cited by Thrupp, 2001b, p. 27).
When it comes to teaching practice, the predominant tendency is to lay the concept of culture exclusively on the shoulders of the teachers. During the last two decades, numerous studies have promoted the idea that the teaching culture is the result of the teachers’ interactions in school (Angelides and Ainscow, 2000), and consistent with this idea, it has been judged to be of instrumental origin. The culture through which the actors define present reality “is often a function of problems inherited from the past” (Hargreaves, 1995, p. 25). A recent study mentioned above (Yasumoto, Uekawa and Bidwell, 2001) is a paradigmatic example. The authors investigated the effect of the pedagogical culture of high school mathematics and science teachers regarding students’ achievement. It is assumed that teaching will make various and sometimes contradictory goal demands, while at the same time it is technically imprecise (Rowan, 1990; Weick, 1976). As a result, teachers seek to solve problems endemic to the classroom through informal day-to-day interactions with colleagues, supposedly focused on the everyday work of teaching. These clusters of interaction generate a convergence of teachers on the diagnosis of problems and how to solve them, that is, the effective way to teach and manage the classroom in that particular school. If in a department (or disciplinary area) there exist these interactive clusters it is more probable that a local culture will be generated—”a variety of social capital that will sustain vigorously normative discourses on good teaching practice, achievement of goals, procedures and teaching standards” (Yasumoto, Uekawa y Bidwell, 2001, p. 183), producing a greater collective consistency in the teachers’ pedagogical practice, and consequently intensifying the effect of this on the students’ achievement progress. In sum, the authors conclude that when teachers have close communication (social relationships), they share intense critiques of the teaching process (ethos), and are consistent in the way they teach (behaviors, teaching processes). The effects of their pedagogical practices on the progress of student achievement are intensified. However, in many other studies, no measurements of the sociodemographic composition were included, much less those for student culture, although there was information about the student’s personal characteristics (social origin, ethnicity, gender, and previous achievements) which could have been added to create variables of composition. Consequently, it is impossible to know how much of the effect of the teacher’s culture is superimposed on the student-body composition, or if it is an indirect effect of that composition, or how much is simply the adaptation of the teacher to the cultural ethos of his/her students.
Beyond the theoretical relevance of disputing the various voices or cultural perceptions, the imperative for including the student’s perceptions is based on two well-founded reasons. First, the student is the subject of a set of scholastic attitudes and expectations, the result of the dynamic and conflictive process of the conversion of cultural capital (provided by his/her family social origin) into scholastic capital (Bourdieu, 1988; Bourdieu and Passeron, 1981) through scholastic action. Motivation and a positive attitude toward the school and academic knowledge are part of this set, and are related with scholastic achievement. According to Carroll (1963), the degree of learning is based on the time the student actually spends on a particular learning task, and the time the task requires. If the time assigned by the instructor to the teaching of a specific task, the aptitude of the student, and the quality of the teaching are maintained constant, then learning will depend on the student’s perseverance or effort (the time the student is actively involved), which in turn depends not only on the student’s motivation—a mixture of the cultural capital inherited from his/her family, and of previous school experiences—but also on incentive, “the product of specific strategies for increasing motivation” (Slavin, 1996, p. 8) which the teacher adopts in the teaching/learning process.
Second, the review of studies has established that the behavior of a given factor may vary according to the source of information. Stringfield (1994) has observed that the measurements of school climate based on the perceptions of the principal, the teachers and the students are not necessarily parallel. Generally, the perceptions of the first two are more closely associated with socioeconomic status than with student achievement, while measurements based on the students’ perceptions are more independent of the socioeconomic level. Therefore, the author suggests that “future large-scale studies on scholastic effects” should no longer ask teachers or principals, but should” ask students directly about their school climate” (Stringfield, 1994, p.68.) In sum, both the attitudes and behaviors of students (ethos) and their perception of the school climate, seem theoretically and methodologically inescapable when one wishes to evaluate any factor of school effectiveness.
There are different approaches to the relationship between the concepts of culture and climate. In Tagiuri’s proposal (1968), culture is part of the institutional climate, a broad concept that includes aspects such as physical environment, actors’ demographic characteristics (socio-economic composition), organizational structure and operational procedures (system social). Moos (1979), however, in another focus prefers to limit the expression of social climate to social behavior (sociodynamic aspect of the classroom environment), which together with the characteristics of the physical environment (ecological) and the group of students (social origin, educational background, gender, etc.) make up the ecological environment of the classroom.
In any case, there is assumed a close overlapping and causal relationship between the regulatory structure and behavior of individuals. Behaviors indicate the existence of rules and therefore “a scrutiny of practices allows us to reveal the hidden cultural assumptions that guide those practices” (Angelides and Ainscow, 2000). So for example, Yasumoto, Uekawa and Bidwell (2001) see density of interrelationships and consistency in teaching practice (observed) as indicating the existence of a normative culture pertaining to good teaching practice. Or, the best school climate is based on a consensus and a morality shared by its members (Bryk, 1988; Coleman, Hoffer and Kilgore, 1982a 1982b). Indeed, measurements of culture tend to include both dimensions. For example, the School Culture Scale applied to students (Power, Higgins and Kohlberg, 1989) leads to four factors or subscales: normative expectations concerning students’ behavior (e.g. discipline); the quality of student-teacher and student-school relationships; and student perception regarding the educational opportunities provided by the school.
Finally, whatever the concept adopted, there seems to be an unquestionable need to include indicators on the subject of the physical environment. In the educational setting, this dimension should consider not only the operative aspect (infrastructure, maintenance, etc.) and furnishings, but also the availability of resources and teaching aids. The debate on the importance of these requirements has a long history beginning with the Coleman Report, in developed countries. The greater homogeneity of material provisions in these countries explains, in great part, why the effect estimated on performance for this type of variable is often non-significant. It is therefore quite common in those countries for investigations on effectiveness to lack measurements of the level and quality of material resources. For Third-World countries, however, the pronounced institutional heterogeneity in the offer of school supplies urges that such measurements be included.
1.3. The analysis technique
The technique of multilevel analysis (or hierarchical linear models) was developed recently to address the serious technical problems that arose when the traditional method of ordinary least squares (OLS) was applied to the analysis of data with a “hierarchical clustering structure” (Aitkin and Longford, 1986; Bryk and Raudenbush, 1992; Goldstein, 1987), as is the case of data in the education sector (students are part of a classroom, which in turn is part of a school situated in a district within a state, etc.) So for example, the new correlational technique is more appropriate for analyzing variations in the mathematics achievement of students who are members of a school, which in turn is part of a state, allowing the decomposition of one variable (performance) in components within the group (intra-school; inter-state) and within a group (inter-school, inter-state) and for analyzing the association between variables in these different levels of aggregation (characteristics of the student, classroom or school). Given these advantages, the new methodology has been widely accepted, and its use has experienced a sustained expansion. Some studies (Ridell, 1993, 1997) have shown that the application of the traditional technique (OLS) and the multilevel to the same data usually produces different results. If this is true, the findings and results of a large part of school effectiveness studies should be placed in parentheses.
In summary, review of the current debate surrounding the research proposed to identify factors of effectiveness demonstrates the importance of addressing certain methodological criteria. It is important to include various and different measurements of the student’s individual records, but it is also necessary to transform these variables into measurements of context or student-body composition, and to include them in the analysis along with other contextual measurements from different sources; for example, variables relating to the physical environment and availability of school resources in the establishment. For the study of institutional, classroom and teaching practice it is advisable to include measurements based on students’ own perceptions of the dynamics of these aspects of school reality, as well as their attitudes, expectations and school behaviors (student ethos). Finally, the analysis technique recommended for this type of data is that of multilevel models (or hierarchical linear models); especially when it has to do with educational systems with strong inter-institutional variations, such as in the case of Argentina. This work is guided by these criteria.
II. Objectives and Methodology
2.1. Objectives b) The student-body composition variables are scholastic averages of each student’s individual variable except gender and repetition (dummies), defined as scholastic percentages. They are named using the same acronym as the individual variables, except for adding the ending _e. For example, libro_e is the average number of books in the school. Within the concept student-body composition are also included two variables on school resources and the physical environment of the school (Tagiuri, 1968; Moos, 1979); both come from the school principal’s questionnaire: c) The school process variables are the results of gathering—scholastic level—information provided by the student questionnaire. The questions have to do with attitudes regarding mathematics and the school, behavior in school, and the student’s perception of the school and the classroom. A majority of the questions have a four-point (Likert-type) scale as a response option. To reduce the numbers of variables, questions relating to attitudes towards mathematics and the classroom were subjected to a principal component analysis with varimax rotation, yielding three factors. The same analysis was applied to the questions regarding (the attitude toward) evaluating the school, and obtained a single factor. In all cases, the variable is the sum of the scores of each question of the factor, after reversing the direction where relevant. Other measures consist of unusual or unique questions. This set of variables covers the two dimensions of the school or the classroom identified by the literature on school climate (Fraser, 1989; Moos, 1980): the physical and sociodynamic contexts. The first (student-body composition) refers to the characteristics of the group of students (social origin, gender, academic ability) configured by the type of student attending or attracted by the school (extracurricular determination) and school resources available in the school. The second (process) are measurements of the sociodynamic aspects, close to the idea of social climate (Moos, 1979) and school culture (Tagiuri, 1968). The measurements refer to certain characteristics that school effectiveness research has repeatedly confirmed, such as an “orderly atmosphere” at school (Scheerens and Bosker, 1997), “positive academic climate” (Stringfield, 1994) or “normative expectations regarding student behavior” (Power, Higgins and Kohlberg, 1989) —indiscipline, violence, dropout— “effective learning time (Scheerens and Bosker, 1997), “cordiality and availability shown by the teacher to the student” (Moos, 1987), and “remedial education” (Creemers, 1994) —teacher. There are included as well, measurements of the student ethos, a set of attitudes toward mathematical knowledge—motivation, valuation—of evaluation of the school experience or “perception of the learning opportunities offered by the school” (Power, Higgins and Kohlberg, 1989) —success—and of behavior in school—effort. Ethos is assumed to be the result of the joint determinations of family origin and the school’s action.
This study investigates the relationships between (a) the composition of the school’s student body; (b) certain features of the schooling process, according to students’ perceptions, and (c) the mathematics achievement of students in the final year of Argentina’s high schools, using the technique of multilevel statistical analysis with three levels (student, school and state), and data available from the 1998 National Census of High School Completion, carried out by the nation’s Ministry of Culture and Education. It has to do establishing whether the school culture and social climate (school process), according to the students of the school have a specific effect on mathematics achievement after controlling the effect of variables related to socioeconomic and cultural origin, gender, and students’ academic background, considering both the level of the student and that of the school (school student-body composition). The end result of this exercise will be used as a reference model in future studies aimed at evaluating institutional factors measured on the basis of the questionnaire answered by the school principal.
2.2. Data
We have analyzed data provided by the (1) mathematics test, (2) the student questionnaire, and (3) the questionnaire applied to the principals during the 1998 National Census of High School Completion.1 The evaluation was made at the end of the school year. Both questionnaires were self-applied. In the analysis were included only the students who took the mathematics test and answered the corresponding questionnaire, and whose principal also answered his/her questionnaire. Two of the three modalities of Argentina’s high school education were included: high school and business school.2 Furthermore, schools with less than 10 students were not considered. With these conditions, the file is made up of 134,939 students in 2,708 schools from 25 states.3
2.3. Variables
The dependent variable or criterion is the score (gross) obtained by the student on a standardized mathematics test. The independent variables are the characteristics of the student and of the school, and may be organized in three blocks:
a) Student’s individual variables refer to family financial capital, family cultural capital, gender, academic background and daily work hours. They are operationally defined as follows:
2.4. Technical analysis and strategy
For the analysis of the relationship between performance and the different variables, we used the technique called multi-level statistical analysis or hierarchical linear models (Aitkin and Longford, 1986; Bryk and Raudenbush, 1992; Goldstein, 1987). First, we analyzed the correlations between the characteristics of school process (Sp) among themselves; and those of the student-body composition (Sbc). Second, we estimated the null model (with no predictor) and the effect of each Sp indicator, which would indicate the initial relevance of each of these measurements. The third step was to estimate two models: one with all the variables of the Sp and another with all the variables of the Sbc. The objective was to compare the magnitude of the effect of both sets of variables on mathematics achievement. The fourth step incorporated the student’s individual variables into the two previous models. The objective was to evaluate the behavior of the variables of Sbc and Sp when controlled by individual student inputs. Then we analyzed a model with all the independent variables available. Thus, there was offered an idea of the relative importance of both sets of variables under study, i.e. the context and school process variables. Finally, there was developed an analysis of interaction between some of the student’s characteristics and the Sp. To analyze the data we used the computer program MLWIN (Goldstein, 1987). To estimate the probability of the effect of variables we used the test of the ratio of maximum probability.4
3.1. Relations between the variables of student-body composition (Sbc) and school process (Sp) Table I. Coefficients of correlation between the variables 3.2. Null model and school process variables (Sp) Table II. Results of multilevel analysis. Performance in mathematics 3.3. The effect of student-body composition and school process Table III. Results of multilevel analysis. Performance in mathematics with individual student variables and with process and student-body composition variables added 3.7. Analysis of interaction Table IV. Results of analysis of interaction between the scholastic process variables and some personal characteristics of the student (repetition and family education)
First, we explored the relationships between school composition variables (Sbc) and school process (Sp) through the coefficient of correlation (Table I). Some variables of the Sp maintain a high correlation with almost all the measurements of the Sbc. This is the case of success and dropout. Overall, then, schools in the higher socioeconomic level, with a lower proportion of repeaters and more institutional resources available, usually have a higher expectation of future success due to the school and lower (perceived) dropout levels. Although less intensely, effort behaves in the same way. The effect of gender composition is unique: it is not related with success and dropout, but it is related with effort; that is, the greater the percentage of women in school, the higher the average of task completion. Attitudes towards mathematics (academic knowledge) are those least associated with the Sbc variables. This behavior is consistent with the hypothesis that these attitudinal variables express in large measure, the effect of school action. It is also interesting to observe that there is no significant covariation between the perception of the indiscipline level, and of Sbc variables. The perception of the threshold of violence shows a greater adjustment to the Sbc, but much smaller than that displayed by dropout. If there is adopted the hypothesis, common enough, that the objective level of indiscipline increases with the decreasing economic level of the student, then the behavior of indiscipline suggests that the student’s sense of discipline changes according to the social composition. Therefore we see the weak association between the level of indiscipline and the Sbc also registered by international literature (Stringfield, 1994). Nor is the students’ perception of the effectiveness of teaching practice and their relationship with teachers significantly associated with the Sbc. We might ask whether the reversal hypothesis is also applicable to this variable.
of process and student-body composition
In the multilevel analysis, the first step is the initial partition of the performance variance in all three levels of aggregation (state, school and student).5 The set of these estimates is called the null model because it has no predictors at all. The results are presented in Table II. About 47% of the variation in performance is due to differences between regions and schools, although 34% belongs mostly to the latter. However, the results indicate the importance of including the state level of aggregation, since failure to do this would cause 13% of the total variation of performance to be incorrectly judged as inter-school variation. Based on a review of more than 40 Third-World investigations and using multilevel analysis, Ridell (1997) concluded that inter-school differences are smaller than the gap between students, that is, “the influence of the home is greater than that of the school” (p. 185). According to data presented by Ridell regarding the performance in high school mathematics, the average inter-student difference is slightly below 55%, an estimate similar to that reached in this work (56.5%).
There is introduced in this null model, each of the variables of the Sp. It can be observed that the estimates for all the measurements are significant when considered individually (Table II). According to students’ perceptions, the more intense the indiscipline, violence and dropout in the school, the lower the mathematics achievement. On the other hand, the more intense the motivation, valuation of mathematics, expectation of success (in what is learned at school), and the completion of homework, the greater the achievement. Finally, the more positive the student’s image of the quality of interaction with teachers of their effectiveness, the greater the achievement. In principle, then, the measurements behave according to the expectation, and in a manner consistent with international literature.
with variables of process and student-body composition added
In this step are analyzed two multilevel models: one with all the variables of the Sp (Model 1), and the other with the variables of the student-body composition (Model 2). The object is to evaluate the effect of the set of indicators on the variations of Levels 2 and 3, and not the behavior of each individual indicator. Let us note, however, that the two indicators of organized climate (indiscipline and violence) and valuation lose the high significance they had. This is probably due to the “colinearity” between the first two variables, and between the last and motivation.6 Nor are the variables property_e and educa_e significant; this has already been proved and analyzed in previous works (Cervini, 2002; 2003).
The effect of the set of Sp variables on performance in the three levels (Model 1) is very similar to that shown by the Sbc (Model 2). Each set of indicators, separately, explains half of the inter-school variation. The same behavior is observed concerning the geographic variation, although here the Sbc is slightly more effective than the Sp. The total unexplained variation has fallen about 20% in both models, indicating a high level of statistic significance.7
3.4. Contextual variables and individual student backgrounds
This step monitors the effect of Sp and Sbc by the measurements referring to the student’s personal background.8 The variable property is not included because a previous study (Cervini, 2002) demonstrated that it is dispensable when acting together with other variables. Extracted as well are educa_e and property_e, also dispensable (Model 2). The % female was redundant too, and therefore removed from the subsequent analysis. Estimates of the variables of the Sp (Table II, Model 3), dropout and success show a sharp decline, while the rest decline slightly. Furthermore, we observed that the decrease in inter-school and inter-state variations are of the same magnitude as that experienced by the intra-school (or inter-student) variation. They indicate the high socioeconomic and aptitudinal selectivity of students, especially at the school level. Regarding the Sbc variables (Model 4), there is shown a drop in the estimates, due to the selectivity of the system, but now the rest of the inter-school inter-state and variations are not altered. This result was to be expected, and indicates that the effect of the individual variables of the state and school is completely mixed up with that of the Sbc variables.
3.5. Final model, complete
At this stage the three groups of variables were analyzed together. None of the estimates of individual student variables was altered; that is, all retain their significance. In general, both the estimates of the Sbc and the Sp decreased, although the former did it in a more pronounced fashion (Table III, the complete final model). Measurement of family and institutional teaching resources and school physical environment lost significance.9 The effect of the composition of repeaters also fell sharply.
On the side of the Sp, the perception of school dropout and the Sp variables, the perception of ESL and expectation of future success (evaluation of the school) show the greatest changes, while motivation toward mathematics and finishing homework were almost unaffected. The two measurements of perception of order could not be clearly evaluated because of the existence of “colinearity”, but the comparison with the estimates presented in Model 1 indicates that they are very little affected. Finally, effort and teacher maintain the same level of significance. The pronounced decline in dropout and success were expected because of the high correlation both have with all the Sbc measurements (Table I), with a probable overlap of their effects on performance. By contrast, attitudes towards mathematics, having low correlations with the variables of the Sbc, maintain their own effect.
These results indicate that, overall, an important part of the effects of student-body composition and school process overlap. In spite of this, the majority of the estimates of Sp variables remained significant even after considering the variables of student-body composition; that is, they maintain their own effect on student performance. Thus, for example, the highest performance a hypothetical student would get by switching to a school with a lower percentage of repeaters and with the same characteristics of student-body composition would decrease if in his/her new school, process characteristics were less positive than those in the average schools. On the other hand, the effects of some Sp indicators are more independent of the Sbc variables. This is the case of attitudes toward mathematics.
Beyond the statistical significance of each of the estimates, the most important question concerns how much the Sp variables add to the prediction of performance once the Sbc variables are taken into account. The set of Sp measurements manages to reduce the rest of the state level (.049), and school (.179) (Model 4), to .047 and .157, respectively, i.e. approximately 2.5% of the total regional variation (state) and institutional (school).
3.6. Final model, reduced
In this model the redundant or dispensable variables have been removed. Its value is merely instrumental, since it will serve as a reference model for subsequent analysis and for future research, as stated in the objectives.
Finally, interactions were explored between the effects of Sp variables and that of two indicators of individual student records, namely the student’s academic background—repeater—and the educational level of the family—education. The final model, reduced, was used as reference.10 First, there were included all the terms interactive with repeaters. The results are presented in Table IV, Model 7. Only two terms are highly significant. The positive sign of the interaction dropout x repeater indicates that in the institutional context where dropout (perceived) is higher, and the average performance notably lower, there was verified a shortening of the distance between the performances of repeater and non-repeater students. This drawing closer is due to the fact that the decline in average performance of the non-repeaters is more pronounced than that of the repeaters. Moreover, the interactive term effort x repeater indicates that in the contexts where the level of doing homework is low, the distance between the performance of the repeaters and the non-repeaters diminishes, because of the average performance of the non-repeaters. The effect of the parents’ education, one of the student’s indicators of social origin, interacts significantly with an indicator of the school culture of the student body—success— (Table IV, Model 8). According to the results, as the positive attitude toward the benefits of future schooling increases, the effect of the student’s social origin on performance decreases. This behavior suggest the redistributive ability of some aspects of the student ethos regarding achievement in mathematics. Model 9 shows the estimates of the significant interactive terms in Models 7 and 8, but now acting together on performance. No important alteration in the estimates was observed, and therefore, the previous conclusions were maintained. Finally, it is important to observe that the inclusion of the interactive terms has produced no reduction in those remaining (unexplained variation). Thus, they help in understanding the relationships between the effects, but do not increase our predictive power on performance.
The particularity of this study has been to explore the association between indicators referring to different aspects of the schooling process (Sp), as perceived by students, and math performance of students in the final year of Argentina’s high schools. Apparently, the results of the initial analysis of these relationships validated the thesis of the school effectiveness approach. All measurements available on the institutional life of the school showed a significant correlation with performance. Thus, when the institutional environment is more “orderly”, the scholastic ethos of the students is stronger or teacher-student relations are of better quality; then the level of performance will be higher. This set of variables, without any “adjustment”, seems to explain 50% of the differences in total average performance between schools, an extremely high estimate.
However, this conclusion did not hold when the analysis went deeper, that is, when other relevant factors were considered. First, it included a heterogeneous set of variables related to the student, such as family economic and cultural capital, gender, length of work hours, and academic history. Thereafter, all these measurements were added to the school level, and indicators of student-body composition (Sbc) were obtained, supplemented with measurements referring to the physical environment and the availability of resources and scholastic aids in the school (principals’ questionnaire). All the individual and group variables were integrated into the analysis with the objective of evaluating their relationships with performance and Sp variables.
The analysis confirmed previous findings (Cervini, 2002). First, the student’s personal variables principally affect the inter-school variation and not the intra-school variation, which can be interpreted as the operational expression of a high level of social selectivity or segmentation of the educational system’s institutional network. In other words, schools tend to be very homogeneous, socially. Second, the family and contextual cultural capital is what shapes the profile of the distribution of school achievement, and not the family’s financial capital. The action of this last comes first, and determines the possibility of accessing and reaching the final year of high school education. Third, the effect of student-body membership on performance is greater than that attributable to the set of indicators for the student’s personal and family background.
To this previous knowledge, this analysis contributed new inferences about the behavior of mathematics achievement in high school education. It was confirmed that, although slightly, the effect of the Sbc on performance is greater than that exercised by the Sp measurements. It was also verified that there are important relationships of some Sp variables with those of the Sbc. The valuation of what is learned in school in general—one aspect of the existing student culture, and the (perceived) level of school dropout—an indicator of the positive learning environment—are highly correlated with all the Sbc indicators studied, except for gender. The attitude toward typically scholastic mathematical knowledge does not behave in the same way. The low association with the Sbc could be reflecting the effect of the school over time. On the other hand, the perception of the level of indiscipline is shown to be much more independent of the Sbc variables than the perception of the violence level. These confirmations suggest that the relationships between student-body composition and process can experience important variations, according to the specific aspects considered, and how they are measured.
When the variables of Sbc and Sp were included together in the analysis, most estimates of the variables in both groups dropped significantly. Other studies have confirmed empirically that “a substantial portion of the variance at school level is caused by a combined effect of characteristics of the school process and student-body composition” (Opdenakker and Van Damme, 2001, p. 422). From this behavior there is inferred the inescapable requirement to include student-body composition measurements in studies of school effectiveness, as recently acknowledged both by proponents and by opponents of this type of studies. In addition, the empirical behavior and the high correlations between some variables of the Sp and the Sbc make reasonable the hypothesis of a complex structure in the determination of scholastic learning, where part of the total effect of some features of the Sbc on performance could be direct, while the other could be through certain characteristics of the school process, or indirectly.
Either way, the data support the conclusion that certain characteristics of the school process have a specific effect all their own, in other words, not everything is attributable to the direct and indirect effect of the Sbc. However, the magnitude of this effect is less than the optimistic expectations of some speeches on school effectiveness. In this conclusion, the present study converges with the findings of recent research at the international level (Scheerens and Bosker, 1997; Opdenakker and Van Damme, 2001).
We identified two significant interactions, one referring to the students’ academic record (repetition); the other to their social background (parents’ education). Regarding the first, it was confirmed that performance differences between non-repeater students, and repeaters (more/less academic aptitude) decrease in schools with evidence of negative academic climate (high dropout and low effort or performance of the students), and that this decrease is mainly due to the decline of the average performance of non-repeater students. Or put another way, schools with positive academic climate get better results by extending the distance between the two categories of students, benefiting non-repeaters. The second interaction, highly significant, showed that in schools where the average positive valuation of student learning (expectation of future success) is high, the effect of student social background (parental education) on performance is reduced. Of two schools having the same Sbc, in the one where the future benefits of schooling is more highly esteemed (on the average), there are registered higher achievements. Then, while the first interaction speaks of greater inequality (referring to aptitude), the second identifies results of equity (referring to social origin), demonstrating the complexity of the determinations in the educational system.
The study has demonstrated the effectiveness of the measurements constructed based on the student questionnaire, both referred to as the Sp and the Sbc. The results obtained indicate that any future study designed to identify factors of institutional effectiveness through information from other sources (e.g. questionnaire applied to the principal), should include measurements largely unexplored in this work. Moreover, over 45% of the differences in average performance between schools—15% of the total variation of performance—has been unexplained. To attribute this variation a priori to the Sbc or Sp is methodologically unacceptable. The failures of measuring both concepts may be similar. Only through additional analysis, incorporating new measurements, will it be possible to increase the understanding of the unexplained remainder.
Aitkin, M. & Longford, N. (1986). Statistical modeling issues in school effectiveness. Journal of the Royal Statistical Society, 149 (A), 1-42. Cervini, R. (2002). Desigualdades socioculturales en el aprendizaje de matemática y lengua de la education secundaria en Argentina: Un modelo de tres niveles. Revista Electrónica de Investigación y Evaluación Educativa, 8 (2), 1-25. Retrieved October 24, 2002, from: http://www.uv.es/RELIEVE/v8n2/RELIEVEv8n2_1.htm Cervini, R. (2003). Diferencias de resultados cognitivos y no-cognitivos entre estudiantes de escuelas públicas y privadas en la education secundaria de Argentina: Un análisis multiLevel. Education Policy Analysis Archives, 11 (5). Retrieved February 3, 2003, from: http://epaa.asu.edu/epaa/v11n6/ Coe, R. & Taylor, C. (1998). School effectiveness research: criticism and recommendations. Oxford Review of Education, 24 (4), 421-438.Coleman, J., Hoffer, T., & Kilgore, S. (1982a). Cognitive outcomes in public and private schools. Sociology of Education, 55 (2-3), 65-76.
Anderson, C. S. (1982). The search for school climate: A review of the research. Review of Educational Research, 52 (4), 368-420.
Angell, A. (1991). Democratic climate in elementary classroom: A review of theory and research. Theory and Research in Social Education, 19 (3), 241-266.
Angus, L. (1993). The sociology of school effectiveness. British Journal of Sociology of Education, 14 (3), 333-345.
Barr, R. & Dreeben, R. (1983). How schools work. Chicago: University of Chicago Press.
Bourdieu, P. (1988). La distinción. Criterios y bases sociales del gusto. Madrid: Taurus.
Bourdieu, P. & Passeron, C. (1981). La reproducción. Elementos para una teoría del sistema de enseñanza. Barcelona: Laia.
Bryk, A. (1988). Musing on the moral life of schools. American Journal of Education, 96 (2), 256-90.
Bryk, A. & Raudenbush, S. (1992). Hierarchical linear models for social and behavioral research: Applications and data analysis methods. Newbury Park, CA: Sage.
Carroll, J. B. (1963). A model of school learning. Teachers College Record, 64 (8), 723-733.
Coleman, J. Hoffer, T. & Kilgore, S. (1982b). High school achievement: Public, catholic and private schools compared. NY: Basic Book.
Creemers, B. (1994). Effective instruction: An empirical basis for a theory of educational effectiveness. In D. Reynolds, B. Creemers, P. Nesselrodt, E. Schaffer, S. Stringfield, & C. Teddlie, Advances in school effectiveness research and practice (pp.189-203). Oxford: Pergamon.
Daly, P. & Ainley, J. (2000). Recent critiques of school effectiveness research. School Effectiveness and School Improvement, 11 (1), 131-143.
Dunn, R. & Harris, L. (1998). Organizational dimensions of climate and the impact on school achievement. Journal of Instructional Psychology, 25 (2), 100-114.
Fraser, B. (1989). Twenty years of classroom climate work: Progress and prospect. Journal of Curriculum Studies, 21 (4), 307-327.
Gamoran, A. (1987). The stratification of high school learning opportunities. Sociology of Education, 60 (3), 135-155.
Gerwitz, S. (1998). Can all school be successful? An exploration of determinants of school “success”. Oxford Review of Education, 24 (4), 131-143.
Gibson, A. & Asthana, S. (1998). School performance, school effectiveness and the 1997 White Paper. Oxford Review of Education, 24 (2), 195-210.
Goldstein, H. (1987). Multilevel models in educational and social research. London: Griffin.
Goldstein, H. (1998). A response to Gibson and Asthana. Oxford Review of Education, 24 (4), 521-523.
Hallinan, M. (1988). School composition and learning: A critique of the Dreeben-Barr model. Sociology of Education, 61 (3) pp.143-146.
Kreft, I. (1993). Using multilevel analysis to assess school effectiveness: A study of Dutch secondary education. Sociology of Education, 66 (2), 104-129.
Leiter, J. (1983). Classroom composition and achievement gains. Sociology of Education, 56 (3), 126-132.
Moos, R. (1979). Evaluating educational environments. Washington, DC: Jossey-Bass Publishers.
Moors, R. (1980). Evaluating classroom learning environments. Studies in Educational Evaluation, 6 (3), 239-252.
Moors, R. (1987). The social climate manual (2nd. ed.). Palo Alto, CA: Consulting Psychologists Press.
Opdenakker, M. & Van Damme, J. (2001). Relationship between school composition and characteristics of school process and their effect on mathematics achievement. British Educational Research Journal, 27 (4), 407-432.
Power, F., Higgins, A., & Kohlberg, L. (1989). Lawrence Kolberg’s approach to moral education. NY: Columbia University Press.
Purkey, S. C. & Smith, M. (1983). Effective schools: a review. Elementary School Journal, 83 (4), 427-52.
Raudenbush S. & Bryk, A. (1986). A hierarchical model for studying school effects. Sociology of Education, 59 (1),1-17.
Resh, N. & Dar, Y. (1992). Learning segregation in junior high-school in Israel: causes and consequences. School Effectiveness and School Improvement, 3 (3), 272-292.
Reynolds, D., Creemers, B., Nesselrodt, P., Schaffer E., Stringfield, S., & Teddlie, C. (1994). Advances in school effectiveness research and practice. Oxford: Pergamon.
Ridell, A. (1993). The evidence on public/private educational trade-offs in developing countries. International Journal of Educational Development, 13 (4), 373-386.
Ridell, A. (1997). Assessing designs school effectiveness research and school improvement in developing countries. Comparative Education Review, 41 (2), 178-204.
Rowan, B. (1990). Commitment and control: Alternative strategies for de organizational design of school. Review of Research in Education, 16 (4), 359-89.
Scheerens, J. & Bosker, R. (1997). The foundation of educational effectiveness. Oxford: Pergamon.
Scheerens, J. & Creemers, B. (1989). Conceptualizing school effectiveness. International Journal of Educational Research, 13 (7), 691-706.
Slavin, R. (1996). Salas de clase efectivas, escuelas efectivas: Plataforma de investigación para la reforma educativa en América Latina. Santiago: Programa de Promoción de la Reforma Educativa en América Latina y el Caribe.
Slee, R., Weiner, G., & Tomlinson, S. (Eds.). (1998). School effectiveness for whom? London: Falmer Press.
Stringfield, S. (1994). The analysis of large data bases in school effectiveness research. In D. Reynolds, B. Creemers, P. Nesselrodt, E. Schaffer, S. Stringfield, & C. Teddlie, Advances in school effectiveness research and practice (pp.55-72). Oxford: Pergamon.
Teddlie, C. (1994a). The integration of classroom and school process data in school effectiveness research. In D. Reynolds, B. Creemers, P. Nesselrodt, E. Schaffer, S. Stringfield, & C. Teddlie, Advances in school effectiveness research and practice (113-132). Oxford: Pergamon.
Teddlie, C. (1994b). The study in school effects research: History, method, results, and theoretical implications. In D. Reynolds, B. Creemers, P. Nesselrodt, E. Schaffer, S. Stringfield, & C. Teddlie, Advances in school effectiveness research and practice (85-110). Oxford: Pergamon.
Teddlie, C. & Reynolds, D. (2000). International handbook of school effectiveness research. London: Falmer Press.
Teddlie, C. & Reynolds, D. (2001). Countering the critics: Responses to recent criticism of school effectiveness research. School Effectiveness and School Improvement, 12 (1), 41-82.
Thrupp, M. (2001a). Recent school effectiveness counter-critiques: problems and possibilities. British Educational Research Journal, 27 (4), 443-457.
Thrupp, M. (2001b). Sociological and political concerns about school effectiveness research: Time for a new research agenda. School Effectiveness and School Improvement, 12 (1), 7-40.
Yasumoto, J., Uekawa, K., & Bidwell, C. (2001). The collegial focus and high school students’ achievement. Sociology of Education, 74 (3), 181-209.
Waxman, H. C. & Ellett, C. D. (Eds.). (1992). The study of learning environments. Houston, TX: University of Houston.
Weick, K. (1976). Educational organizations as loosely coupled systems. Administrative Science Quarterly, 21 (1), 1-19.
Translator: Lessie Evona York-Weatherman 2The technical schools are not included because of the important curricular differences which impede their direct comparison with other modalities. 3For the analysis, the state of Buenos Aires is divided into Greater Buenos Aires (Urban) and the rest of the state. 4Probability is estimated based on the difference between the values of the ratio of maximum probability of the model analyzed and the background model, a difference that can be referred to the chi-square distribution, and whose degrees of freedom are defined by the number of new parameters that have been adjusted in the model analyzed. 5The sum of the proportions of each level is slightly higher than the unit due to fluctuations in the sample. 6This interepretation is based on the high correlations between these variables: 456 for motivation and valoration and .634 for indiscipline and violence. 7The difference in the maximum probability test of Model 1 (313019,8) regarding the “null” model (314624.7) is 1604.9 which, with 8 df. (number of new parameters set), has a probability of occurrence of less than 1 per thousand. Model 2 produces an even greater distance (=1623.4), and therefore, the same conclusion applies. 8Processing not presented indicated that the joint action of all the student’s individual variables decreases the inter-state variation to .103, the inter-school to .260, and the inter-student to .544. Therefore, their major effect is located in the variations of levels 2 and 3, but is notably less than than of the variables of Sbc and Sp (Cervini, 2002; 2003). 9Three models were processed to analyze the effect of each of the three variables when excluding the other two. None were significant. In this way, the doubt as to whether the collinearity between them could explain the non-significant estimates was cleared up. 10The presentation of the estimates of the fixed part of the models is obvious because there is produced no change that could alter the conclusions drawn above.
UABC Mexicali
1The principals’ questionnaire was applied in 1997 and in 1998. However, in 1998, many of the questionnaires were answered after the evaluation workday, and were mailed afterward. This circumstance throws doubt on the quality of the data collected, as well as an imporantant decrease in the effective coverage. Because of this, we decided to use the 1997 questionnaires applied according to the standard established procedure. If the principal did not answer the 1997 questionnaire, but did answer that for 1998, the 1998 questionnaire was used (323 schools).
Cervini, R. (2003). Relationships among student-body composition, school process, and mathematics achievement in Argentina’s high schools. Revista Electrónica de Investigación Educativa, 5 (1). Retrieved month day, year, from: http://redie.ens.uabc.mx/vol5no1/contents-cervini2.html