The results observed are consistent with the reported in most of the national and international literature: the women present better performance than the men in reading comprehension, and the men register better performance in math; the students with ages over the ones that correspond to the sixth grade of elementary school or third year of secondary school, present lower performance levels to those of their schoolmates; the students of the morning shift present higher performance levels compared to the evening shift students; and the students of private schools present better results than those of public schools.

It was also analyzed, through hierarchical lineal models, the effect of the socioeconomic level of the context of the school and of the professor in the educational achievement. In all the cases, the levels of variance explained turned out to be smaller that the registered in other similar national studies.

Key words: Context effect, assessment of student knowledge, standardized tests, elementary education.

In Mexico, evaluation of academic achievement at the level of basic education emerged as the country’s most important evaluative strategy, which receives large allocations of funding and is the object of the greatest attention among the leading players in the Mexican education system. The results of such evaluations are the product of participation in various national and international programs coordinated by the Ministry of Public Education (Spanish acronym SEP) and the National Institute for Evaluation of Education (Spanish acronym INEE) (Caso, Gonzalez, Contreras, and Gonzalez, 2007).

In Mexico, for more than a decade various actions have been taken focused on evaluating the quality of basic level education. The principal evaluation methods are: the Academic Achievement Factor, National Standards Tests, Educational Quality and Achievement Exams, the Children’s Olympics of Knowledge, the Diagnostic Instrument for New Junior High School Students, Census Diagnostic Exams, and the National Evaluation of Academic Achievement in Schools. Mexico also has taken part in international evaluations such as the Third International Mathematics and Science Study (TIMSS), the Second Regional Comparative and Explanatory Study conducted by the Latin American Laboratory for Evaluation of Quality in Education, and the Programme for International Student’s Assessment (PISA).

National Standards Tests represent one of the evaluative strategies that have contributed most to measuring levels of educational achievement in Mexican students at the basic level. These tests, administered for the first time in 1998, were initially intended to evaluate knowledge and skills in reading comprehension and mathematics in second, third, fourth, fifth, and sixth grade elementary school students and students in the first, second, and third years of junior high school. In 2003, their name and orientation changed; they were renamed National Tests and focused exclusively on evaluating the areas of interest in students in sixth grade and third year junior high school students. The administration of these tests was interrupted in 2005 with the creation of a new generation of tests developed by the INEE, known as Educational Quality and Achievement Exams (Backhoff, Andrade, Monroy, Tanamachi, Bouzas, Sanchez & Peon, 2005).

The application of the National Tests provided the opportunity to generate several detailed reports which describe students’ skill levels by domain, modality, and state of origin (National Institute for Evaluation and Education [Spanish acronym INEE], 2003, 2004, and 2005). It also provided the basis for an evaluation of the behavior of personal and contextual variables, such as gender, school background, socioeconomic level, configuration, environment, and family expectations, and their impact on achievement levels.

However, the conclusions obtained from analyzing the nationwide sample have some limitations. In the first place, they ignore particularities associated with Mexico’s vast social, economic, and educational inequalities, which leads to inaccurate results because it is incorrect to assume that the impact of contextual variables on educational achievement is the same for students from different regions or states. On the other hand, unfamiliarity with the specific behavior of contextual variables in each state limits the justification for strategies of differential intervention that could help reduce the specific problems state education systems face.

In this regard, the influence of socioeconomic levels and family factors on academic performance is unquestionable, as several international and national researches confirm. Expectations and level of involvement with schoolwork; parents’ occupational and educational level; as well as climate, structure, cultural background, family income, and child raising styles, are variables that largely determine a student’s academic success or failure (Casanova, Cruz, De la Torre & De la Villa, 2005; Gonzalez-Pienda, Nuñez, Gonzalez-Pumariega, Alvarez, Roces & Garcia, 2002; Jones & White, 2000; Ruiz, 2001). At lower income levels, the probabilities of an unstable family dynamic and parents’ disinterest in supervising academic work and underestimating their children’s scholastic and extracurricular activities tend to be greater, leading to lack of stimulus and support for students, which can affect and lower their academic performance.

Studies with Mexican adolescents that involved considerable samples of students explored the impact of parents’ academic background and occupation, family income, and cultural capital on academic performance. The results of various nationwide evaluations of junior high school students revealed that enrollment in private schools, high parental expectations, parents’ academic level, organization in household tasks and schoolwork, and the desire to continue studying explain, with levels of statistical significance, the results reported in mathematics and Spanish (Backhoff et al., 2006).

Consequently, this study has been proposed with the aim of analyzing the impact of contextual variables on educational achievement in a sample of students in the State of Baja California and comparing the results with the findings of national and international studies.

The sample of Baja California students was obtained using the sample design proposed for application of the National Tests, broken down into two stages: 1) selection of schools using the model of probability proportional to size, and 2) selection of students within each school (INEE, 2004). Except for elementary schools offering indigenous education and urban private schools, as well as private junior high schools and telesecundarias [secondary schools with television support], the schools offered statewide representativity.

1.2. Participants

In all, 1,817 sixth-grade students and 1,239 third-year junior high school students, enrolled for the 2003-2004 school year in 71 elementary schools (38 urban public schools, 24 rural public schools, 5 urban private schools, and 4 schools offering indigenous education) and in 40 junior high schools (22 private general junior high schools, 14 public technical junior high schools, 3 private general junior high schools, and 1 telesecundaria) in Baja California.

The sample of sixth-grade students was made up of 896 boys (49.3%) and 921 girls (50.7%), with ages ranging from 10 to 13 years, of whom 1,259 (69.3%) were enrolled in the morning shift and 558 (30.7%) in the evening shift. The sample of third-year junior high school students was made up of 588 boys (47.5%) and 651 girls (52.5%), with ages ranging from 13 to 16 years, 1,183 (95.5%) of whom were enrolled in the morning shift and 56 (4.5%) in the evening shift.

1.3. Instruments

The two versions of the National Tests are divided in two sections: achievement tests and context questionnaires.

• Academic achievement questionnaire. The achievement tests for sixth grade consisted of 86 multiple choice items, 42 reading comprehension items, and 44 mathematics items. The tests for third year of junior high school included 89 multiple choice items, 45 reading comprehension items, and 44 mathematics items. The results of students’ academic achievement in reading comprehension and mathematics are shown separately on the Rasch model, with a minimum of 200 points and a maximum of 800, with the mean at 500 points. Indices of reliability in nationwide application for the four evaluations show internal consistency, with values between .838 and .866.
• Context questionnaires. The questionnaires evaluated contextual aspects of the environment, including school background, family configuration and environment, parents’ academic level, living conditions, organization of free time, occupational status, study activities, family expectations, and student-teacher relationships, among others. The version for elementary school used 51 questions and the junior high school version used 53.

1.4. Data analysis

The following analyses were conducted: a) comparative analysis, to compare academic achievement scores by gender, age, shift, and type of school; b) multilevel analysis using the hierarchical linear modeling to determine the effect of context, school, student, and teacher on educational achievement; and c) analysis of the impact of contextual variables on educational achievement. The data was processed and analyzed with the aid of SPSS version 14.0 and HLM version 6.02 software.

Table I. Differences in academic achievement by gender

On the other hand, the relationship between the results of these evaluations and the subjects’ age shows a drop in academic performance of students who exceed the estimated “natural” age for sixth grade or third year of junior high school, as the case may be (Table II). In other words, students over age 13 elementary schools and over age 16 in junior high school had lower academic achievement scores than their younger classmates. These differences are statistically significant.

Table II. Differences in educational achievement by age

Statistically significant differences were also observed in the achievement of students in different school shifts; students enrolled in the morning shift performed better in the subjects and educational levels evaluated (Table III).

Table III. Differences in educational achievement by school shift

Students’ performance also showed variations related to school of origin, for both sixth grade and third year of junior high school. In the case of primary education, private school students performed better than those in urban public schools in the two subjects evaluated, urban public schools outperformed rural schools, and indigenous schools in turn scored higher than rural schools (Table IV).

Table IV. Differences in educational achievement by type of school in elementary schools

At the junior high school level, private schools scored higher than the rest in both subjects evaluated; public general schools outperformed public technical schools and telesecundarias in reading comprehension; and telesecundarias scored higher than technical and general schools in mathematics (Table V). The differences observed presented statistically significant values.

Table V. Differences in educational achievement by type of school in junior high schools

On the other hand, two- and three-level models were used to analyze the impact of contextual variables on subjects’ performance. The first level corresponded to the student, the second to the group or teacher, and the third to the school. When two-level models were applied, the level corresponding to the teacher was suppressed.

The first analysis determined, by means of the intra-class correlation coefficient, the proportion of variance explained by the school at the two educational levels and in the two subjects evaluated, presenting values of 23.5% in reading comprehension and 21.8% in mathematics for sixth grade, and 20.2% in reading comprehension and 14.9% in mathematics for third year of junior high school (Table VI).

Table VI. Effect of school

The second analysis measured the impact of context within the variation caused by the school’s impact on educational achievement. Most of the literature on scholastic efficacy recognizes the impact of the school setting as a key factor in students’ performance, and students’ average socioeconomic level within a school is one of the most widely used indicators for the purpose (Carvallo, 2005).

The 2004 National Tests included several questions related to students’ socioeconomic level, for both junior high school and elementary school students, which consider families’ housing conditions and possessions, focusing on specific resources or assets considered representative nationwide. The results for impact of school context are shown in Table VII.

Table VII. Impact of school context

Then, the percentage of the reduction in variation between schools was estimated and students’ socioeconomic level (SEL) was included as a first-level variable. The results are shown in Table VIII.

Table VIII. Reduction in variation between schools on including the variable student SEL

A third analysis used three-level models which included the group or teacher level to calculate the teacher’s impact. The results are shown in Table IX.

Table IX. Percentage of variance explained at each level

As in the previous case, the impact of socioeconomic level on the model was estimated, as shown in Table X.

Table X. Percentage of variance explained at each level controlled by SEL

As regards the impact on educational achievement of the specific contextual variables analyzed simultaneously in multiple linear regression models, as well as in HLM models, personal, family, school, and teacher related variables were identified, which explain students’ educational achievement with values with statistical significance on the order of p<.05.

In this regard, as shown in Tables XI and XII, only a few variables have a significant impact on the four subjects evaluated. The impact of other variables is differentiated depending on the subject and grade analyzed.

In the case of sixth grade, father’s academic background, personal academic expectations, family supervision of schoolwork, and monitoring of tests favor academic achievement for both mathematics and reading comprehension (Table XI). Also, having a computer at home and being a girl favor scores in reading comprehension, whereas the number of books at home, time spent on schoolwork, and number of years in preschool favor scores in mathematics.

On the other hand, levels of educational achievement in junior high school are favored by enjoyment of reading, study expectations, and the frequency with which families enquire about test results (Table XII). In contrast, teachers’ advice, family demand on going over schoolwork in elementary school, and special homework assignments, visits to school libraries, and teacher’s anger in junior high school are factors that lower academic performance.

Table XI. Impact of contextual variables on educational achievement in sixth-grade students

Table XII. Impact of contextual variables on educational achievement in students in third year of junior high school

On the other hand, the results observed in our analysis of the impact of contextual variables on subjects’ performance are similar to the nationwide results reported in 2003 (Carvallo, 2006), where the school’s impact is greater in elementary school than in junior high school and greater in reading comprehension than in mathematics (Fernandez, 2004; Marzano, 2003; Murillo, 2003; Lastra, 2001).

The proportions of variance observed are also consistent with the findings reported by Fernandez & Blanco (2004), who analyzed the results of several studies conducted in Mexico between the years 1998 and 2003, which documented schools’ impact on educational achievement with values between 26% and 38% in mathematics and from 29% to 34% in Spanish. In 2001, after isolating context in mathematics, the resulting variation was 20%, unlike the results from 1998, when it reached 36%. In the same context, the differences in Spanish reached 29.3% in 1998, up from only 19.5% in 2001.

In this study, the fact that the impact of context was markedly similar in the subjects and school levels evaluated is noteworthy. Prior nationwide studies, which measured the impact of context on such tests, reported results with variations between 10% and 64% in third year of junior high school (Carvallo, 2006). These results present minimal differences when estimating the percentage drop in variation between schools caused by including students’ socioeconomic level (SEL) as a first level variable; this variable explains, then, a minimal proportion of the variation in educational achievement.

The fact that the drop in variation between schools is explained so little by students’ socioeconomic level contradicts the conclusions reported in national and international literature on the subject. There would appear to be no significant variation in this variable among students in the sample, as few of them are from high and low socioeconomic levels, causing the indicators used to behave as they would in a normal distribution. Another possible explanation is the sum of conditions characteristic of the local border context, which needs to be interpreted with caution, as in the Mexico-US border region, particularly on the Baja California-California border, as they can hardly be considered accurate indicators of socioeconomic level. In this regard, the explanatory argument expresses the possibility that the equation proposed to determine socioeconomic level is not the most accurate or the most plausible for the state.

On the other hand, the teacher’s impact on students’ educational achievement is low, and considerably lower than the school’s (except in the case of mathematics in third year of junior high school). This finding contrasts with that reported in specialized literature, which has attributed a greater impact to teachers than schools. The result obtained for impact in third year of junior high school in the mathematics test is noteworthy, where, in addition to being very low, the school’s impact on the corresponding model is insignificant. This would imply that all schools have equal impact and that the variation among them is insignificant, an explanation that would be compounded by the argument that the data observed suggest that the impact of socioeconomic level is also insignificant.

Finally, on analyzing the individual impact of contextual variables on educational achievement, enjoyment of reading and frequency with which families enquire about test results are relevant. In recent studies that combine several variables from the family setting in a broader dimension referred to as cultural capital, which has proven its predictive value in similar subjects and academic levels, the two aforementioned variables were documented as predictors of educational achievement (Backhoff et al., 2006).

In conclusion, the information generated by this study, allowing for limitations associated with the size and configuration of the state sample, helps to broaden our knowledge of the impact of contextual variables on academic achievement of students at the level of basic education. The proposed analytic approach, supported by multilevel analyses, represents a useful tool which, in addition to helping to assess the differentiated impact of student, school, and context, may lead to fairer evaluations.

Backhoff, E., Andrade, E., Peón, M., Sánchez, A. & Bouzas, A. (2006). El aprendizaje del español, las matemáticas y la expresión escrita en la educación básica en México: Sexto de primaria y tercero de secundaria. Resumen Ejecutivo. México: Instituto Naconal para la Evaluación de la Educación

Carvallo, M. (2005). Análisis de los resultados obtenidos en estudios de eficacia escolar en México, comparados con los de otros países. Revista electrónica iberoamericana sobre calidad, eficacia y cambio en la educación, 3 (2). Consultado el 10 de septiembre de 2005 en: http://www.ice.deusto.es/rinace/reice

Carvallo, M. (2006). Factores que afectan el desempeño académico de alumnos de educación secundaria. Tesis doctoral no publicada, Centro Universitario de la Costa de la Universidad de Guadalajara, México.

Casanova, P., Cruz, M., de la Torre, M. J. & de la Villa, M. (2005). Influence of family and socio-demographic variables on students with low academic achievement. Educational Psychology, 25 (4), 423-435.

Caso, J., González, M., Contreras, L. A. y González, I. (2007). Antecedentes de la evaluación del aprendizaje a gran escala en Baja California. (Report 07 001). Ensenada, Mexico: Universidad Autónoma de Baja California, Unidad de Evaluación Educativa.

Fernández, T. (2004). Distribución del conocimiento escolar, clases sociales, escuelas y sistema educativo en Latinoamérica. Tesis doctoral no publicada, El Colegio de México.

Fernández, T. & Blanco, E. (2004). ¿Cuánto importa la escuela? El caso de México en el contexto de América latina. Revista Electrónica Iberoamericana sobre Calidad, Eficacia y Cambio en Educación, 2 (1). Retrieved October 14, 2006 from: http://www.ice.deusto.es/RINACE/reice/vol2n1/FernandezyBlanco.htm

González-Pienda, J. A., Nuñez, J. C., González-Pumariega, S., Alvarez, L., Roces, C. & García, M. (2002). A structural equation model of parental involvement, motivational and aptitudinal characteristics, and academic achievement. The Journal of Experimental Education, 70 (3), 257-287.

Instituto Nacional para la Evaluación de la Educación (2003). La calidad de la educación básica en México. México: Autor.

Instituto Nacional para la Evaluación de la Educación (2004). La calidad de la educación básica en México. México: Autor.

Instituto Nacional para la Evaluación de la Educación (2005). La calidad de la educación básica en México. México: Autor.

Jones, I. & White, S. (2000). Family composition, parent involvement, and young children´s academic achievement. Early Child Development and Care, 161, 71-82.

Lastra, E. (2001). School effectiveness, a study of elementary public schools in a Mexican city. Unpublished doctora dissertation, Stanford University, California.

Marzano, R. (2003). Classroom management that works. Virginia: ASCD.

Murillo, F. J. (2003). El movimiento de investigación de eficacia escolar. In F. J. Murillo (Coord.), La investigación sobre eficacia escolar en Iberoamérica. Revisión internacional del estado del arte (pp. 53-92). Bogotá: CIDE.

Ruiz, C. (2001). Factores familiares vinculados al bajo rendimiento. Revista Complutense de Educación, 12 (1), 81-113.