Untitled Document

Revista Electrónica de Investigación Educativa

Vol. 19, Núm. 4, 2017

Attitude towards School Science in Primary Education in Spain

Raquel Fernández Cézar (*) raquel.fcezar@uclm.es
Natalia Solano Pinto (*) natalia.solano@uclm.es

(*) Universidad de Castilla-La Mancha

(Received: May 10, 2016; accepted for publishing: June 9, 2016)

How to cite: Fernández, R. & Solano, N. (2017). Attitude towards school science in primary education in Spain. Revista Electrónica de Investigación Educativa, 19(4), 112-123. https://doi.org/10.24320/redie.2017.19.4.1393

Abstract

This report shows Spanish elementary school students’ attitude towards school science. The analysis is performed through a questionnaire with a sample consisting of 183 10 to 12-year-old students. The association of this attitude with factors like sex, age, school environment and teachers’ use of experiments has been researched. The results show a positive attitude towards school science, and null association with sex or age. Besides that, there is an association between school environment and attitude: the students with the most positive attitude values belong to rural schools. A sex bias is observed in the group of students with the most positive attitude, with girls making up the majority. The small size of this reduced sample limits the impact of this conclusion. A difference in the perception of experiments is observed between teachers and students. Attitude towards school science is not associated with students’ perception of experiments in science classes.

Keywords: Elementary Education; Sciences; Experiments; Student attitude.

I. Introduction

Attitude is a cognitive, affective and behavioral predisposition with respect to any issue. Such a tendency is not natural but acquired (Rodríguez & Seoane, 1989) and individually developed through models, that is, through mental visions of a social issue, which are influenced by cultural beliefs (Morales, 2007). When the issue is science, we refer to the attitude towards science.

The relationship between student attitude towards science and interest in scientific content is reported by several authors (Klug, Krause, Schober, Finsterwald & Spiel, 2014). All these studies reveal that the more positive the student attitude towards science, the deeper the interest in scientific content. In a PISA report on Spanish students’ results, one of the conclusions reached is that student attitude towards the subject matter is crucial in guaranteeing outstanding academic achievement (Instituto Nacional de Evaluación Educativa [National Institute of Educational Evaluation] (INEE), 2013). This positive attitude is also reflected in the development of practical and logical reasoning abilities (Prieto-Patiño & Vera-Maldonado, 2008; Huey-Por et al., 2011), and not only in achievement.

Studies on student attitude towards science (Pelcastre, Gómez & Zavala, 2015) show that although students find some positive aspects in science at school level, they are not willing to start studying science majors. As a consequence, there has been a decline in the number of students enrolled at university in science, technology, engineering and mathematics (STEM) majors in the western world (National Science Education Standards, 1996; Vázquez & Manassero, 2008).

1.1 Theoretical framework

Attitude towards Science. Several authors have considered the attitude towards science by examining different aspects: the social and geographical context, and the social perspective.

The study offering the widest scope in this respect on the European continent was ROSE (available at http://www.uv.uio.no/ils/?vrtx=search&query=ROSE), carried out in 2004 by Schreiner and Sjøberg. The sample was made up of 15-year-old students from 40 countries. The questionnaire was developed by an experts’ committee and included 10 aspects of science in student life. The results are similar to those obtained in other international projects in developed countries: most of the students find science important in their lives but are not inclined to continue studying science in the future. Nevertheless, similar studies carried out in developing countries show that students like science, and a greater number show an interest in continuing to study science in future courses (Schreiner & Sjøberg, 2004).

In the Spanish context the reported results follow the same tendency as in developed countries (Prieto-Patiño & Vera-Maldonado, 2008). As far as we know, there are no studies on the analysis of the attitude towards science at school, or school science, which is the focus of this report.

Factors. There are numerous studies on the identification of factors affecting attitude towards science (Marbá-Tallada & Vazquez, 2010). These factors belong to different categories: some are external to school, like school environment, sex and age; others are inherent to it, like curriculum design and teacher instruction.

With regard to the influence of sex, studies in the Spanish context (Prieto-Patiño & Vera, 2008; Marbà-Tallada & Márquez, 2010) report a null sex effect. We will analyze if this factor affects attitude towards school science.

Regarding the effects of age on attitude, the association is reported in several studies conducted in different countries (Pelcastre et al., 2015). The effect of age is that the older the student, the worse the attitude towards science. Nonetheless, other studies (Prieto-Patiño & Vera-Maldonado, 2008; Vázquez & Manassero, 2008), performed with 12 to 18-year-old Spanish students, indicate no influence at all from age (Vázquez & Manassero, 2009; Marbà-Tallada & Márquez, 2010; Pelcastre et al., 2015). Since no studies are reported that analyze the attitude towards school science at the very beginning of adolescence, starting at 10, and the evolution in the 10 to 12-year-old range, this is one of the purposes of our study.

Regarding the school geographical environment, there are just a few studies reporting differences in attitude towards science for students from rural or urban schools (Chetcutia & Kiokob, 2012). The literature reviewed in the Spanish context (Vázquez & Manassero, 2009; Marbà-Tallada & Márquez, 2010) shows results from schools located in urban areas in the Balearic Islands and Catalonia, respectively, but not from rural ones. In this report we will attempt to discover the association between this variable and students’ attitude towards school science.

Teacher attitude towards school science and the way teachers view science are currently of interest to science education researchers. Teachers’ attitude towards science is transmitted to students through the way they teach. Pelcastre et al. (2015) studied this in a sample of 174 students between 15 and 16 years old. The conclusions are that although the analyzed students show a positive attitude towards science, teachers should revise their own attitudes and acquire and implement teaching strategies entailing positive attitudes towards science. Therefore, how elementary school teachers feel when teaching science, and how they are trained to teach science, are crucial.

In study programs at Schools of Education in Spain, science subjects do exist but in a low proportion (Oliva & Acevedo, 2005). As a result, these teachers are not experts in science education, and are less trained to teach science than other subjects.

Teacher instruction and student attitude towards school science. Current studies on attitude towards science agree that student attitude is influenced by several factors that come together in school, hence the need to implement forms of instruction and teaching strategies that contribute to this attitude (INEE, 2013; Hampden-Thompson & Bennett, 2013). González & Paollini (2015) attempt to determine factors influencing attitude towards science in a sample of 430 students in Spain taking their last high school course. The correlations between teaching strategies and attitude were significant (p<0.01) and ranged from .35 to .43.

Many authors suggest empowering interest in sciences by means of teaching strategies (Hampden-Thompson & Bennett, 2013; Chin-Tsan & Kai-Ying, 2015). They recommend providing a broader range of activities to suit different student skills or intelligences. One of the recommendations is to include experimentation.

There is a general acceptance of the fact that the inclusion of experiments or practical work in science instruction favors the learning of science, but not by itself.

The reviewed literature reports that experiments are lacking in Spanish elementary school science classes (Oliva & Acevedo, 2005). We wish to analyze whether experiments relate to students’ attitude towards school science.

Research objectives. The aims of this report are:

  1. to analyze Spanish 5th to 6th-grade elementary school students’ attitude towards school science;
  2. to discover the association between the attitude towards school science and sex, age, and school environment (rural or urban) variables;
  3. to discover if students’ perception of the presence of experiments in science classes relates to their attitude towards school science;
  4. to determine the extent to which teachers use experiments in science lessons, and their opinions on these experiments.

II. Methodology

Participants. The sample consisted of 183 elementary school students, 97 from 5th and 86 from 6th grade. In this sample 93 were boys and 90 girls. All participants were recruited from 7 public schools located in central Spain, namely 5 urban schools (97 students) and 3 rural schools (86 students). The teacher sample is made up of 36 elementary school teachers. All teachers answered the questions, but just 10 accepted to be interviewed. The schools were contacted in such a way that the sample is representative of the elementary school population in Spain because all students come from different social, cultural and economic environments.

Materials. The construct used was created for this purpose, as other authors did in their studies (Vazquez & Manassero, 2008), and consisted of a set of 8 statements chosen from the “My science classes” section of The Relevance of Science Education (ROSE) questionnaire (Schreiner & Sjøberg, 2004). The answers are provided with a semantic differential scale from 1 to 5, with 1 referring to the word appearing on the left and 5 the one on the right. The sentences are phrased in such a way that 1 is not always the most positive answer, nor 5 the most negative, in order to counterbalance the choices. The student questionnaire is shown in Appendix 1. Other student data such as sex, grade, and school, including the city, are also collected.  

To determine to what extent experiments are present in science lessons, two questions were posed to students, Q1 and Q2, shown in Appendix 2.a Teachers were also asked about the presence of experiments in their lessons, and some of them were interviewed, to collect information on this issue from them too. The questions dealing with this topic are included in Appendix 2.b

Procedure and Data Analysis. To measure school science attitude we used a survey that was sent to the participating schools in the first semester of the academic year. We asked the teachers working in 5th and 6th grades (10 to 12-year-olds) to administer it to students. The student and teacher survey answer sheets were collected and analyzed. After that, teacher interviews were conducted with those who accepted.

Student data are analyzed quantitatively by using the software IBM19. There are eight statements (S1-S8) that are added up to obtain the Total Attitude towards School Science (TAttSS). The internal consistency is tested through the Cronbach-Alpha coefficient, providing .701, and the validity through item-total correlation, providing values between .500 and .682. The TAttSS value was used to discover the possible association with the factors researched – sex, age and school environment – through a non-parametric mean test due to the fact that the answer distribution is not normal in most of the cases. The p value is always smaller than .05.

Statements 1, 3, 5, 6 and 8 were inverted to make 1 the most positive value in all cases. The TAttSS values range from 8, the most positive, to 40, the most negative, with 24 as the neutral value.

Regarding experiments in teaching, a quantitative analysis of surveys and a qualitative analysis of interviews were performed. We collected information on students’ and teachers’ perception of experiments. The questions for students are shown in Appendix 2.a These questions seek to ascertain the students’ perception of experiments in school science. On the other hand, the questions for teachers on this issue are in Appendix 2.b These questions intend to determine, firstly, whether teachers use experiments as part of their science lessons; secondly, whether students and teachers perceive experiments similarly; and thirdly, to what extent the students’ perception of experiments may relate to students’ attitude towards school science. The open-ended question for teachers is analyzed qualitatively, and an attempt is made to clarify the reasons for the results.

III. Results and Discussion

In the first stage, the TAttSS is analyzed, and then its association with sex, age and school environment factors is studied. The teachers’ use of experiments in science classes and the students’ perception of them is studied in the last part, in an attempt to discover any possible relationship with school science attitude.

Attitude towards School Science. Table I shows positive and negative percentages for TAttSS. The positive values range from 8 to 16, and the negative ones from 32 to 40. The mean and standard deviation are also shown.

Table I. Positive and negative percentages for TAttSS

Because the mean is a central value highly affected by extremes, we compare it with the median, 17.00, and they are almost the same. Furthermore, 50% of students display very positive values of attitude towards school science. Even the 3rd percentile, 21, is well below the value expected for neutral TAttSS, which is 24. It is in concordance with the low percentage for negative answers shown in Table I.

We can say that the attitude towards school science in 10 to 12-year-old elementary school students in Spain is positive, which follows the same direction reported by Prieto-Patiño and Vera-Maldonado (2008) but is even more positive, as more than 75% of students are well below the average (24), which is contrary to the study reported by Vazquez & Manassero (2008), although both studies worked with older students.

Attitude towards school science and sex, age and school environment factors

Table II. Mean and Standard Deviation of TAttSS for different sexes, school environments and ages

Sex. The mean value for TAttSS seems to be slightly more positive for boys than for girls. Nevertheless, the mean test does not produce any significant difference. Therefore, our work confirms previous results on the attitude towards science in general, reported by other authors (Schreiner & Sjøberg, 2004; Vázquez & Manassero, 2009; Pelcastre et al., 2015).

Age. (10-12 years). Table II shows that, although the mean value for 10-year-old students is less positive than for 12-year-olds, the mean test does not produce any significant difference. It is not aligned with other studies performed with older students, 12 to 18 years old (Vázquez & Manassero, 2009; Marbà-Tallada & Márquez Bargalló, 2010; Pelcastre et al., 2015), but it is aligned with those reported by Prieto-Patiño & Vera-Maldonado (2008) and Vázquez & Manassero (2008). Therefore we can say that an insignificant effect on TAttSS is detected for age in 10 to 12-year-old Spanish elementary school students.

School environment. The mean value for rural school students (Table II) reveals a more positive attitude towards school science in rural school environments (p<.05).

The effect of school environment on attitude towards school science has not been studied in most of the referenced works. Only Chetcutia and Kiokob (2013) considered it as a possible notable factor in attitude towards science. There are no studies of this kind in Spain or any other European country. We consider it pertinent because the environment is part of students’ and teachers’ lifestyle. The role of school itself is not the same in urban environments as in rural ones, and neither is school science. These results show that this factor influences students’ attitude towards school science the most, among the ones analyzed here.

Due to the differences observed, it could be interesting to go deeper and analyze the effect of the previous factors on the subset with the most positive TAttSS.

Cases with the most positive attitude. Because of the differences observed, we have restricted the sample to the students with the most positive attitude, and researched the possible association with sex, age or school environment.

Table III. Mean and Standard Deviation of TAttSS for different sexes, ages and school environments in the set with the most positive attitudes

Table III reveals that, in this small sample, there is a clear difference between girls and boys, while no difference exists for age or school environment. Girls do show a more positive attitude; or in other words, the majority of students that show the most positive attitude towards school science are girls. This has never been observed in any previous study: they either reported no sex effect at all, or if any, a more positive attitude for boys. The weakness of our new result is the fact that the sample size is particularly small, n<10, which makes it impossible to generalize it. Prospectively a deeper analysis must be performed with larger samples to see if this may be confirmed.

Experiments as Part of Teaching. To analyze this issue, students were asked two questions (see Appendix 2.a).

Table IV. Positive and negative answer percentage, mean and Standard Deviation for each question

In Q1 we asked students about the presence of experiments in their science classes. In this case, 20.3% of students view experiments as part of their science lessons, while 51.3% do not perceive experiments at all. On the other hand, 70.5% of students would like to have more experiments in school science (Q2), and only 14.2% would not.

The first part of the previous paragraph contrasts with what teachers say about using experiments in their science lessons: 98% answered “no” to the first question (see Appendix 2.b), meaning that they do not use experiments in science classes, and 100% answered “yes” to the second one, meaning that they would like to use them.

Negative student answers amount to 51.3%, while negative teacher answers reach 98%. What could be the reason for this contradictory view? It may be that students and teachers have differing ideas of experiments. There are other studies that find this mismatch between teachers’ and students’ ideas of teaching strategies and activities. Wai Yung et al. (2013) reported this observed difference in a study carried out with 110 teachers and 4,024 Year 7 students. We interpret that students consider they do experiments when they perform any hands-on or practical activity, while teachers know that these activities are not always experiments. Experiments involve discovery and above all, the implementation of the scientific method, focusing on observations and later reflections, while practical activities simply involve observations without further reflection in most cases, as teachers stated in the interviews.

The fact that although teachers consider they would use experiments in their science lessons (Q2, Appendix 2.b), they don’t and have never implemented any, is significant. Reasons for this contradictory behavior were identified with the answers to the third open-ended question, and the interviews.

On the one hand, the majority of teachers (28) complained about the lack of laboratory rooms in elementary schools, and even the lack of facilities in the classroom. A significant number (26) said they didn’t feel confident conducting classes involving experiments or teaching science through experiments. They would like to learn how to teach this way because they are not familiar with this methodology. On the other hand, just a few (6) said they would suggest that the parents pay for it as an extracurricular activity, instead of performing experiments in class because they consider them leisure activities.

As the sample we have been dealing with amounts to 36 teachers, it cannot be considered representative of the population of elementary school teachers in Spain, but is significant enough to allow us to reach some conclusions on science teaching in elementary schools.

Elementary school teachers have not followed the proposals to improve science teaching made by science education researchers from the beginning of this century (Oliva Martínez & Acevedo Díaz, 2005) regarding the use of experimentation in science lessons. They teach science as a collection of facts, without providing opportunities for comprehension through observation, experimentation and reflection.

Although just the inclusion of experiments does not entail an improvement of the teaching of science, the scientific community considers them a main part of the teaching of science because they are present in all the science courses. So how could experiments be included in elementary school science? The main reasons offered by teachers are the lack or scarcity of facilities in schools, which cannot be solved in this paper, and the lack of teacher training in methodologies involving the use of experiments. In this last respect, we can suggest actions to reverse this situation: the improvement of science education programs for preservice teachers at university level, and the effective use of lifelong learning projects including refresher courses in science education methodologies, aligned with others currently developed in countries with similar weaknesses and aimed at improving the perception of science (Fernández Cézar, 2014; De Juan et al., 2016).

Attitude towards School Science and Experiments. We attempted to determine whether students’ perception of the use of experiments is related to their attitude towards school science.

Table V. Mean and Standard Deviation of TAttSS for the presence or lack of experiments in science classes

Studies have been conducted with undergraduate students that report experiments in science classes with a more positive student perception of the subject. But as far as we know, there has been no work reporting science experiments with Spanish students, except for infant education. López Dalmau and Alsina (2015) find that the methodology used influences motivation and learning of the subject, and say nothing specifically about attitude. It fits our hypothesis on attitude that cannot be proven with our results, as can be seen in Table V. These results show that students’ attitude towards school science shows no association with their perception of the presence of experiments in science classes. The Spearman coefficient, -0.074 (p<.331), shows a weak correlation between the two aspects. For the future we plan to implement a program to include experiments in science classes, and analyze the possible association with the students’ attitude towards school science in the new teaching situation.

IV. Conclusions and Future Implications

Our results show that the attitude towards school science in 10 to 12-year-old elementary school students in Spain is positive, contrary to what has been observed in other studies with older students.

As in previous studies performed in Spain, null difference due to sex or age is observed in this attitude, except for a majority of girls in students showing the most positive attitude towards school science. But the restricted sample is too small to allow us to generalize the result. It would be interesting to repeat the same kind of analysis in the future with larger samples. Contrary to the aforementioned factors, school environment does have a significant effect on students’ attitude: there are more students reporting a more positive attitude towards school science in rural schools. Therefore, it is necessary to take steps in urban schools to improve elementary school students’ perception of science.

Regarding the presence of experiments in science classes, we conclude that the different answers given by students and teachers are indicative of a difference in the students’ and teachers’ ideas of experiments, and this should be thoroughly researched. The majority of teachers say they do not perform experiments in their science classes, which confirms that science teaching in Spain is far from the proposals made by experts in the final years of the 20th century. The majority of teachers argue that the lack of education in this respect is the reason for not including experiments in science classes. Therefore, reversing this situation requires improving preservice teacher programs at university and designing lifelong learning programs.

On the other hand, there is no association between students’ perception of the presence of experiments in science classes and the TAttSS in the type of students we worked with. A program to include experiments in science classes is planned to be implemented. At the end of this program, students’ attitude towards school science and possible associated factors will be studied.

Acknowledgements

This study has been carried out under the project “Conocer la ciencia hoy abre las puertas del mañana” (Knowing science today opens doors tomorrow) with funds from the Spanish Foundation for Science and Technology, FECYT, in the years 2012 and 2015, and the Spanish Royal Society of Chemistry from 2012 to 2015.

References

Chetcutia, D. A., & Kiokob, B (2012). Girls’ Attitudes towards science in Kenya. International Journal of Science Education, 34(10), 1571-1589. doi:10.1080/09500693.2012.665196

Chin-Tsan, W., & Kai-Ying, W. (2015). Investigation of the cognition and attitude of junior high school students on energy resources: the case of kai-syuan junior high school, I-Lan, Taiwan. International Journal of Information and Education Technology, 5(12), 924- 930. doi:10.7763/IJIET.2015.V5.639

De Juan, J., Pérez-Cañaveras, R. M., Segovia, Y., Girela, J. L., Martínez-Ruiz, N., Romero-Rameta, A. & Vizcaya-Moreno, M. (2016). Student perceptions of the cell biology laboratory learning environment in four undergraduate science courses in Spain. Learning Environments Research, 19(1), 87-106. doi:http://dx.doi.org/10.1007/s10984-015-9201-6

Fernández, R. (2014). Incorporación de simulaciones en el laboratorio de química general: influencia en el dominio afectivo del aprendizaje [Simulations in chemistry laboratory: influence in the affective learning domain]. Didáctica de las Ciencias Experimentales y Sociales, 28, 197-219.

González, A., & Paoloni, P-V. (2015). Implicación y rendimiento en Física: el papel de las estrategias docentes en el aula, y el interés personal y situacional del alumnado [Engagement and performance in physics: the role of teaching strategies in the classroom, and students’ personal and situational interest]. Revista de Psicodidáctica, 20(1), 25-45. doi:10.1387/RevPsicodidact.11370

Hampden-Thompson, G., & Bennett, J. (2013). Science teaching and learning activities and student engagement in science. International Journal of Science Education, 35(8), 1325-1343. doi: 10.1080/09500693.2011.608093

Huey-Por, C., Chin-Chang, C., Gwo-Jen, G, Yeong-Jin, C., Chen-Yung, L. & Tsung-Hau, J. (2011). The development of a competence scale for learning science: Inquiry and communication. International Journal of Science and Mathematics Education, 9(5), 1213-1233.

Instituto Nacional para la Evaluación de la Educación (2013). PISA 2012. Programa para la evaluación internacional de los alumnos. Informe español (volumen I) [Program for International Student Assessment. Spanish report (volume I)]. Madrid: Ministerio de Educación.

Klug, J., Krause, N., Schober, B., Finsterwald, M., & Spiel, C. (2014). How do teachers promote their students’ lifelong learning in class? Development and first application of the LLL Interview. Teaching and Teacher Education, 37, 119-129. doi.org/10.1016/j.tate.2013.09.004

López, M., & Alsina, A. (2015). La influencia del método de enseñanza en la adquisición de conocimientos matemáticos en educación infantil [The influence of teaching methods on the acquisition of mathematical knowledge in early childhood education]. Edma 0-6: Educación Matemática en la Infancia, 4(1), 1-10.

Marbà-Tallada, A., & Márquez, C. (2010). ¿Qué opinan los estudiantes de las clases de ciencias? Un estudio transversal de sexto de Primaria a cuarto de ESO [What do students say about their science class? A cross-study from the sixth grade (primary school) to the fourth grade (secondary school)]. Enseñanza de las Ciencias, 28(1), 19-30.

Morales, J. F. (2007). Psicología Social [Social psychology]. Spain: McGraw-Hill.

National Academy of Sciences (1996). National Science Education Standards. National Academy Press : Washington, DC. Retrieved from http://www.nap.edu/openbook.php?isbn=0309053269

Oliva, J. M., & Acevedo, J. A. (2005). La enseñanza de las ciencias en primaria y secundaria hoy. Algunas propuestas de futuro [Science teaching in primary and secondary education today. Proposals for the future]. Revista Eureka sobre Enseñanza y Divulgación de las Ciencias, 2(2), 241-250.

Pelcastre, L., Gómez, A. R., & Zavala, G. (2015). Actitudes hacia la ciencia de estudiantes de educación preuniversitaria del centro de México [Attitudes toward science of high school students from central Mexico]. Revista Eureka sobre Enseñanza y Divulgación de las Ciencias, 12(3), 475-490. doi:10498/17603

Prieto-Patiño, L., & Vera, A. (2008). Actitudes hacia la ciencia en estudiantes de secundaria [Attitudes towards science in high school students]. Psychologia. Avances de la disciplina, 2(1),133-160. Retrieved http://www.redalyc.org/pdf/2972/297224999005.pdf

Rodríguez, A., & Seoane, J. (Coords.) (1989). Creencias, actitudes y valores [Beliefs, attitudes and values]. Madrid: Alhambra.

Schreiner, C., & Sjøberg, S. (2004). Showing the seeds of ROSE. Background, Rationale, Questionnaire Development and Data Collection for ROSE (The Relevance of Science Education) –a comparative study of students’ views of science and science education. Acta Didactica, 4. Retrieved from http://www.roseproject.no/key-documents/key-docs/ad0404-sowing-rose.pdf

Vázquez, A., & Manassero, M. A. (2008). El declive de las actitudes hacia la ciencia de los estudiantes: un indicador inquietante para la educación científica [The decline of students’ attitudes toward science: A disturbing indicator for science education]. Revista Eureka Enseñanza y Divulgación de Ciencias, 5(3), 274-292.

Vázquez, A., & Manassero, M. A. (2009). La relevancia de la educación científica: actitudes y valores de los estudiantes relacionados con la ciencia y la tecnología [The relevance of science education: students’ values and attitudes related to science and technology]. Enseñanza de las ciencias, 27(1), 33-48.

Wai Yung, B.H., Zhu, Y., Ling Wong, S., Wai Cheng, M. & Yin Lo, F. (2013). Teachers’ and Students’ Conceptions of Good Science Teaching. International Journal of Science Education, 35(14), 2435-2461. doi:10.1080/09500693.2011.629375