Revista Electrónica de Investigación Educativa

Vol. 24, 2022/e24

Benefits of Using the Learning Management System based on Flipped Learning Methodology

Santiago Pozo-Sánchez (*) https://orcid.org/0000-0001-8125-4990

Adrián Segura-Robles (*) https://orcid.org/0000-0003-0753-7129

Antonio José Moreno-Guerrero (*) https://orcid.org/0000-0003-3191-2048

Jesús López-Belmonte (*) https://orcid.org/0000-0003-0823-3370

(*) Universidad de Granada, Spain

(Received: April 18, 2020; accepted for publishing: December 2, 2020)

Abstract

Among the different tools that can be used in the implementation of flipped learning, learning management systems stand out as one of the educational technology systems that best solve problems related to usability, accessibility, and evaluation. The objective of this study is to analyze the academic impact of flipped learning through LMS platforms versus the use of flipped learning in other non-interactive digital media, as traditional learning. For this, a quasi-experimental design was used in a sample of 231 Spanish secondary school students. A questionnaire was used to collect the data. The results show that students who have followed the teaching-learning process with flipped learning obtain better results than those who only apply the flipped learning method. It is concluded that the use of an LMS during the application of the flipped learning methodology contributes to an optimization of the general teaching-learning process.

Keywords: assessment, learning strategy, secondary education, educational innovation

I. Introduction

The reality of education today is characterized by the proliferation of methodologies that are highly adaptable to the needs of the current learner (Llanos & Bravo-Agapito, 2017). Under this premise, flipped learning is a methodological alternative to fulfill the demands of an education in which both information and communication technologies (ICTs) and students themselves stand as protagonists of the teaching-learning process (Froehlich, 2018; Zainuddin et al., 2019). Flipped learning began in the United States thanks to the training actions of teachers Jonathan Bergmann and Aaron Sams, aimed at providing digital audio-visual material to those students who could not attend class regularly (Bergmann & Sams, 2012). Since then, flipped learning has evolved and adapted to achieve a high degree of effectiveness and establish itself as a methodology with a growing number of followers (Fuentes-Cabrera et al., 2020; He et al., 2016; Pozo et al., 2019; Segura-Robles et al., 2020).

The flipped learning methodology consists in reversing the traditional learning processes (Bauer-Ramazani et al., 2016). Students use the after-school period to work with the theoretical and conceptual elements through digital tools (Long et al., 2017; Schmidt & Ralph, 2016) and allocate the time in the classroom to carrying out practical and dynamic activities under supervision by the teacher (El Miedany, 2019). This enhances the protagonist role of the students (Touron & Santiago, 2015), who self-regulate and adapt their own educational process individually (Boelens et al., 2018; López et al., 2019), allowing them to learn at times when they are more available, better able to access material, and more motivated (Shih & Tsai, 2017).

The main research on flipped learning has found that its implementation fosters student motivation and participation in training activities (Chyr et al., 2017; Huang et al., 2018; Tse et al., 2019) that are configured under a collaborative approach (DeLozier & Rhodes, 2017) that promotes socialization within the peer group (MacLeod et al., 2018). This cooperative climate allows a higher rate of development in problem solving (Bognar et al., 2019; Pozo et al., 2020), in the assimilation of curricular content (Karabulut et al., 2018), in the achievement of didactic objectives (Awidi & Paynter, 2019), and thus in the grades attained by the students (Hinojo-Lucena et al., 2018; Sola et al., 2019).

However, not all these potentialities of the flipped classroom are without limitations. Some authors have mentioned the need for high levels of abstraction in students in order to carry out activities (Hwang et al., 2015), the inability shown by some students to solve problems (Bognar et al., 2019; Mengual-Andres et al., 2020), and the difficulties of accessing and managing learning platforms (Hwang & Lai, 2017; López et al., 2019; López-Belmonte et al., 2019; Yilmaz, 2017). In reference to this last limitation, one of the different tools that can be used in implementing flipped learning is the learning management system (LMS), one of the educational technology systems best able to adapt to this type of methodology to overcome problems of accessibility and usability (Almarashdeh, 2016; Rhode et al., 2017).

We understand that LMSs are tools intended for the administration of high-volume information banks. These tools are used together in digital frameworks that allow the generation of content, practical activities, and assessment tools (Gilhooly, 2001). These types of tools allow multiple courses to be managed at once with a multitude of students, who can register and log in to the platform to complete the different courses and carry out flexible assessment processes (Gallagher, 2007), all based on a functional, intuitive, and simple interface (Hall, 2002) with a high level of digital security (Biech, 2008).

This produces a dynamic educational context that takes advantage of the learning management capacity offered by technology to improve the academic performance of students (Mwalumbwe & Mtebe, 2017; Han & Sug-Shin, 2016) and their satisfaction during the teaching-learning process (Ramírez-Correa et al., 2017), and bring about positive effects on interaction and collaboration (Dias et al., 2017; Hadjileontiadou et al., 2015). Watson and Watson (2007) affirm that the defining characteristics of the LMS are based on the ability to provide individualized teaching units incorporated into a standardized curriculum. Courses are expanded taking into account the progression of levels and degrees of difficulty, individual compilations of student results, and finally, a teaching-learning process configured with a focus on the particularities of each student.

Santiago and Bergman (2018) explain the use of the LMS as a cycle based on learning analytics:

• Students’ use of an LMS allows effective interaction with the teacher, the optimal distribution of content, and the individualization of the assessment process.
• The data is collected and stored in a database, which can be used to make predictions that guide the learning process and provide feedback that guides the student towards the achievement of the objectives. Students receive digital teaching material tailored to their interests and needs.
• Teachers manage and develop the entire process, and are able to intervene whenever they see fit.

Currently, there are many LMSs available for use in education. Moodle, one of the most relevant platforms, focuses on the social constructivist perspective of learning, allowing it to integrate tools for individualized creation of H5P content, while also being compatible with PHP7 (Kerimbayev et al., 2017). Google Classroom, meanwhile, has an increasing number of users within the educational field thanks to the effectiveness of the compendium of Google Apps for Education applications, all organized as a cloud-based educational management tool (Ramadhani et al., 2019). On the other hand, Chamilo is a tool based on the use of resources in the Sharable Content Object Reference Model (SCORM) format and is much more focused on interactive and social aspects (Ardila et al., 2015). Also worth noting is WordPress, a digital tool that is not an LMS itself but can combine with certain utilities allowing them to be used as such, in a simple and intuitive way (Harris & Rea, 2019).

In recent years, research has emerged that analyzes the specific effects of using LMSs on flipped learning methodology, showing that LMSs are a permanent, key factor in optimizing flipped learning. Nouri et al. (2019) explain that these systems allow the students’ tasks to be monitored while allowing individualized comments to be made to support their learning progress. In this way, the flipped classroom takes advantage of these management tools to insert didactic sessions in video and digital content, enabling increased student engagement with content and materials and enhancing student self-regulation and the teacher’s capacity for intervention.

In a similar way, Andujar et al. (2020) place special emphasis on the insight provided to teachers as to the student’s current position in the teaching-learning process, in addition to the fact that teachers are able to exercise control specific to the actions carried out by students within the management platform when working with the curricular materials or performing tasks. Karaca and Akif (2017) affirm that the increase in teacher interaction becomes especially important during corrections of the evaluation tests, since the management platforms allow the valid answers to be disclosed and make it possible to insert video links with specific explanations or redirect students to various materials individually to plug any gaps in their learning.

Likewise, other authors who have analyzed the effects of using LMSs in flipped learning have found improvements in student engagement and participation, all based on a self-reflective and self-evaluative learning culture that favors the performance of activities based on problem solving (Wang, 2017). This pedagogical combination allows students to become involved and motivated in the teaching-learning process based on a varied and dynamic interaction with the assessment tests (multiple choice, true and false, short answer, open-ended answer, etc.) and with the curricular content and learning materials (Karaca & Akif, 2017).

1.1 Objectives and research questions

In recent years, the impact literature has revealed that flipped learning and LMS platforms have great potential in teaching-learning processes, as shown by their influence on various academic indicators such as social interaction, motivation to learn, cognitive aspects, and the students’ own grades in assessment tests (Awidi & Paynter, 2019; Boelens et al., 2018; Bognar et al., 2019; DeLozier & Rhodes, 2017; Hinojo-Lucena et al., 2018; Sola et al., 2019; Huang et al., 2018; Karabulut et al. 2018; MacLeod et al., 2018; Shih & Tsai, 2017; Touron & Santiago, 2015).

This study aims to analyze the academic incidence of flipped learning through LMS platforms compared to the use of flipped learning in other non-interactive digital media. The following specific objectives are derived from the general line of inquiry for the research. Likewise, the various research questions (RQ) associated with each objective are presented below.

1. To determine the methodological influence at the social level.
   • RQ1: Does the type of flipped learning influence the possibility of working with other colleagues?
   • RQ2: Does the type of flipped learning influence interaction between students?
2. To find out the methodological influence at the cognitive level.
   • RQ3: Does the type of flipped learning influence critical thinking work?
   • RQ4: Does the type of flipped learning influence interaction with the teacher?
   • RQ5: Does the type of flipped learning improve access to learning content?
   • RQ6: Does the type of flipped learning influence the quality of materials for the learning style?
   • RQ7: Does the type of flipped learning influence the scope of knowledge in different ways?
3. To discover the methodological influence at the motivational level.
   • RQ8: Does the type of flipped learning influence a formative action according to the needs of the students?
   • RQ9: Does the type of flipped learning influence the process by which decisions are made on student learning?
   • RQ10: Does the type of flipped learning influence whether the teacher takes into account the interests and strengths of the students?
   • RQ11: Does the type of flipped learning influence the effectiveness and practicality of learning?
   • RQ12: Does the type of flipped learning influence teachers’ knowledge of students?
4. To gain insight into the methodological influence on student grades.
   • RQ13: Does the type of flipped learning influence the grades achieved by students in assessment tests?

II. Methodology

The scope of the objectives and the answers to the research questions were pursued through a post-test quasi-experimental design with a descriptive and correlational scope, based on a quantitative research methodology. To perform this research adequately, we followed the guidelines of experts in this type of study (Hernandez et al., 2014; Rodriguez, 2011). Likewise, we employed the analytical structure of studies in recent years in order to implement a study model validated by experts (Hinojo et al., 2020; Pozo et al., 2019). Two study groups – a control group (CG) and an experimental group (EG) – were established in this research. The CG engaged in flipped learning with audio-visual content hosted on a non-interactive website. The EG, on the other hand, used flipped learning with an LMS to house the ready-made teaching materials. This design gives rise to two types of variables: the medium where the contents are housed as the independent variable, and the efficacy obtained in the academic indicators as the dependent variable.

Basic statistics such as the mean (M) and standard deviation (SD) were used. The distribution trend was determined with the skewness (Skw) and kurtosis (Kme) tests. The means between the analysis groups were compared with Student’s t test (tn1 + n2 - 2). Furthermore, the strength of association and the size of the effect were obtained with Cohen’s d and the biserial correlation (rxy). The data were analyzed with Statistical Package for the Social Sciences (SPSS) v.25, with a significance level of p < 0.05.

A total of 231 Spanish secondary school students participated in this didactic learning experience. These subjects were chosen through purposive sampling, due to the researchers’ close contact with the school in question. The sample size in this type of research does not influence, alter, or bias the results; this has been previously verified by experts (Chou & Feng, 2019; Yılmaz & Soyer, 2018).

The selected sample was made up of 41.9% boys and the rest girls, with an average age of 13 years (SD = 1.48), in the first year of secondary education in the Spanish education system.

Eight groups of students with the aforementioned characteristics were taken. The composition of the groups was natural, that is, they corresponded exactly to the groups taught in the school at that level (n = 8). This configuration gave rise to two group types. A total of four groups were formed as control groups (CG1, n = 30; CG2, n = 30; CG3, n = 29; CG4, n = 27), while four others were experimental groups (EG1, n = 30; EG2, n = 29; EG3, n = 29; EG4, n = 27). Students were randomly assigned to each type of group. The experience was carried out over the last three months of the 2019-2020 school year.

An ad hoc questionnaire was designed and used to collect data, with various instruments widely validated by the scientific literature taken as a reference (den Brok et al., 2006; Kyrgiridis et al., 2014; Watson & Glaser, 2002). Ad hoc questionnaires are specifically designed for research, based on other validated questionnaires from the scientific literature (Achilli, 2005).

The final questionnaire is made up of two well-differentiated blocks. The first block encompasses all the classic sociodemographic variables in educational research (gender, age, educational stage). The second block covers other items, ordered according to the different dimensions of the study objective, notably the ability of flipped learning to develop:

• Peer work: possibilities for collaborative work with other classmates.
• Peer interaction: development of interaction between students.
• Critical thinking: development of analytical capacity and rational thinking.
• Teacher interaction: development of interaction and communication between teacher and students.
• Content access: ease of accessing physical and digital teaching materials.
• Learning materials: adaptability of teaching materials to different learning styles.
• Knowledge forms: potential to demonstrate knowledge in various ways.
• Pace: potential for students to work at their own pace.
• Decisions: ability to empower the student to make decisions.
• Interests-strengths: teacher’s ability to consider the interests and strengths of the student.
• Effectiveness: potential to promote active and experiential learning.
• Student knowledge: teacher’s ability to obtain more information about and better know students.
• Student grades: final grades obtained at the end of the course by students.

To select the various items that were to comprise the final questionnaire, we followed a number of inclusion criteria. On the one hand, we included items that allowed us to collect data on the different dimensions that made up the objectives of the study. At the same time, these items should be concise and short, to avoid tiring students. Exclusion criteria were also established, eliminating, for example, any items that may have led to confusion or which were discarded by the expert panel that we consulted, thus avoiding an excessive number of questions.

The expert panel gave a few recommendations to improve the questionnaire, like reducing the number of questions or rephrasing some items. Feedback was analyzed with Kendall’s W to determine the concordance and relevance of the judgments made (W = 0.83). This is a classic statistic employed to evaluate concordance between experts (Craigie et al., 2002).

The questionnaire employs a 4-point Likert scale. A 5-point scale was not used to prevent participants from tending to respond centrally (Matas, 2018). Finally, Table 1 presents the reliability and validity scores obtained.

Table 1. Reliability and validity index scores for the attitudinal factor

After participants were assigned to the study groups, the different educational methodologies were applied. The CG students carried out a didactic unit based on flipped learning with audio-visual content that did not allow interaction between students or with the teacher. They could only view the teaching materials hosted on a web server. The EG students carried out a didactic unit based on flipped learning with instructional material on an LMS platform that allowed all kinds of interactions (chat messaging, thematic discussion forums, private messages to the teacher, private messages between students, and public messages in each of the sections that contained the videos).

The same type of class activities were carried out in all the groups. The face-to-face educational tasks focused on inquiry, collaboration, and problem solving by students. The audio-visual educational material was prepared by two teachers from the school in question, experts in this type of active learning methodology. These same teachers implemented the necessary methodology to carry out the research.

Once the didactic unit was completed, the questionnaire was administered in order to collect measurements. The data collected was statistically processed to achieve precise results to achieve the proposed objectives and answer each of the research questions.

This research was framed in accordance with the ethical principles expressed in the Declaration of Helsinki. Prior to involving the students, the research project was approved by the academic committee of the school itself, and informed consent was obtained from the participants.

III. Results

The results found in the descriptive analysis show that the means achieved by the experimental group are superior to those of the control group in all the dimensions analyzed. The averages achieved in all dimensions, both in the control group and in the experimental group, are above 2.5, which demonstrates a medium-high evaluation by students of the general application of flipped learning. The distribution of the sample is normal, since the values of asymmetry and kurtosis are between -1.96 and +1.96 (Jöreskog & Moustaki, 2001). The standard deviation of both the control and experimental group dimensions shows an equal impact among the students. All cases are platikurtic, except for the students’ decision-making about their learning, which is mesokurtic (Table 2).

Table 2. Results obtained for the study dimensions in the Control Group and Experimental Group

The relationship among students in the experimental group shows significant differences in all the dimensions of the study. Students in the experimental group obtained higher values than the control group in almost all variables. However, similar results were observed between the two groups in the variables “Decisions”, “Interests-strengths”, and “Student knowledge” (Figure 1).

Figure 1. Comparison between Control Group and Experimental Group

In this study, we analyzed the independence between classic flipped classrooms and LMS-associated flipped classrooms, applying Student’s t tests for independent samples. The data shown in the analysis demonstrate a very significant impact from student-teacher interaction and access to learning content being enhanced by the use of an LMS in conjunction with the flipped learning method. This shows the strength of association between media.

The other dimensions show significant associations, except in students’ possibilities to make decisions about their learning and in teachers’ knowledge of the interests and strengths of students, where there is no significant relationship. The strength of association in this case is low. The size of the effect, according to Cohen’s d, is very low in all the dimensions analyzed (Table 3).

Table 3. Independence between the Control Group and Experimental Group

IV. Discussion

The use of LMSs to apply the flipped learning methodology is increasingly widespread among the teaching community as a way to optimize the results of this methodological approach (Andujar et al., 2020). In addition, an LMS overcomes the limitation posed by flipped learning in terms of access and usability (Almarashdeh, 2016; Rhode et al., 2017). The scientific literature has been nourished – especially in recent years – by various studies that have analyzed the impact of flipping learning moments together with these management platforms (Harris & Rea, 2019; Kerimbayev et al., 2017; Ramadhani et al., 2019).

As far as the interactive dimension is concerned, the results of this study show that the possibilities of working with other colleagues and the degree of interaction between colleagues are high in a flipped classroom, but this interaction becomes even more frequent if management tools are used during the teaching-learning process. These results are in line with other studies that have analyzed this aspect (DeLozier & Rhodes, 2017; MacLeod et al., 2018).

Regarding the cognitive aspect, and along the same lines as other studies, using an LMS during the application of flipped learning yields higher levels of optimization in dimensions related to critical thinking (Wang, 2017) and the degree of interaction with the teaching staff (Karaca & Akif, 2017; Santiago & Bergman, 2018). Likewise, positive results have also been obtained regarding access to content, the quality of learning materials and resources, and the variability of forms of knowledge acquisition, in line with findings reported by other authors (Karabulut et al., 2018; Karaca & Akif, 2017; Santiago & Bergman, 2018).

On the other hand, it is pertinent to highlight that, of all the dimensions analyzed, those related to motivation reflected the lowest level of significance. This study obtained relevant results from the application of an LMS in dimensions related to the adaptability of the methodology to student needs and in relation to the effectiveness and practicality of learning. These results are similar to those found in the scientific literature (López et al., 2019; Santiago & Bergman, 2018). However, it has not been possible to verify that the use of an LMS in flipped learning affects dimensions related to decision-making, nor were we able to confirm that using an LMS helps to boost teachers’ knowledge of their students or teachers’ capacity to take into account the interests and strengths of students, contrary to the results of other similar studies (Bognar et al., 2019; Mengual-Andres et al., 2020; Santiago & Bergman, 2018).

Regarding academic performance, the scores obtained by students who have used an LMS are slightly higher than those of students who have used digital media in the flipped classroom. This is consistent with results found in the scientific literature (Hinojo-Lucena et al., 2018; Sola et al., 2019).

V. Conclusions

This research has analyzed the impact of the use of an LMS or other digital media in the application of the flipped learning methodology at an interactive, cognitive, motivational, and academic performance level. At all these levels, the results allow us to conclude that using an LMS while applying the flipped learning methodology contributes to an optimization of the general teaching-learning process, particularly in interactive, cognitive, motivational, and academic performance aspects. The use of an LMS also makes it possible to overcome limitations related to student access to didactic contents.

The main limitations encountered in implementing this research lie in the fact that the selected sampling technique was purposive. Thus, no sampling was carried out of the total population, so we recommend generalizing the results found in the study or extrapolating them to population samples with significant differences. On the other hand, access to the different schools was complex due to an excess of bureaucracy and poor training of teachers in some schools that had accepted the proposed intervention but lacked qualified staff to put it into practice.

Despite the aforementioned limitations, this study aims to show and offer the scientific community and the teaching community an effective pedagogical approach in flipped learning, which can be greatly benefited and enhanced if combined with the use of an LMS. Therefore, our intention is to offer a forward-looking analysis that disseminates the benefits of using this methodology and encourages the scientific community to carry out contrasting analyses with respect to the data obtained in this study, from population samples of different characteristics. For this reason, as a future line of research, the team of researchers is developing a pilot study on the application of the flipped learning method with an LMS as part of a multi-stage analysis that considers the particularities of applying the methodology based on educational stage.

This study has a series of implications, both theoretical and practical. On a theoretical level, this study contributes to the increase in scientific literature on the use of flipped learning, as a methodology, through virtual teaching platforms. This type of study provides more data to the scientific community. In addition, the research method employed, with a control group and an experimental group, may in the future enable meta-analyses with similar studies. Also, the theoretical framework developed may serve as a reference and a basis for other researchers who wish to carry out research on this line of study.

On a practical level, this study may have various implications in education. The teaching method analyzed can be applied in various educational contexts, particularly at present, considering that the COVID-19 pandemic has led to changes in teaching-learning processes. For this reason, the teaching method analyzed can be a very efficient resource for developing the educational process with confidence. In addition, using virtual teaching platforms with the flipped learning method facilitates interaction between all those involved, which will allow for a more individualized and personalized knowledge of students. This brings with it more appropriate and individualized teaching. Finally, the data obtained in this study can serve as a basis for institutions and educational bodies to establish pedagogical training plans, and for adapting and adjusting techno-pedagogical resources for future development, with a guarantee of success.

Referencias

Achilli, E. L. (2005). Investigar en antropología social. Los desafíos de transmitir un oficio [Conducting research in social anthropology. The challenges of passing on a trade]. Laborde.

Almarashdeh, I. (2016). Sharing instructors experience of learning management system: A technology perspective of user satisfaction in distance learning course. Computers in Human Behavior, 63, 249-255. https://doi.org/10.1016/j.chb.2016.05.013

Andujar, A., Salaberri-Ramiro, M. S., & Cruz, M. S. (2020). Integrating flipped foreign language learning through mobile devices: Technology acceptance and flipped learning experience. Sustainability, 12(3), 1-12. https://doi.org/10.3390/su12031110

Ardila, J. Y., Ruiz, E. M., & Castro, I. L. (2015). Estudio comparativo de sistemas de gestión del aprendizaje: Moodle, ATutor, Claroline, Chamilo y Universidad de Boyacá [A comparative study of learning management systems: Moodle, ATutor, Claroline, Chamilo, and Universidad de Boyacá]. Revista Academia y Virtualidad, 8(1), 54-65. https://doi.org/10.18359/ravi.445

Awidi, I. T., & Paynter, M. (2019). The impact of a flipped classroom approach on student learning experience. Computers & Education, 128, 269-283. https://doi.org/10.1016/j.compedu.2018.09.013

Bauer-Ramazani, C., Graney, J. M., Marshall, H.W., & Sabieh, C. (2016). Flipped learning in TESOL: Definitions, approaches, and implementation. Tesol Journal, 7(2), 429-437. https://doi.org/10.1002/tesj.250

Bergmann, J., & Sams, A. (2012). Flip your classroom: Reach every student in every class every day. ISTE.

Biech, E. (2008). American Society for Training & Development handbook for workplace learning professionals. ASTD Press.

Boelens, R., Voet, M., & de Wever, B. (2018). The design of blended learning in response to student diversity in higher education: Instructors’ views and use of differentiated instruction in blended learning. Computers & Education, 120, 197-212. https://doi.org/10.1016/j.compedu.2018.02.009

Bognar, B., Sablić, M., & Škugor, A. (2019). Flipped learning and online discussion in higher education teaching. In L. Daniela (Ed.) Didactics of Smart Pedagogy. Springer. https://doi.org/10.1007/978-3-030-01551-0_19

Chou, P. N., & Feng, S. T. (2019). Using a tablet computer application to advance high school students’ laboratory learning experiences: A focus on electrical engineering education. Sustainability, 11(2), 1-14. https://doi.org/10.3390/su11020381

Chyr, W. L., Shen, P. D., Chiang, Y. C., Lin, J. B., & Tsia, C. W. (2017). Exploring the effects of online academic help-seeking and flipped learning on improving students’ learning. Educational Technology & Society, 20(3), 11-23. https://www.jstor.org/stable/26196116

Craigie, M., Loader, B., Burrows, R., & Muncer, S. (2002). Reliability of health information on the Internet: An examination of experts’ ratings. Journal of Medical Internet Research, 4(1), 1-15. https://doi.org/10.2196/jmir.4.1.e2

DeLozier, S. J., & Rhodes, M. G. (2017). Flipped classrooms: a review of key ideas and recommendations for practice. Educational Psychology Review, 29, 141-151. https://doi.org/10.1007/s10648-015-9356-9

den Brok, P., Brekelmans, M. & Wubbels, T. (2006). Multilevel issues in research using students’ perceptions of learning environments: The case of the Questionnaire on Teacher Interaction. Learning Environments Research, 9(3), 199-213. https://doi.org/10.1007/s10984-006-9013-9

Dias, S. B., Hadjileontiadou, S. J., Diniz, J. A., & Hadjileontiadis, L. J. (2017). Computer-based concept mapping combined with learning management system use: An explorative study under the self- and collaborative-mode. Computers & Education, 107, 127-146. https://doi.org/10.1016/j.compedu.2017.01.009

El Miedany, Y. (2019). Flipped learning. In C. Reidsema, L. Kavanagh, R. Hadgraft & N. Smith (Eds.), The flipped classroom: Practice and practices in higher education (pp. 285-303). Springer. https://doi.org/10.1007/978-3-319-98213-7_15

Froehlich, D. E. (2018). Non-technological learning environments in a technological world: Flipping comes to the aid. Journal of New Approaches in Educational Research, 7(2), 88-92. https://doi.org/10.7821/naer.2018.7.304

Fuentes, A., Parra-González, M. E., López, J., & Segura-Robles, A. (2020). Educational potentials of flipped learning in intercultural education as a transversal resource in adolescents. Religions, 11(1), 1-13. https://doi.org/10.3390/rel11010053

Gallagher, P. S. (2007). Assessing SCORM for its affordances in facilitating a simulation as a pedagogical model [Doctoral dissertation], George Mason University. Dspace Repository.

Gilhooly, K. (2001). Making e-learning effective. Computerworld, 35(29), 52-53.

Hadjileontiadou, S. J., Dias, S. B., Diniz, J. A., & Hadjileontiadis, L. J. (Eds.). (2015). Fuzzy logic-based modeling in collaborative and blended learning. Information Science Reference, an imprint of IGI Global.

Hall, B. (2002). Six steps to developing a successful e-learning initiative: Excerpts from the e-learning guidebook. In A. Rossett (Ed.), The ASTD E-learning Handbook (pp. 234-250). McGrawHill.

Han, I., & Sug-Shin, W. (2016). The use of a mobile learning management system and academic achievement of online students. Computers & Education, 102, 79-89. https://doi.org/10.1016/j.compedu.2016.07.003

Harris, A. L., & Rea, A. (2019). Web 2.0 and virtual world technologies: A growing impact on IS education. Journal of Information Systems Education, 20(2), 137-144. https://aisel.aisnet.org/jise/vol20/iss2/3

He, W., Holton, A., Farkas, G., & Warschauer, M. (2016). The effects of flipped instruction on out-of-class study time, exam performance, and student perceptions. Learning and Instruction, 45, 61-71. https://doi.org/10.1016/j.learninstruc.2016.07.001

Hernandez, R., Fernandez, C., & Baptista, P. (2014). Metodología de la investigación [Research methodology]. McGraw-Hill.

Hinojo, F. J., López, J., Fuentes, A., Trujillo, J., & Pozo, S. (2020). Academic effects of the use of flipped learning in physical education. International Journal of Environmental Research and Public Health, 17(1), 1-14. https://doi.org/10.3390/ijerph17010276

Hinojo-Lucena, F. J., Mingorance-Estrada, A. C., Trujillo-Torres, J. M., Aznar-Díaz, I., & Caceres, M. P. (2018). Incidence of the flipped classroom in the physical education students’ academic performance in university contexts. Sustainability, 10(5), 1-13. https://doi.org/10.3390/su10051334

Huang, B., Foon, K., & Kwan, C. (2018). Investigating the effects of gamification-enhanced flipped learning on undergraduate students’ behavioral and cognitive engagement. Interactive Learning Environments, 27(8), 1106-1126. https://doi.org/10.1080/10494820.2018.1495653

Hwang, G. J., & Lai, C. L. (2017). Facilitating and bridging out-of-class and in-class learning: An interactive e-book-based flipped learning approach for math courses. Journal of Educational Technology & Society, 20(1), 184-197. http://www.ifets.info/journals/20_1/17.pdf

Hwang, G.J., Lai, C.L., & Wang, S.Y. (2015). Seamless flipped learning: a mobile technology-enhanced flipped classroom with effective learning strategies. Journal of Computers in Education, 2(4), 449-473. https://doi.org/10.1007/s40692-015-0043-0

Jöreskog, K. G., & Moustaki, I. (2001). Factor analysis of ordinal variables: A comparison of three approaches. Multivariate Behavioral Research, 36(3), 347-387. https://doi.org/10.1207/S15327906347-387

Karabulut, A., Jaramillo, N., & Hassall, L. (2018). Flipping to engage students: Instructor perspectives on flipping large enrolment courses. Australasian Journal of Educational Technology, 34(4), 123-137. https://doi.org/10.14742/ajet.4036

Karaca, C., & Akif, M. (2017). Effect of flipped learning on cognitive load: A higher education research. Journal of Learning and Teaching in Digital Age, 2(1), 20-27. https://dergipark.org.tr/tr/pub/joltida/issue/55466/760080

Kerimbayev, N., Kultan, J., Abdykarimova, S., & Akramova, A. (2017). LMS Moodle: Distance international education in cooperation of higher education institutions of different countries. Education and information technologies, 22(5), 2125-2139. https://doi.org/10.1007/s10639-016-9534-5

Kyrgiridis, P., Derri, V., Emmanouilidou, K., Chlapoutaki, E., & Kioumourtzoglou, E. (2014). Development of a questionnaire for self-evaluation of teacher effectiveness in physical education (SETEQ-PE). Measurement in Physical Education and Exercise Science, 18(2), 73-90. https://doi.org/10.1080/1091367X.2013.866557

Llanos, G., & Bravo-Agapito, J. (2017). Flipped classroom como puente hacia nuevos retos en la educación primaria [The flipped classroom as a bridge to new challenges in primary education]. Revista Tecnología, Ciencia y Educación, 8, 39-49. https://www.tecnologia-ciencia-educacion.com/index.php/TCE/article/ view/153

Long, T., Cummins, J., & Waugh, M. (2017). Use of the flipped classroom instructional model in higher education: Instructors’ perspectives. Journal of Computing in Higher Education, 29(2), 179-200. https://doi.org/10.1007/s12528-016-9119-8

López, J., Pozo, S., & del Pino, M. J. (2019). Projection of the flipped learning methodology in the teaching staff of cross-border contexts. Journal of New Approaches in Educational Research, 8(2), 184-200. https://doi.org/10.7821/naer.2019.7.431

López-Belmonte, J., Pozo, S., Fuentes, A., & López, J. A. (2019). Creación de contenidos y flipped learning: un binomio necesario para la educación del nuevo milenio [Content creation and flipped learning: A necessary combination for education in the new millennium]. Revista Española de Pedagogía, 77(274), 535-555. https://doi.org/10.22550/REP77-3-2019-07

MacLeod, J., Yang, H. H., Zhu, S., & Shi, Y. (2018). Technological factors and student-to-student connected classroom climate in cloud classrooms. Journal of Educational Computing Research, 56(6), 826-847. https://doi.org/10.1177/0735633117733999

Matas, A. (2018). Diseño del formato de escalas tipo Likert: un estado de la cuestión [Likert-type scale format design: State of the art]. Revista Electrónica de Investigación Educativa, 20(1), 38-47. https://doi.org/10.24320/redie.2018.20.1.1347

Mengual-Andres, S., López, J., Fuentes, A., & Pozo, S. (2020). Modelo estructural de factores extrínsecos influyentes en el flipped learning [Structural model of extrinsic factors that influence flipped learning]. Educación XX1, 23(1), 75-101. https://doi.org/10.5944/educXX1.23840

Mwalumbwe, I., & Mtebe, J. S. (2017). Using learning analytics to predict students’ performance in Moodle learning management system: A case of Mbeya University of Science and Technology. The Electronic Journal of Information Systems in Developing Countries, 79(1), 1-13. https://doi.org/10.1002/j.1681-4835.2017.tb00577.x

Nouri, J., Saqr, M., & Fors, U. (2019). Predicting performance of students in a flipped classroom using machine learning: Towards automated data-driven formative feedback. Systemics, Cybernetics and Informatics, 17(2), 17-21. http://www.iiisci.org/journal/CV$/sci/pdfs/EB614LI19.pdf

Pozo, S., López, J., Moreno, A. J., & López, J. A. (2019). Impact of educational stage in the application of flipped learning: A contrasting analysis with traditional teaching. Sustainability, 11(21), 1-15. https://doi.org/10.3390/su12093729

Pozo, S., López-Belmonte, J., Moreno-Guerrero, A. J., Sola, J. M., & Fuentes, A. (2020). Effect of bring-your-own-device program on flipped learning in higher education students. Sustainability, 12(9), 1-11. https://doi.org/10.3390/su12093729

Ramadhani, R., Umam, R., Abdurrahman, A., & Syazali, M. (2019). The effect of flipped-problem based learning model integrated with LMS-Google classroom for senior high school students. Journal for the Education of Gifted Young Scientists, 7(2), 137-158. https://doi.org/10.17478/jegys.548350

Ramírez-Correa, P. E., Rondan-Cataluña, F. J., Arenas-Gaitán, J., & Alfaro-Perez, J. L. (2017). Moderating effect of learning styles on a learning management system’s success. Telematics and Informatics, 34(1), 272-286. https://doi.org/10.1016/j.tele.2016.04.006

Rhode, J., Richter, S., Gowen, P., Miller, T., & Wills, C. (2017). Understanding faculty use of the learning management system. Online Learning, 21(3), 68-86. http://dx.doi.org/10.24059/olj.v21i3.1217

Rodriguez, N. (2011). Diseños experimentales en educación [Experimental designs in education]. Revista de Pedagogía, 32(91), 147-158. http://saber.ucv.ve/ojs/index.php/rev_ped/article/view/5248

Santiago, R., & Bergmann, J. (2018). Aprender al revés: Flipped Learning 3.0 y metodologías activas en el aula [Learning the other way round: Flipped Learning 3.0 and active methodologies in the classroom]. Paidós Educación.

Schmidt, S. M. P., & Ralph, D. L. (2016). The flipped classroom: A twist on teaching. Contemporary Issues in Education Research, 9(1), 1-6. https://doi.org/10.19030/cier.v9i1.9544

Segura-Robles, A., Fuentes-Cabrera, A., Parra-González, M. E., & López-Belmonte, J. (2020). Effects on personal factors through flipped learning and gamification as combined methodologies in secondary education. Frontiers in Psychology, 11, 1-8. https://doi.org/10.3389/fpsyg.2020.01103

Shih, W., & Tsai, C. (2017). Students' perception of a flipped classroom approach to facilitating online project-based learning in marketing research courses. Australasian Journal of Educational Technology, 33(5), 32-49. https://doi.org/10.14742/ajet.2884

Sola, T., Aznar, I., Romero, J. M., & Rodriguez, A. M. (2019). Eficacia del método flipped classroom en la universidad: Meta-análisis de la producción científica de impacto [Efficacy of the flipped classroom method in universities: Meta-analysis of impact scientific production]. Revista Iberoamericana sobre Calidad, Eficacia y Cambio en Educación, 17(1), 25-38. https://doi.org/10.15366/reice2019.17.1.002

Touron, J., & Santiago, R. (2015). El modelo Flipped Learning y el desarrollo del talento en la escuela [The flipped learning model and talent development in schools]. Revista de Educación, 1(368), 196-231. https://doi.org/10.4438/1988-592X-RE-2015-368-288

Tse, W. S., Choi, L. Y. A., & Tang, W. S. (2019). Effects of video‐based flipped class instruction on subject reading motivation. British Journal of Educational Technology, 50(1), 385-398. https://doi.org/10.1111/bjet.12569

Wang, F. H. (2017). An exploration of online behaviour engagement and achievement in flipped classroom supported by learning management system. Computers & Education, 114, 79-91. https://doi.org/10.1016/j.compedu.2017.06.012

Watson, G. B., & Glaser, E. M. (2002). Watson–Glaser critical thinking appraisal. The Psychological Corporation.

Watson, W. R., & Watson, S. L. (2007). An argument for clarity: What are learning management systems, what are they not, and what should they become? TechTrends, 51(2), 28-34. https://doi.org/10.1007/s11528-007-0023-y

Yılmaz, A., & Soyer, F. (2018). Effect of physical education and play applications on school social behaviors of mild-level intellectually disabled children. Education Sciences, 8(2), 1-8. https://doi.org/10.3390/educsci8020089

Yilmaz, R. (2017). Exploring the role of e-learning readiness on student satisfaction and motivation in flipped classroom. Computers in Human Behavior, 70, 251-260. https://doi.org/10.1016/j.chb.2016.12.085

Zainuddin, Z., Habiburrahim, H., Muluk, S., & Keumala, C. M. (2019). How do students become self-directed learners in the EFL flipped-class pedagogy? A study in higher education. Indonesian Journal of Applied Linguistics, 8(3). https://doi.org/10.17509/ijal.v8i3.15270