Key words: Computer uses in education, collaborative learning, computer software evaluation, groupware, computer mediated communication.

Without entering the debate on this issue, the question reveals an imminent problem: the inadequate relationship between developers of technology for education and users of the technology (teachers, students, directors). In most cases, the developers do not have an explicit pedagogical conception or they design their products with purposes different to those of educators. On the other hand, users constantly have to adapt their educational objectives and needs to the technological tools available. Educational developments historically run behind technological developments; this makes an a posteriori evaluation of technologies necessary to determine whether they were used for the most appropriate pedagogical model.

The use of the Internet with educational purposes is new and limited in countries like Mexico; as a consequence, there are few evaluative studies of the devices used and the results obtained in learning experiences. A first problem posed by the evaluation of educational experiences in Web environments concerns the decision of what to evaluate, since factors intervene in these experiences different to those in conventional educational processes. Clearly, the evaluation of learning results is essential both in conventional experiences as well as in those that make use of computers and new technologies; nevertheless, a determining factor for the success of a computer assisted learning experience is the degree to which the software devices are tailored to the aims of the experience. As mentioned before, these devices are rarely conceived specifically for a particular educational design.

The suitability of a device to the aims of a specific didactic experience can usually be verified through three distinct yet complementary procedures:

a) The appraisal by participants in the learning experience of the device’s performance.
b) Verifying the requirements of the device, taking as a starting point the theoretical model used as the foundation for the design of the learning situation.
c) The comparison of standards that define the design, development and evaluation of software with the characteristics of the device used.

a) each collaborative group is composed of three subgroups of experts;
b) each subgroup comes from a different school (Mexican of Canadian);
c) each subgroup has from three to five students from one same school.

The first Web device used in the Tactics project was the e-group. It is free access software that runs on the Internet through the Yahoo site (http://mx.yahoo.com). Although it is clearly not an education platform, this device was chosen because its free character allowed us to explore the large-scale application of a design within the educational system at the same time as it permitted the management and supervision of student participation.

Access to e-groups is similar to subscribing to Yahoo electronic mail: a user-name and password are established; from there, one can form a new group or join an existing one. The group offers the following services for communication at a distance and shared work: individual messages and discussion lists; chat; database and shared files; polls; space for storing photos; agenda; links of interest. Figure 3 shows an e-group homepage.

For the aims of the Tactics project, one e-group was defined for each base team; each student was assigned a user number, a password, and an email address within the “yahoo.com” domain. Students could make use of all services in the group; the administrator and owner of all groups was a project researcher.

b) WebCT

The second Web device used by Tactics was Web Course Tools (WebCT). Originally developed to support the needs of the University of British Columbia, it complies with the characteristics of an education at a distance platform (Ministère de l’Education Nationale, de la Recherche et de la Technologie, 1999). Currently, WebCT is commercially distributed at a cost that ranges from 500 to 5000 dollars according to the quantity of users.

WebCT is a computer server that uses Internet and a series of tools that permit the design and development of interactive courses as a complement to teaching in the classroom. These tools add communication and evaluation activities in a structured and contextualized way, as well as a certain number of utilities that complement the student’s workspace. WebCT anticipates three kinds of participation: administrator, course creator or instructor, and student. There is also an additional role, that of the tutor who intervenes in the teaching to evaluate and comment on students’ work.

A course was defined for the Tactics project on the University of Montreal’s5 WebCT6platform (jointly responsible with CINVESTAV for the Tactics project). It was adapted as a space for interaction and exchange, but not linked to specific curricular contents. Figure 4 shows University of Montreal’s electronic service homepage.

The platform administrator (a researcher from the project) assigns a username and password for entering the area.

The space used by Tactics offers the following services, similar to those of e-groups: chat, discussion forums for communication within the base team and among expert groups; personal messages and messages for the work group; shared files for documents created by the students; space for personal homepages; spaces for participating schools; agenda. Figure 5 shows how these services are presented on the screen.

Although the space is general for all participating students, each base team belongs to a specific group and has access only to the work created within it, although it has the possibility of sending emails to all students in the project. The chat has three spaces available: the first one allows communication in real time with the base teams; the second one communicates to all Tactics project participants; and the third one is shared with all University of Montreal students. The first chat can be recorded for supervisory ends.

The questionnaire has two sections; the first one seeks to characterize the level of knowledge and familiarity with Internet use and with the different services by teachers involved in the Tactics project. The second one is geared towards the comparison of both tools used in the project; this part is divided into six subsections:

a) functionality,
b) efficacy and usability,
c) effects on the individual,
d) effects on teamwork, and
e) pedagogical effectiveness.

It should be noted that a general 19% chose the “no opinion” option, which could mean that teachers as well as researchers have certain difficulties to visualize technology as a tool articulated to their work and to define their own role in the face of technology.

The differentiated answers of teachers and teacher-researchers deserve more detailed analysis. Teachers’ answers show that, in their view, WebCT is better adapted to collaborative work (in the categories: efficacy and usability 77%; effects on individual aspects 75%, effects on work in group 55% and pedagogical effectiveness 67%). Conversely, researchers (who, as noted before, are more familiar with the Internet and have a global vision of the project) do not make a distinction between e-groups and WebCT.

Teachers’ and researchers’ differentiated answers are similar to those reported in other studies made for CSCW (Potts et al., 1999), where it has been seen that previous experience and work practices allow users to avoid or overcome the tool’s problematic areas.

If the devices analyzed had been developed for this project, we could say that there were problems of design in both, since even when teachers lean towards WebCT, a consensus of opinions between teachers and researchers would be expected in aspects such as “effects on teamwork” and “pedagogical effectiveness”, central to this experience; as noted before, this consensus did not come about.

This sort of design fault, which evinces the software’s lack of adaptation to the task, has been reported in other areas of CSCW (Scott, 1997; Potts et al., 2002). It has been pointed out that to overcome these faults, the software must be an improvement on the work system already developed by the user group, which is why a greater understanding of how work is organized and developed is required.

In this case, support for coordinating the interactions between expert groups (which is not reflected in the answers on the effects of teamwork) is an important deficiency in e-groups as well as in WebCT, since these interactions are central to the “jigsaw” model structured by the Tactics educational design.

On the other hand, our study tends to indicate that the user (teacher or researcher) has still not incorporated into his or her pedagogical reflections which requirements software should have to foster teamwork and each participant’s learning in the collaborative experience. This happens despite the fact that the importance of analyzing the capacities of software on learning results has been suggested in some studies on the use of groupware in educational processes (Alavi, 1994). This same author suggests that: “This line of research would have important implications for the development of software environments that are directed towards supporting learning group processes” (p.172).

In the case presented here, both devices have advantages and disadvantages with regards to the proposed pedagogical design. The fact that e-groups have no cost paradoxically presents an advantage and a disadvantage: although its free-of-charge aspect does not weaken the support for communications among different groups, the lack of greater autonomy in the administration and therefore the coordination of group tasks does result disadvantageous. Another adverse element of e-groups that can even become an obstacle is the propaganda for articles or services that constantly appears within the work area or its associated services, such as email, since it distracts students’ attention.

The main advantage of WebCT is the fact that it is a platform designed for distance learning. However, like the greater part of commercial platforms, it presents an architecture adapted to the model of teacher-student didactic relationships and to the tasks associated conventionally to each one, even though it has elements for work among students. This architecture is not sufficiently modifiable to adapt it to essential requirements implied by the dynamics of collaborative learning.

Other collaborative work experiences, for example computer-supported intentional learning environments (CSILE) (Scardamalia & Bereiter, 1994; Bielaczyc, 2000) show computing developments appropriate for each perspective and didactic conception. Some others, for example the Virtual Learning Environments, contemplate explicit didactic approaches during the process of software development (Paquette, 2002) and its later use in real didactic situations with a collaborative focus (Henri &Lundgren-Cayrol, 1998).

This pioneering study breaks new ground in the field of Web-device evaluation for CSCL. Although it is crucial to improve the instruments used, they allow us to specify some areas for characterizing adequate software for collaborative work.

Although the results obtained until now provide an answer to the question we proposed above, it is still necessary to get information from the students, as they were users of both devices. However, it is no longer possible to obtain the opinion of the students that used e-groups, since they have left the educational level in which the project is developed. The evaluation of the students that are current WebCT platform users is therefore pending.

The sample was made up of nine teachers involved in the Tactics project. Four of them are also researchers; this fact marks a difference in the sample profile. Although in general it could be said that participants have a good level of technological ability in the use of the Internet, in this case the researchers have managed to incorporate it more clearly into all their activities.

All sample participants except for one have Internet access from their homes. The researchers have used email for 10 years on average, while teachers have only used it for 2.6 years. Researchers have from 2 to 3 email accounts, while the majority of teachers have only one. Researchers receive and send over 10 messages a week, and teachers from 5 to 10. Researchers prefer to use Google as a search engine (more appropriate for academic searches), whereas teachers use Yahoo more (more appropriate for seeking commercial information). In general it can be seen that researchers use the Internet for a greater variety of activities and that they can transfer these abilities more easily to spaces other than academic.

2) General comparison

Table II shows a synthesis of answers given on the aspects evaluated in the second part of the questionnaire. It should be made clear that only six teachers that had the experience with both platforms answered this second part; of these, four are researchers from the project and two are participating teachers. Each aspect evaluated will be seen in the following section.

3) Functionality

Within the language of software engineering, functionality refers to aspects such as the reliability, efficiency and robustness of the system. In this part of the questionnaire, questions refer to how accessible devices are when working in diverse circumstances, how long they can be worked on without being blocked or without extensive waiting times, how easy is it to use other devices at the same time, etc.

If we take the sample total, the participants’ opinion on either one of the devices is not conclusive. In fact, 45% of the 162 answers in this section correspond to option (c) “both”, with 26% of the answers favouring WebCT and only 4% leaning towards e-groups.

Nevertheless, if we take the differentiated answers of teachers and researchers (Table III), preferences appear: the majority of teacher answers (57%) favour the WebCT option, with 0 answers in e-groups, while the majority of researcher answers (57%) do not show a definitive advantage in either one of the devices, and some even prefer e-groups (6%). It is interesting to note that 18% of the total answers choose the option “No opinion”, which is perhaps telling with regards to the fact that these issues related to pedagogical design are not stimulating reflection either in teachers or researchers.

4) Efficacy and usability

Software efficacy and usability refer to whether it favours (or is an obstacle for) specific collaborative work tasks such as communication, planning, coordination, tutoring, etc. In this part of the questionnaire, questions refer to the device’s speed, security and ease with which tasks are carried out by user groups.

In this case there is also not a conclusive opinion for the total sample of answers (156), since 42% of them correspond to “both”, with a greater percentage (33%) that has an inclination for WebCT, vs. only 3% that feel e-groups is better. The differentiated answers, on the other hand, again indicate preferences (Table IV): 77% of the teachers’ answers are grouped in option (b) WebCT, versus 57% of researchers’ answers who think “same in both”. It is also striking here that a fifth of the total answers are grouped in “No opinion”.

5) Effects on the individual

According to Ramage’s definitions (1999), this refers to significant psychological questions when evaluating the effect of the system on individual users; one of the questions asked, for example, is whether the user can situate his work in relation to the group, if he or she can develop certain tasks by him or herself, etc.

Of the 48 answers given in this section, 38% favour “WebCT” and an equal percentage leans toward “both”. However, it should be noted that 16% of the answers correspond to the option “No opinion”: once more, this suggests that software requirements that could foster each participant’s work and learning in a collaborative experience have not been defined. Again, the differentiated answers (Table V) give us a point of reflection since 75% of teachers’ answers are grouped in position (b) “WebCt”, versus 19% of researchers’ answers; answers for option (a) “e-groups” again are of 13% in the case of teachers and 0% in the case of researchers; in the case of option (c) “both”, the answers are of 25% of teachers versus 44% of researchers.

6) Effects on group work

This section refers to the effects that foster working in a group that can be attributed to the software. This aspect is different to that of usability in the sense that more emphasis is placed on inter-group relations propitiated by the software, rather than on specific tasks.

Of the 60 answers received, 27% favour Web CT, but 42% lean towards the option “both”. In this section there was no answer that favoured e-groups, but 10% correspond to “neither” and 21% to “no opinion”. The teachers’ answers (Table VI) favour WebCT (55%) versus 13% of researchers. It is interesting to note how in the option “both” these percentages are almost inverted: 15% of the teachers and 55% of the researchers.

7) Pedagogical effectiveness

Pedagogical effectiveness (Notess, 2001) of the device refers to the software’s capacity to promote effective online collaboration to reach educational goals.

Of the 36 answers received, 42% lean towards the option “both”, although 36% favour WebCT. Again, there was not a single answer in favour of e-groups, but 19% correspond to “no opinion”. The differentiated answers (Table VII) again favour WebCT with 67% of the teachers and 21% of the researchers.

Bielaczyc, K. (2000). Designing social infrastructure: The challenge of building computer-supported learning communities. Chestnut Hill, MA: Learning Communities Research and Technology Group.

Henri, F. & Lundgren-Cayrol, K. (1998). Apprentissage collaboratif et nouvelles technologies. Montreal: Centre de Recherche LICEF.

Ministère de l’Education Nationale, de la Recherche et de la Technologie (1999, September). Etude comparative technique et pédagogique des plates-formes pour la formation ouverte et à distance. Une étude de la Direction de la Technologie, sous-direction des technologies éducatives, des technologies de l'information et de la Communication (DT/SDTETIC). Paris: Author.

Moliner, M. (1997). Diccionario de usos del español (Vol. 1, 20th. edition). Madrid: Gredos.

Notess, M. (2001, August). Usability, user experience, and learner experience. E- Learn Magazine: Education and technology in perspective [In-Depth Tutorials Section]. Retrieved November 15, 2002 from:
http://php.indiana.edu/~mnotess/

Paquette, G. (2002). Modélisation des connaissances et des compétences. Un langage graphique pour concevoir et apprendre. Québec: Presses de l´Université du Québec.

Potts, M., Morse, E., Gutwin, C., & Greenberg, S. A. (1999). Comparison of usage evaluation and inspection methods for assessing groupware usability. Gaithersburg, MD: National Institute of Standards and Technology

Ramage M. (1999). The learning way: Evaluating co-operative systems. Doctoral thesis published online. Lancaster University, United Kingdom. Retrieved December 1st, 2002 from:
http://systems.open.ac.uk/objects/MagnusR/LearningWay.pdf

Scardamalia, M. & Bereiter, C. (1994). Computer support for knowledge-building communities. The Journal of the Learning Sciences, 3 (3), 265-283.

Scott, J. (1997, May 5). CSCW. Retrieved October 15, 2002 from:
http://farrer.riv.csu.edu.au/~jscott/Cscw_1.htm

Slavin, R. E. (1978). Using student team learning. Baltimore: The Johns Hopkins Team Learning Project.

Waldegg, G. (2002). El uso de las nuevas tecnologías para la enseñanza y el aprendizaje de las ciencias. Revista Electrónica de Investigación Educativa, 4 (1). Retrieved June 8, 2003 from:
http://redie.ens.uabc.mx/vol4no1/contenido-waldegg.html

2 A summarized version of this paper was sent to the VII National Congress for Educational Research (Guadalajara, Jalisco, October 2003).

3 The term “device” is translated from the word “dispositivo” in Spanish, as defined by the Moliner Dictionary for Spanish Use (1997): “a group of combined elements that are used to facilitate or do a job or to perform a special function”. We do not use the terms “site”, “page” or “portal”, since their characteristics do not coincide with the software we describe further ahead. One of the software programs analyzed is clearly an education at a distance “platform, but the other one cannot be called that.

4 Tactics is an acronym for “Collaborative Work and Learning with Information and Communication Technologies in the Sciences” (Trabajo y Aprendizaje Colaborativo con Tecnologías de Información y Comunicación en Ciencias) with meanings in French and Spanish, both the languages of the project.

5 The University of Montreal uses this platform since January 2000. The version it currently uses is the 3.6 Standard Edition.

6 http://www.coursenligne.umontreal.ca/