D-1) PLANNING PROCESS AND CONSULTATIONS

a) Origins of EvNet: EvNet has its roots in the technology-based learning competition held in 1994-5 under the auspices of the Natural Sciences and Engineering Research Council of Canada and the Networks of Centres of Excellence. The McMaster University-based Research Institute for Studies in Education (RITE) was selected as one of four finalists in a field of twelve network applicants. Its Program Leader was Carl Cuneo, the same as EvNet's. Cuneo negotiated the merger of RITE with another finalist (LINCS) based at the University of Calgary and four non-finalists to form, along with Brian Gaines from Calgary, the Technology-Based Learning Network Canada (TBL.CA). It consisted of 178 researchers, 83 partners making cash and in-kind contributions of $41 million, and 39 stakeholderer organizations expressing an interest in its research, training and dissemination. Cuneo constructed its web site found at: http://www.humanities.mcmaster.ca/~misc2/tblca1.htm. Throughout this process, Cuneo gained valuable experience managing research networks, which he now brings to EvNet as program leader. TBL.CA's central theme of evaluation of technology-based learning and training in K-12, higher learning, workplaces, and culture was developed at McMaster University by Carl Cuneo (Professor of Sociology) and David Conrath (Dean of Business), both members of EvNet. We have added new evaluation expertise, such as Jon Baggaley (Athabasca University), Greg Brown (Ontario Solicitor General Office) and Lynette Gillis (Knowledge Connection Corporation). Some of EvNet's researchers and partners had been members of TBL.CA. Others are new members, some of whom are members of TeleLearning RN based at Simon Fraser University.

b) Real and Virtual Communications: The development of EvNet's Letter of Intent, and the present application, proceeded by means of phone, fax, electronic communications via e-mail and the world wide web, and by face-to-face communications, among 33 academic members at 14 universities and 4 community colleges and 35 collaborators. There are a total of 61 organizations participating in EvNet, divided among 30 public organizations, 21 private corporations, and 10 non-profit organizations. Between October, 1995, and June, 1996, our network members exchanged over 1,000 messages on our moderated national electronic distribution list. Shortly after our Letter of Intent was submitted to SSHRC, we constructed an interactive World Wide Web site containing our Letter of Intent, bibliography, partner statements, researchers' cvs, poll results on an internal communications plan, and a fill-in form for the collective self-rating of our own proposed research projects and the development of our final application, as well as numerous hypertext pointers to our researchers and partners' web sites and other relevant national and international sites. It can be viewed at:http://socserv2.mcmaster.ca/srnet/evtoc.htm. We restricted access to network members, other selected networks, and potentially new researchers and partners. We are now willing to open it up by providing the userid ('beammeup') and password ('takeitdown'). This electronic document has been a tool in the development of the present application; it has saved us considerable monies in paper reproductions, phone, fax, and travel.

c) Development Grants: We underwrote some of the costs of developing the present application with the $5,000 SSHRC development grant, and a contribution of $5,000 from York University's Distributed Knowledge Project which was used to hire part-time clerical staff.

d) National Conference: We organized a national conference of 18 EvNet researchers and guests on May 17, 1996, at McMaster University. It was devoted to the integration of our researchers from sociology, education, educational technology, distance education, business, computer science, biology, health science, languages, writing, economics, instructional development, and management science, and our partners from the public, private, and non-profit sectors, around a central research, training, communications, and dissemination plan. A special feature of this workshop was the participation of graduate students as potential Network research assistants from Concordia, York, and McMaster Universities. It was out of this meeting that we began to develop plans for a parallel EvNet graduate student research assistants electronic network (RASNET), and a research assistants training module produced by the graduate students themselves in cooperation with the network's academic members.

e) Researcher/Partner Meetings: Meetings between researchers and partners occured both at the national level with Cuneo (e.g., UBC Press, Irwin Publishing, Knowledge Connection Corporation, etc), and with regional groups of researchers across the country (e.g. Mount Allison/NBTEL; Concordia/Institute of Canadian Bankers; Queen's University/Kingston Immigrant Council; University of Toronto/Skills for Change; McMaster University/Michener Institute, Softarc, Macdonald Secondary School, and Canadian Employment and Immigation Union/Public Service Alliance of Canada; Athabasca University/Sunrise Foundation). These meetings were devoted to integrating researchers and partners at the level of both research project and network. Regional meetings of researchers, and with other networks, also took place in Montreal, Ottawa, Calgary, Edmonton, Guelph and Hamilton, and at the 1996 Learned Societies Congress at Brock University.

D-2) OBJECTIVES

A paradigm shift is occurring in teaching, learning and training [Simsek and Louis, 1994]. Funding cutbacks, global competition, market restructuring, and flexible job skills are propelling education toward multiple sources of revenue generation, 'any time, any place', situated, lifelong, and/or distant learning, the merger of education and training, shifts from accreditation to competence, and the decoupling of teaching and learning. The medium is technology. Notwithstanding skeptics, it promises to revolutionize teaching, learning and training. Educational institutions, with verbal or financial backing from government and business, are installing technological infrastructure, such as broad-band fibre optic cabling, networking systems, student computer labs, classrooms electronically-networked within and between sites, video conferencing links with remote sites, electronic data bases, world wide web servers, and electronic mail for instructors, staff and students. Educational software and courseware are being developed at in-house multimedia production centres or purchased 'off the shelf' [Oblinger, 1994]. However, Geoghegan [1994a] reports that, in the United States, despite expenditures of $20 billion on instructional technology in higher education between 1970 and 1994, no more than five per cent of faculty use such technologies as anything more than high-tech substitutes for the blackboard and overhead projector; he estimates a 15 per cent saturation level in the diffusion of such technologies among higher education faculty. There is no reason to believe that the situation in Canada is any different [Proulx, 1995; Campbell and Proulx, 1996; Miall, 1995].

Although there are many reasons for this state of affairs [Massy and Zemsky, 1995; Shields, 1995], a primary one is lack of rigorous, or even any, formative evaluation of the technical infrastructure, software, and courseware before, during, and after their attempted introduction, with consequent harm to 'best teaching practices' and effective learning. Many of us who teach in hi-tech networked classrooms have had to struggle with constant hardware failures (connectivity, audio, video) and to find ways to alter the equipment to enhance 'best teaching practices'; this might not have been necessary had proper formative evaluations been conducted. Scriven [1991: 58-9] complains that, in contrast to consumer evaluations of automobiles, many computer companies have little interest in serious formative evaluation and put products on the market that are "...demonstrably, extensively, avoidably, and seriously flawed". Many evaluations are a sham. In published reviews, it is rare to find "below good" ratings of educational software [Dolan, 1987]. Much of this software is not even evaluated before it reaches the teacher and student; hence their widespread dissatisfaction with educational software and courseware.

At this stage in the development of information technologies, we have less need for more hardware than for systematic and rigorous evaluation of what we have -- especially of developing software and courseware -- to protect and promote best teaching practices [Barker and King, 1993].

EvNet's general objective is to rigorously evaluate the role of technology in the delivery of education and training. We will use the results of such evaluation to produce new learnware products and training modules, and develop better teaching and training practices. The benefits to Canadians will be a higher quality, more effective, more efficient, and more practical education and training system attuned to the rapid changes in skill requirements in the job market.

This general objective has two primary components:

1) To comparatively evaluate alternate means education and training delivery in terms of the outcomes of learning effectiveness and cost efficiency (on which we have established close relations with the Canadian Network for the Measurement of Education and Training Outcomes [C-METO] in Guelph). These means are traditional lecture/seminar classroom teaching; print-based distance correspondence; weekend block in regional sites; synchronous internet (such as on-line chat rooms; interactive educational TV; video-conferencing); and, asynchronous internet (e.g., e-mail; usenet discussion groups; listservs; world wide web; on-line databases).

2) To evaluate the mediating role of collaboration on the effect of means of delivery on learning. Collaboration among learners and with instructors has been proven to be an effective mechanism for learning. Traditionally most collaboration has been face-to-face in the classsroom or on-campus, and place and time specific. Can collaboration be effectively conducted at a distance through computer-mediated communications (CMC) where learners do not know one another and the instructor on a face-to-face basis, where informal interactions and visual cues vital for deep learning may be absent or artificial?

These general objectives may be broken down into the following eight specific objectives.

1. Do administrative practices influence the effective implementation of technology in learning? We will evaluate 'best' and 'poor' practices by comparing higher learning institutions in terms of their administrative organization of learning technologies. We will draw lessons for post-secondary institutions from the organization of workplace training practices.
2. What incentives are effective for persuading faculty, teachers, instructors, and trainers to adopt new technologies in teaching & training? We will evaluate institutional differences in faculty reward structures (educational leaves; training programs; merit increases; extra course credits; etc). What incentives are necessary to counteract the negative effect of computer-mediated communications on faculty workload?
3. We will evaluate whether technology always or normally enhances the effectiveness of learning and training. What are the conditions under which computer-mediated communications impair learning effectiveness? We need to separate such conditions from 'best practices'.
4. Can technology enhance the efficiency (reduce the costs) of learning and training? What is the balance between efficiency (a financial issue) and effectiveness (a learning issue)? Is the intial high investment in technology justified by returns in greater learning effectiveness? Or does the long arm of the technological medium reduce the quality of learning by enablling fewer instructors to reach a greater number of students? The most expensive and complex technologies do not always result in the most effective learning. We will evaluate different mixes of effectiveness/efficiency in instructional technology.
5. Can collaboration be effectively implemented at a distance using technology in learning and training? Does effective collaborative learning require face-to-face, immediate, and personal interaction among learners, and between learners and instructors? We will evaluate the mediating role of collaboration between different means of delivery and learning effectiveness in a relatively simple model: means of delivery --> collaboration --> learning effectiveness.
6. Can situated learning be effectively implemented over a distance using CMC? It is relatively easy for instructors to integrate the prior learning experiences of learners in small, face-to-face classroom settings. Can this be done at a distance in which the main or only means of communications is the computer? We will evaluate such models as: modes of delivery --> situated learning --> learning effectiveness.
7. Can problem-based learning be effectively implemented over a distance using CMC? Can learners exploit the opportunities of computer-mediated communications to collectively work on problems while using the internet as a resource tool to find information relevant to the solution of problems? Means of delivery --> problem-based learning --> learning effectiveness.

D-3) APPROACHES

EvNet is based on four related approaches: (1) alternative means of education and training delivery; (2) computer-supported collaborative learning (CSCL); (3) formative and summative evaluation; and, (4) learning organizations.

1) Alternative Means of Education and Training Delivery:

We will evaluate alternative means of delivering education and training to determine which are most effective (learning) and efficient (costs). Traditional means of delivery, such as the classroom, in-house training, and weekend blocks in regional centres (1-3 in Figure 1), are time- and place-specific.

They have the advantage of allowing for potentially rich face-to-face interactions between instructor and students, and among students themselves. Such collaborative learning, made possible but not necessary by this mean of delivery, has been shown to be an effective learning technique (Allen and Thompson, 1995; Mevarech, 1993; Roschelle, 1992; Mavarech et al, 1991; King, 1989; Webb, 1982; Koschmann, 1994; Repman, 1993; Gokhale, 1995; Wan and Johnson, 1994; Koschmann et al, 1994; Feldmann et al, 1991). The classroom is also an ideal setting to integrate student's prior learning experiences into the curriculum (situated learning), and to develop problem-based courses, a pedagogical technique in which McMaster's Faculty of Health Sciences has an international reputation (e.g., Woods, 1994). However, time and place dependency are also disadvantages in classroom and block teaching and in-house training. As more job-holders return to school while continuing to work part- or full-time, time and place dependency restricts their access to skills upgrading and life-long education. Place restrictions are maintained but time restrictions are relaxed among students who work individually or in groups on projects in libraries, archives, computer labs (4-6 in Figure 1), though as such facilities come on-line, place restrictions are partly removed. New learning technologies via computer- mediated communications (CMC), especially at a great distance, provide untapped opportunities to overcome such place and time restrictions. Place restrictions are relaxed, but time restrictions remian in place, in synchronous communications where several users must be logged on at the same time ( Internet chat rooms), or must be simultaneously located at distributed video-conferencing or interactive TV sites (7-9 in Figure 1). Both time and place restrictions are completely relaxed in asynchronous communications (e-mail; web; listservs; newsgroups) where only one user must be logged on at any given time; other users can pick up their messages later. But what effects do such technologies have on learning effectiveness and efficencies? CMC means of delivery may be more expensive than traditional delivery, at least initially, and it is still an open question whether they deliver education and training more effectively. This is a central area of exploration in the EvNet projects outllined below. We will evaluate alternative means of delivery from the point of view of both efficiency and effectiveness, especially in the area of collaborative learning.

2) Computer-Supported Collaborative Learning (CSCL):

The rise of distance educaton has brought with it attendant problems in achieving essential educational outcomes. The high rate of student dropout has been repeatedly cited (e.g., Kember 1989, 1990; Sweet 1986) as a negative consequence of offering courses at a distance. In addition, despite remarkable efforts to design effective learning materials, learning at a distance is seldom superior to traditional instruction and may fail to promote higher level achievement and the development of complex learning skills. Differences between students who populate distance education and traditional courses may partly explain the high dropout rate and other academic disadvantages of distance education. However, other differences are also evident and important to consider; features which relate to the instructional materials and, especially, the technology used to deliver them.

Social and intellectual isolation are two course-related factors which may contribute to weaknesses in distance education. Distance learners appear to experience fewer and less essential opportunities to interact with instructors and other students to discuss course content, assignments, learning strategies, and personal concerns about learning. Distance education students are physically separated from the social learning environment. This may have effects on their perceptions of psychological isolation and detachment which affect learning. Isolation may amplify student perceptions of external control--the belief that success at learning and course achievement are a function of factors beyond individual control (Kember 1990). Current distance education practices may encourage students to surface process material rather than attempt to develop a deep understanding of course content. Distance education courses may also encourage performance orientations to learning, where students use minimal effort to maximize evaluation results, and discourage mastery orientations to learning, where students use maximal effort to accomplish personal learning goals. Without interaction with peers, learners at a distance have few, if any opportunities to experience complex problem-solving in a collaborative environment. Thus, they may fail to develop and refine those cognitive and interpersonal skills increasingly necessary for business and professional careers.

Nevertheless, computer-supported collaboration and interactions provide learners separated by time and/or space a forum for learning and work that may reduce social and intellectual isolation with a consequential decrease in the distance education drop-out rate.

Abrami et al. (1995), researchers in EvNet, have described a multitude of instructional techniques and classroom structures to enhance positive interdependence among students and individual accountability which emphasize: WHY students work together or task purpose (outcome structures), HOW students work together or solution purpose (means structures), and with WHOM students work together or social purpose (interpersonal structures). While these techniques are becoming widely used in traditional classrooms, they have been less frequently used in virtual classrooms where students may be separated by both space and time. We will evaluate whether unique adaptations will allow the application of such structures to virtual classrooms.

Building on the logic of Figure 1, alternative means of delivery create the necessary, but not sufficient, conditions for variations in the quantity and quality of interactions. The lowest quality and quantity of interactions are in the place-specific libraries, archives, student labs, and through traditional correspondence by mailed manuals and audio tapes (4-7 in Figure 2). Such means of delivery were not created to encourage collaborative learning, though a number of studies have shown how this is possible in computer labs (e.g., Lee, 1993). The quality of collaborative learning may be highest in place- and time-specific classrooms, regional weekend block courses, and in-house training facilities because of their great physical potential for face-to-face interaction among learners and instructors (1-3, Figure 2).

However, interactions often flow from instructor to student (Schank and Jona, 1991: 8-10) rather than student to student, though informal learning outside of classrooms is an important ingredient of traditional residential campus settings (Kuh, 1995). We investigate this in Project 4A below. Such informal learning is recreated in both synchronous and asynchornous communications on the Internet, and in video-conferencing and interactive distance TV, along with more formal flows of communication (8-11, Figure 2). The Internet is a vast, untapped resource for facilitating at least a high quantity of interactions among users and instructors (which, ironically, leads to heavy faculty workload, on which we have proposed four research projects). We have inserted a question mark in the upper-right cell of Figure 2 because there is insufficient information about which means of delivery provide the optimal conditions for collaborative learning in both its quantitative and qualitative dimensions. Our Network research will evaluate which means of delivery provide for.such optimal conditions.

3) Formative and Summative Evaluation:

The pursuit of best teaching, learning, and training practices must tie the purpose of action-oriented research to an applied outcome. Evaluation methodology is one of the best vehicles for doing so (e.g., Tyler, Gagne and Scriven, 1967; Scriven, 1993). In the relationship between researcher and partner, the value or goal of a program, product, courseware, software, or manual must be clearly specified at the outset (Winne, 1993). Two such values or outcomes are "learning effectiveness" and "efficiency". The tangible product is then 'evaluated' by its approximation to the purpose or objective of the product as stated by the client or partner. The user, learner, and trainee, and the partner organizations in which they are located, take centre stage in establishing the benchmarks for evaluation. In other words, evaluation of the means of education and training delivery is 'learner-driven'. We are not conducting 'ivory tower', pure or abstract research. Nor are we pursuing 'knowledge for knowledge's sake'. We are seeking knowledge for the applied ends of improving the quality of peoples' lives rooted in income, status and inner satisfactions derived from jobs. As the quickening pace of global economic and technological change shortens the length of careers, the quality of peoples' lives depends increasingly on life-long learning and repeated upgrading of skills in training progams.

The distinction and relation between formative and summative evaluation, terms coined by Michael Scriven (1967), have received broad support in the large and multi-disciplinary evaluation literature (Chambers, 1994; Levin, 1986; Dempsey, 1989; Fuchs et al, 1993; Dagley and Orso, 1991; Beevers et al, 1989; Leahy, 1990; Reichardt, 1994). Formative evaluation is conducted while a program, courseware, or product is under development. Its purpose is to use data gathered during development to detect errors and weaknesses, and to apply it in order to improve the product. Summative evaluation is conducted after a product has been developed and is ready for the market. In a restaurant, the chef engages in formative evaluation as s/he repeatedly tastes the soup simmering on the stove; the customer in the restarant engages in summative evaluation as s/he eats the soup in a final act of consumption (Scriven, 1991).

4) Learning Organizations:

House (1990: 26) recommends that evaluation take into account organizational contexts. We believe that organizational structures and administrative practices have profound influences on modes of education and training delivery and ultimately on the outcome of learning effectiveness. We view post-secondary institutions and workplaces as learning organizations with the following characteristics (Senge, 1990). (1) Systems Thinking: the whole of the organization is greater than the sum of its parts;(2) Personal Mastery: individuals develop their skills on a lifelong basis by connecting their personal learning with organizational learning; (3) Mental Models: individuals attempt to become conscious of deeply held assumptions that block higher-level learning and individual development;(4) Building Shared Vision: individuals at all levels of an organization become commited to a democratically created shared vision of goals and means; and, (5) Collaborative Learning or Teamwork: individuals learn more by working cooperatively than in isolation.We view means of education and training delivery within the context of physical and virtual learning organizations. Such a context is the integration of physically stationary learning organizations (universities, colleges, schools, corporations, trade unions, community organizations), and virtual workspaces (Internet and electronic networks) connecting physical organizations and their members in a maze of computer-mediated communications (see the education/training organizational linkage in Figure 6, Appendix A-3). We will evaluate this integrated real/virtual organizational context of the means of delivery outlined in Figures 1 and 2. Only in this way can we properly assess the effectiveness of lifelong and informal learning and training.

D-4) FOUR THEMES

Using learning organization theory, we have arranged our research activities under four integrated themes (see Figure 3). First we will evaluate the extent to which Senge's systems thinking informs how educational institutions plan the introduction and implementation of means of education and training delivery (Theme One: Evaluating Administrative Practices). Second, in a learner-driven model of evaluating technologies in terms of the effectiveness and efficiency of outcomes, we will assess how the designer (instructor) and their their courseware products meet the needs of the user, student, trainee, or learner along the dimensions of personal mastery, mental models, and development of a shared vision within the learning organization. To do this, research activities are organized under Theme 2: Evaluating Design Roles, and Theme 3: Evaluating Delivery Results. Finally, the role of teamwork in learning organizatons must be assessed. Theme 4: Evaluating Collaboration will be an assessment of the extent to which adminstrators as policy-makers, teachers as designers, and students and trainees as consumers of the delivery results, come together in a joint endeavor of collaborative learning in a physical/virtual workspace/learnspace in order to enhance lifelong learning and skills. As depicted in Figure 3, Theme 4 (Evaluating Collaboration) frames all activities by administrator, teacher, and learner in the learning organization in Themes 1-3.

Figure 3: EvNet's Four Evaluation Themes

	1. Evaluating Administrative Practices
	2. Evaluating Design Roles
	3. Evaluating Delivery Results

D-5) COMPARATIVE EVALUATION ACROSS LEARNING ORGANIZATIONS

We will evaluate the means of delivery in Figures 1 and 2 across a wide variety of workplace, post-secondary, and K-12 organizations, both physical and virtual, in terms of our four themes. Figure 3 is reproduced in each of the institutional boxes of Figure 4 (see Appendix A-1). The first three themes correspond to the administrative, faculty/teacher, and output (courseware) levels of an organization. We will assess their hierarchical relationships, symbolized by the vertical arrows. As in Figure 3, they are framed or contextualized by Theme 4 on collaboration. In our research projects, collaboration is an underlying theme in the systems approach of administrative practices (getting administrators, technical staff, and faculty to collaborate on implementing learning technologies guided by a shared vision of the learning organizations), and of teachers's and trainer's roles in designing courseware for the classroom, workplace, and Internet. The horizontal arrows in Figure 4 (Appendix A-1) symbolize the comparative evaluations we will conduct across workplace and educational organizations Good research design involves variability in the cases under study. Two such dimensions of variability affecting learning outcomes are the technological infrastructure of the delivery mode, and the social/spatial living/learning arrangements of students, users, and trainees affecting opportunities for working, learning, and training collaboratively together. At one end of a continuum are educational and workplace organizations with a high proportion of residential students or on-site workers wired internally by broad band fibre optic cabling (see the centre of Figure 4, Appendix A-1). Such organizations have the opportunity of face-to-face collaborative learning in the classroom and collaborative training in the workplace, besides remote collaborative learning and training on the Internet. One of the best Canadian examples is Mount Allison University. For several years, it has been Macleans Magazine's (1996) top-rated small undergraduate Eastern Canadian school with a high proportion of residential on-campus students, and the most advanced broadband fibre- optic wiring (connecting all residence rooms, offices, and classrooms) of any university in Canada. Companies and government agencies that heavily network their employees internally with an INTRANET configuration also fit this end of the continuum. At the other extreme are learning organizations with no residential, on-campus students, or on-site workers. Their only link to the organization is via surface mail or computer-mediated communications. Students and workers in such organizatons can engage only in remote collaborative learning, training, and working via computer-mediated communications. Among educational institutions, the best example is Athabasca University, a unique western Canadian distance-education institution, with no commuter or on-campus residential students. It has only recently decided to move to an electronic offering of courses and degrees on the World Wide Web. Among workplace organizations, the best examples are those that have moved almost completely to telecommuting and telework. Arranged between these two extremes are most other organizations. The ones we will use for comparative purposes in EvNet research as a contrast to Mount Allison and Athabasca are McMaster, Queen's, Toronto, York, and Concordia Universities. They are much larger central Canadian institutions that are only now starting to convert to the Mount Allison type of infrastructure, and with a mix of residential on-campus and commuter students. Perhaps the one exception is Concordia which has no on-campus residential units. All of its students are commuter (primarily the Montreal area) who learn in classrooms, libraries, computer labs, and on the Internet. It thus combines some of the characteristics of Mount Allison and Athabasca.

D-6) NETWORK RESEARCH PLAN

There are many guides on the methodological procedures and pitfalls in formative and summative evaluation (Weston et al, 1995; Volman and Diana, 1993; Barker and King, 1993; Posavac, 1994; Micceri et al, 1989; Burkhardt, 1992; Pearlstein, 1988; Jeffries and Desurvire, 1992; Paterno, 1994; Peled et al, 1992; Webster and Finney, 1990; Salomon1991; McDougall and Squires, 1995; Hix et al, 1994). Scriven [1991: 21] outlines four phases in formative evaluation:

a) alpha testing or in-house critiques by colleagues or employees not in the development group;

b) hot-house, focus groups, or field trials with hand-holding (the development team supervises off-site tests);

c) hands-off field trials or beta tests at "remote sites by supposedly typical users working on their own in their usual environment"; and,

d) the 'review preview' phase: "full-scale commissioned evaluations by external expert reviewers... in which the reviewer may and should run systematic experiments with end-users" [also Dick, 1980; Dempsey, 1989].

A model similar to Scriven's, but more detailed, was proposed by Sivasailam Thiagaragan (1991). He expands Scriven's four phases into six, as shown in Column 4 of Figure 5 (Appendix A-2): (1) goal specification; (2) Preparation of data and sample; (3) Data Collection; (4) Quantitative and qualitative data analysis; (5) Revision of the product on the basis of the data collected and analysed; and, (6) Reclycling in a retesting of the product, and a final summative evaluation before the user adopts the product. We are adopting both Scriven's and Thiagaragan's methodological apporaches of sequenced phases of evaluation in our network research. Evaluation methodology is well suited to the multiple site, teamwork, multidisciplinary, and multisector requirements of network reserarch. A network is perhaps the best enviornment to conduct all four of Scriven's phases and Thiagaragan's six steps. We propose that Scriven's alpha testing and hand-holding field trials, especially of the courseware in Theme 3, be done by researchers who develop them, in their own sites, and in the first two years of SSHRC funding. Scriven's hands-off field trials and the final 'review preview' phase will be conducted at the other institutions in EvNet, or independently by partner organizations, primarily in years three and four, though the specific schedule and milestones will be determined on a case-by-case basis. This will avoid the conflict of interest involved in many past evaluations. Thiagaragan's six evaluation steps are more suitable for the research to be conducted under Themes 1, 2, and 4 since product development and application come at the end of the research process. Year One will be devoted to his first two steps - goal specification and data preparation. Thiagaragan's third ste;p (data collection) will generally occur in Year Two. Step Four, Data Analsis, will be in Year Three, and Steps 5 and 6 (revision and reclycling) will be in Years Four and Five. These last two hyears will genrally be taken up with summative evaluation and dissemination.

Go to EvNet Research Projects