Figure 7: Phases of formative evaluation:
inter-institutional grid of EvNet project schedules and milestones, 1997 - 2002

Phases of Formative Evaluation
---»»»

Scriven (1991)

1. Alpha Testing (in-house critiques)

2. Hand-Holding Field Trials

3. Hands-off field tests (remote beta bests)

4. External Evaluation

5. Dissemination (summative evaluation)

Thiagarajan (1991)

1. Goal specification -> 2. Preparation -> 3. Data Collection -> 4. Data Analysis -> 5. Revision -> 6.Recylcing

Project

1997-8

1998-9

1999-2000

2000-2001

2001-2

THEME 1: EVALUATING STRATEGIES

(1) Evaluating Administrative Practices and Faculty Incentives (Conrath, McMaster)

1997-8: Construction of university and college sample, Questionnaires and Interview Guide; seek access permission.

1998-9:First phase of data collection from universities and colleges within Evnet.

1999-2000:2nd phase- collect data from national sample of universities and colleges; analyze data from first phase

2000-2001:Controlled comparison with selected case studies from private corporations, educational institutes, and Internet educational providers.

2001-2002:Select meetings with senior administrators in post-secondary educational institutions to recommend 'best practices' model

THEME 2: EVALUATING DESIGN ROLES: COMPARATIVE EVALUATION OF INSTRUCTOR APPROACHES TO ALTERNATIVE MODES OF EDUCATION AND TRAINING DELIVERY

2A) Diffusion of Computer Technologies into University Teaching: Professional Practices (Campbell, Mount Allison)

1997-8: Survey Instrument development and refinement

1998-9:First faculty computer use survey (Mount Allison); Identification of case study faculty; Data Analysis

1999-2000:Second faculty computer use survey (Concordia); Development of case studies; Data Analysis

2000-2001:Third faculty computer use survey (McMaster; Waterloo); Final case studies; Data Analysis

2001-2002:Dissemination: Report writing; meetings with, and policy recommendations to, CAUT, faculty associations, university administrations.

2b) Overcoming Obstacles to Teacher Adoption and Student Use of Technology: Computer-Supported Collaborative Learning (Abrami, Concordia)

1997-8: K-12: gather interview & questionnaire data about implementation obstacles from teachers (traditional methods plus use of the Small Schools Net), Kingston, Hamilton schools; develop on-line projects under 4 headings: resource library; school, class, & student project involvement (e.g., KIDLINKS); class-to-class projects and links (e.g., HTML pen pals); students in touch with experts.

1998-9:K-12 & Higher Ed: collect/develop instruments to measure teacher role and workload, student roles, task demands and resource requirements; comparative data collected at Waterloo, Mount Allison, McMaster.

1999-2000:Initiate data gathering on project implementation and course development process from previous years; revise projects/process as appropriate (formative evaluation) (includes a conferencing site where "reviews" are posted and discussions held).

2000-2001:expand project development by introducing collaborative courseware development tools via the Net

2001-2002:development of prescriptions and prototype projects, development methodologies and infrastructure support (best practice in an educational domain); dissemination of research findings & recommendations to teachers and faculty groups (CAUT).

2c) Evaluating Teacher Attitudes to Distance Telelearning on Learner Outcomes (Webster, Waterloo)

1997-8: Design of statistical and qualitative samples; gain access with instructors permission

1998-9:Collection and analysis of quantitative and qualitative data at Waterloo, Guelph and McMaster; conference and EvNet presentations

1999-2000:Collection and analysis of quantitative and qualitative data at Western, Toronto, and Queen's; conference and EvNet presentations

2000-2001:Extension of study to Mexican universities; conference and EvNet presentations

2001-2002:Publish findings in leading journals; present recommendations to CAUT and university staff and administrators

THEME 3: EVALUATING DELIVERY RESULTS

3a) Evaluating five modes of delivering distance education courseware (Cuneo, McMaster )

I: Skills for Distance Education

1997-8:Consultation with Research Team on design of study; Development; Literature Search of Electronic and Print- Based Resources; Draft of Print-based & Internet courses

1998-9:Development; Field testing of Print- Based & Internet Courses

1999-2000:Data Collection; Data analysis; Course revisions; testing at other sites (Athabasca; Guelph; Concordia)

2000-2001:Report on comparative delivery formats

2001-2002:Dissemination to Canadian Association of Distance Education

II. Ethics for Human Resource Practice

1997-8:Consultation with Research Team on design of study; Development; Literature Search of Electronic and Print- Based Resources; Draft of Print-based Internet courses

1998-9:Development; Field testing of Print- Based, Internet, On-site evening lecture, and On-site block courses

1999-2000: Data Collection and analysis; Course revisions Testing at Concordia.

2000-2001:Report on comparative delivery formats; external evaluation through the Canadian Association of University Continuing Education.

2001-2002:Summative Evaluation; Dissemination

III. Problem Solving: Concepts Skills and Strategies

1997-8:Consultation with Research Team on design of study; Development; Review of existing course content; Conversion to print-based correspondence & INTERNET formats

1998-9:Data collection & analysis; On-site, print-based correspondence and Internet delivery of courses

1999-2000:Testing at Athabasca; Concordia; Guelph..

2000-2001:External evaluation through the Canadian Association of University Continuing Education.

2001-2002:Summative Evaluation; Dissemination

IV. Introductory Financial Accounting

1997-8:Consultation with Research Team on design of study; Development; Review of existing course content; Conversion to print-based correspondence & INTERNET formats

1998-9:Data collection & analysis; On-site, print-based correspondence and Internet delivery of courses

1999-2000:Report writing; evaluation at other EvNet sites (Mount Allison University)

2000-2001:External evaluation through the Canadian Association of University Continuing Education.

2001-2002:Summative Evaluation; Dissemination

V. Addictive Behaviours

1997-8:Consultation with Research Team on design of study; Development Literature search and of course content in collaboration with ARF; Presentation of course in block delivery format

1998-9:Data collection Data analysis; Conversion of course to print- based correspondence and Internet. Delivery of course in tele-conference format

1999-2000:Data Collection; Presentation of course in print-based correspondence, Internet, and evening lecture formats

2000-2001:Data Analysis; External evaluation

2001-2002:Dissemination

3b) Evaluating Modular Curricula Design in Distance Education (Holt & Baggaley, Athabasca)

1997-8:development and refinement of research tools; assessment of paced and unpaced delivery to undergraduates and graduates

1998-9: assessment of paced delivery to SUNRISE consortium & Grande Prairie Regional College

1999-2000:assessment of paced delivery to college students; comparisons with Mount Allison, University of Toronto, and McMaster, and graduate research assistant network

2000-2001:assessment of partners' materials in Athabasca context; ongoing comparisons with other EvNet sites and research assistant network

2001-2002:Summative evaluation and dissemination

3c) Evaluating the effects of First Class Client and the Web on Problem-Based Learning Among Medical Lab Technologists (Harnish, McMaster)

1997-8:Development of the environment and course content for PBL. First course offering (Jan, 1998)

1998-9:Assessment of the first initiative, development of targeted material

1999-2000:Continuing assessment, expansion to include more potential employers and students

2000-2001:Continuing assessment, modifications of programs with respect to student and industrial perspective

2001-2002:Summative assessment of project; dissemination to health care sector.

(3d) Evaluating Computer-Assisted Literacy Learning (Proctor, Toronto)

1997-8:pairings of sites, selection of CAI material and methods; training of instructors, observers, & students in uses of chosen material and pedagogies; design of observational methods and kinds of measurement

1998-9:on-site observations and gathering of data

1999-2000:analysis of data, with input from other network researchers and stakeholder consultants (publishers)

2000-2001:revision of pedagogical methods and/or rewriting of some software, leading to a repetition formative evaluation at other sites within the network (Toronto and Kingston

2001-2002:summative evaluation; dissemination in form of 'best practices' writing and reading manual.

3e) Computational Support Mechanisms for Spatial Literacy In Education: Evaluating Computer-Assisted Spatial Literacy Learning Environments (Wright, Toronto)

1997-8: Establish survey instruments for pedagogy and virtual design studio observations; develop preliminary case study VDS; develop spatial media with City of Toronto, CACD, and synchronous/asynchronou s tools compiled and documented on the WWW project server.

1998-9:Revise initial survey used to study past VDSs based on survey results to support VDSs being evaluated; run and observe preliminary VDS.

1999-2000:Update data, documentation, and tools based on preliminary findings; update direct VDS observation and survey techniques based on their success/weakness in the baseline study; run a second VDS under the newly revised techniques (tools/data/strategies).

2000-2001:A second round of revision will occur and be tested in a final VDS

2001-2002:Disseminate research findings; train industry/practice partners in the tools and strategies found to be most effective in the studied VDSs.

THEME 4: EVALUATING COLLABORATIONS

4a) Student sub- cultures, groups, home study, and informal learning: face to face vs computer- supported collaborative learning (Campbell, Mount Allison)

1997-8:Instrument development and refinement; Qualitative protocol development; Baseline student computer use survey (Mount Allison; Concordia).

1998-9:Second annual student computer use survey (McMaster; Concordia; Athabasca); Qualitative participant observation (Concordia; Mount Allison).

1999-2000:Third annual student computer use survey (Athabasca; Concordia; McMaster); qualitative data analysis.

2000-2001:Fourth annual student computer use survey (Concordia; McMaster; Athabasca).

2001-2002:Dissemination to student groups; refereed publishing.

4b) Face to Face vs. Computer- Supported Collaborative Learning Among Students with Physical Disabilities (Catherine Fichten, Dawson College)

1997-8:Mail questionnaire to a random sample of coordinators of services to students with disabilities in Canadian community colleges and universities.

1998-9:Interviews with sample of 50 coordinators of services to students with disabilities concerning their experiences with distance education offered by their institutions

1999-2000:Evaluation of academic and social outcomes of in vivo vs virtual problem solving groups where some groups know about members' disability status while others do not.

2000-2001:Compare of the impact of virtual and in-vivo collaborative groups on "informal" educational outcomes of students with disabilities (e.g. friendship choices, attitude change in nondisabled students, self-esteem of students with disabilities).

2001-2002:Focus group meetings with college and university administrators re policy recommendations for computer support for students with disabilities.

4c) Workplace applications of computer-supported collaborative learning (cscl): communicating and cooperating (Richard Schmid, Concordia)

1997-8:completion of documentation of needs assessment for each initiative; design of application; design of assessment tools

1998-9:Development of application and tools (including performance support systems with networking capabilities using Lotus Notes or First Class Client); development of prototype; learner verification

1999-2000:Pilot testing of prototype and gathering of data (examining human, technical and environmental factors as they impact on individual, group and organizational performance; field testing and analysis of data

2000-2001:summative evaluation, including development and preliminary distribution of best practices

2001-2002:Repeat of field testing and analysis of data; articulation of conclusions; creation of "best practice" methodologies and dissemination of conclusions and implications.

4d) Labour/Learning and the Virtual Workspace (Lanfranco, York)

1997-8:Coding of logs of archived Labor-L messages

1998-9:Analysis of logs of archived Labor-L messages

1999-2000:Administration and analysis of virtual questionnaires; evaluation of CEIU regional staff network

2000-2001:Development of labour virtual network 'best practices' guide; CEIU regional staff guide.

2001-2002:Global Dissemination