Evaluation of WEBCAL

Abstract:

Over the past decade, as part of a movement to redesign and reform the teaching of Calculus, the use of information technology has become an important part of learning Calculus for many students. At different institutions a variety of technological aids have been incorporated into course designs, in different ways, and for different pedagogical ends. We have developed a web based course, WEBCAL, at Vanier. This course is designed to transfer the locus of control of both pace and sequencing of material from the teacher towards the student. Further, interactivity in the form of live or transformable mathematics is incorporated through the use of MathView, an interactive CAS (Computer Algebra System) from Waterloo Maple for which there is a free browser plugin. Our main goal in designing WEBCAL has been to provide learners with both the means and motivation to become cognitively engaged. The objective of this proposed research program is to evaluate whether this web based course design meets the standards of good and bad practices in the use of technology, and its own goals. Furthermore, we will ask students in the class to identify those characteristics of WEBCAL that they view as "good practice" and "bad practice", and their perception of the impact of those practices on their understanding of Calculus.

Preamble:

We are three researchers in education, original participants in the EvNet proposal. In the original proposal, because our research focussed on a learning strategy using Concept Maps, a visual representation of information, our efforts were married to those of Robert Wright in Theme 3e: Computational Support Mechanisms for Spatial Literacy in Education: Evaluating Computer Assisted Spatial Learning Environments. Unfortunately, it soon became apparent that his research was quite specific to the area of Landscape Architecture and so no funding from EvNet has found its way to our work. The current proposal covers at least two themes from the original proposal, 3b Evaluating Modular Curricular Design in Distance Education (currently missing in EvNet due to the withdrawal of the team from Athabasca) and 2b Overcoming Obstacles to Teacher Adoption and Student Use of Technology: Computer Supported Collaborative Learning. Thus, we see our project as both filling a gap, and constructing links within EvNet.

As outlined below, since EvNet began, with the assistance of three government grants totalling $110,000 ($108,000 for hardware, $12,000 for student wages), as well as $5000 of in kind assistance from private companies, we have developed a modular web based Calculus course that contains a high level of productive interactivity. This term we are beta testing this course. We hope to offer the expertise gained in this development and implementation to other members of EvNet, at EvNet meetings and through Learn Link. Further, this proposal will help us gain expertise in a novel methodology well suited to evaluating such work and we would anticipate an ongoing dialogue with other EvNet members interested in this type of project.

Objectives:

The main goal of this research program is to evaluate WEBCAL, an interactive course in Calculus whose principal mode of delivery is the web. More specifically, we seek to:

• evaluate whether WEBCAL meets the standards of good and bad practices in the use of technology;
• evaluate whether WEBCAL meets its objectives;
• describe the impact (as perceived by students) on conceptual understanding (as evaluated by researchers) of various WEBCAL instructional design components, to help clarify which are examples of "good" and "bad" practices and to help understand student interpretations of this instructional setting;
• inform mathematics educators about this implementation of technology in Calculus, describing how each component is perceived to impact upon students' conceptual understanding, and thus help guide their own use of technology in the classroom;
• to provide information concerning the EvNet list of good and bad practices, as applied to the "hard" sciences;
• contribute to the EvNet Evaluation Toolkit by adding empirical evidence to the debate on the shifting paradigm in education.




Literature:

While it is a challenge to introduce any pedagogical innovation to teachers who are comfortable with their current teaching practices, it is even more daunting a task to convince them of the merits of using new pedagogical approaches that make use of computer technologies. Most science teachers require extensive training to use the computer in the classroom rather than chalk and blackboard, for not only must they learn computer skills but they must redefine their role as they move from being "dispenser of wisdom" to "coach". We did a survey of Canadian universities and colleges and found that at most institutions almost all courses are still lecture based. For teachers to drastically change their mode of operation there must be strong incentives for the change. From the point of view of value expectancy theory, teachers must see that using computers in the classroom will truly enhance student learning and motivation (value), and they must both see how to carry out such change and believe that they can do it (expectancy).

Success in mathematics is the gateway to many careers in the sciences and increasingly in other fields such as economics and commerce, but failure rates in mathematics courses are high (Ferrini-Mundy & Lauten, 1994). In addition, for over fifteen years there has been general recognition of a crisis in mathematics education in that too many successful graduates have been unable to use their "academic knowledge" once out in the "real world" (American Association for the Advancement of Science, 1982; 1984). That is, the "real world" requires an ability to transfer knowledge, which in turn requires a high level of conceptual understanding that schools don't seem to have provided.

In response to these problems there has been a multiplicity of "reform" efforts directed towards changing both the pedagogy and curriculum of Calculus (Hodgson, 1987). In a standard Calculus course the emphasis has been mainly on computational skills and the need to learn the many algorithms used to compute limits, derivatives and integrals. Reform efforts place their emphasis on the "Rule of Four", that is, the need for students to develop skill at moving flexibly between a verbal, graphical, numerical and algebraic perspective. It is argued that this balance of the four perspectives offers more entry points to understanding for students with a wide variety of learning styles, and reflects an approach used by expert mathematicians in solving problems. Hence, there is the expectation of not only higher rates of student success, but also development of a deeper more expert-like understanding of the central concepts of Calculus (Ferrini-Mundy & Lauten, 1994).

Many reform Calculus courses use CAS to lower the barrier requirement that a high level of symbolic manipulation skill represents. Further, CAS is used to speed the creation of graphs and numerical calculation so that multiple instances are easily arrived at, thus shifting student focus from mechanical creation aspects (the "trees") to meaning or interpretation (the "forest"). This also ties in with Seymour Papert's extensions of Piaget's ideas on human development (Papert, 1980). That is, Piaget proposed a sequence of cognitive developmental levels, tied to specific age ranges. Papert, using computer based "micro-worlds", posited that the order in the sequence, and the age ranges at which they are achieved could both be manipulated. In effect, achievement of certain cognitive abilities depends upon exposure to sufficiently many appropriate situations, and the sequence and timing of development is dictated by environment. Given an environment rich in the appropriate circumstances, abilities can occur earlier or out of sequence than Piaget's "normal". In Calculus, instead of having students labouriously generate graphs by hand, and hence experience only a few, CAS software can be used to generate hundreds, providing that rich environment needed to promote conceptual understanding of certain ideas. Thus, it is not surprising that reform Calculus courses have been shown to have a positive impact on students' attitude towards learning mathematics and on students' failure rates (Ferrini-Mundy & Lauten, 1994; Barber & Narayan, 1994; Cooley, 1997).

There is another shift taking place in education - life long learning. Learning is no longer the "job" of youth, but of everyone. Workers may need to change their career orientation several times during their working life and as such will need to be trained in a new profession at different ages. "Night classes" have been a solution to such re-education problems for several decades or more, and distance education (DE) has been gaining in popularity. However, amongst educators there is often a feeling that the standard of education in night classes or DE classes are lower, and that the achievements of graduates of such courses do not match the achievement of day students. One of the problems is that the current paradigm for day school was transferred to the night education environment, disregarding the differing characteristics of the two student populations. Mature learners, who have many time constraints, are often unable to fit into night or day class schedules, but they can take advantage of DE models that transfer to them more control over the pace and sequencing of the material to be learned. However, DE created another problem, which resulted in high drop-out rates, namely the isolation of the learner. Technology is solving this problem. This new paradigm for education, developed in DE, is making inroads in day school with as much as sixty percent of enrollment in some DE courses consisting of full time day students (Johnson, 1998). This would seem to indicate quite strongly the need amongst daytime students for courses in which they can exert more control over pace and sequencing. Neff (1998) indicates that use of this technological DE approach in daytime classes greatly increases active participation and correspondingly diminishes failure rates.

Guskin (1994, 1994b, 1996) notes that educational institutions are in a time of rapidly diminishing resources and rising costs, and points to a radical restructuring as the only effective solution. Within that restructuring he sees a new role for educators, as coach or mentor, instead of the current model of knowledge provider. The new knowledge provider will be the computer.

We have shared in all of the above visions, and with the aid of the Quebec and Canadian governments (New Technology In the Classroom (NTIC), 1997, $24,000 for computer hardware for development; Industry Canada Notemaker's Project, to 12/31/98, $12,000 for student assistant wages and $5000 of in-kind assistance from private companies; NTIC, 1998, $71,500 for computer hardware for an electronic classroom for science students), using a learning model borrowed from DE, based on principles of the Reform Calculus movement, incorporating the training in and use of CAS software, and following a constructivist perspective, we have built a web based Calculus course that we call WEBCAL. (For a demonstration of the course: visit Waterloo Maple's site http://www.cybermath.com and download and install the appropriate MathView plugin for your browser; then, see http://sun4.vaniercollege.qc.ca:8900/Webcal/Index.html , with only Module 1 - Functions available for demonstration. The demonstration site will soon be mirrored on Industry Canada's server.)




Ongoing Research:

We are in the final stages of a three year project, Changes in Student Knowledge Structures in Science, (Quebec government, Programme d'Aide a la Recherche sur l'Enseignement et l'Apprentissage (PAREA), 1996-1999, $30,000 total over three years). In that project we have discovered that the notions of acceleration and velocity are confused for each other in student discourse in Physics, revealing that these concepts are not distinct in student understanding. Further investigation points to the underlying cause being a misunderstanding of the relationship between f (x), f (x) and f (x) in Calculus I. Our current proposal to evaluate WEBCAL will allow us to focus our attention on this particular topic as representative of student conceptual understanding in Calculus. Thus, the new proposal will supply a missing piece to the puzzle uncovered in our current work. Because the proposed project is to evaluate WEBCAL, this application would be incomplete without a detailed description of WEBCAL.

WEBCAL:

Instructional Paradigm: As mentioned above, the philosophical approach to teaching Calculus that WEBCAL is based on is derived from the Calculus reform movement. As such, emphasis is placed on the need for students to develop the skill of moving flexibly between verbal, graphical, numerical and algebraic perspectives. Further, it is based on the constructivist understanding that conceptual change occurs when appropriate cognitive and motivational conditions are met by both the instructional setting and students' characteristics. The WEBCAL course is also based on a self-paced independent learning model developed for distance education instruction strategies, and as such uses a modular structure. Abrami and his team (Abrami, Bernard, Chambers, Schmid, & d'Apollonia, 1998) use the term computer-supported collaborative learning (CSCL) for an instructional setting which incorporates both the use of technology and collaborative learning. WEBCAL, in its completed implementation, is an example of a CSCL instructional setting.





Instructional Design: The instructional design began with a modular approach, dividing the Calculus I curriculum into four modules, with a pre-Calculus review in one additional module. Since functions are the "objects" manipulated in the Calculus I course, following the program of most reform texts, WEBCAL begins with a review of functions in Module 1. Module 1 focuses on developing student familiarity with the notion of a family of functions, and on experimenting with particular families of functions (e.g., power functions, exponential functions, etc.), in multiple perspectives. While students are reviewing functions from multiple perspectives, they are discovering Calculus concepts such as continuity, rate of change, etc. Students are asked to compare and contrast members within a family, and members of different families of functions, and software is used to facilitate the development of the four different perspectives. Thus, students will develop a richer conceptual structure of functions, which in turn will permit them to understand the concepts of limits and continuity, the first traditional Calculus topics, covered in Module 2. Once students have mastered these notions, again from all four perspectives, they are introduced to the concept of "rate of change" or derivative, the major concern of the Calculus I course.

Each module is subdivided into a series of lessons. Because of the nature of the mathematical topics covered, a suggested model of course content delivery is offered, as well as a map of the interdependence of all lesson topics. However, students may choose to follow a different path which better suits their learning styles because they will have access to all course materials at the beginning of the course.

The diagram below provides an overview of the structure of each lesson.



The pre-instructional components of the design, Prerequisites and Check-in pages, provide grounding for students decisions concerning whether or not they can skip a lesson or must work through the concepts it presents. The design of the lesson itself allows students with different learning styles to choose materials or a path through the materials that suits their needs and or learning strategies. The interactive MathView materials of the lesson pages guide the students to experiment, to observe, to formulate hypotheses and to test them. The post-instructional activities, Assignments, Check-Out and formative quizzes (available only in complete implementation) allow them to assess their knowledge before proceeding further.

Lessons and modules are arranged so that the preponderance of students will be able to complete the course during a 15 week term. As an encouragement to students to stay on track weekly quizzes are incorporated into the design. Within those boundaries, students will pace their work on their own.

Student motivation will be derived from: reasonable deadlines; rapid feedback in terms of formative assessment; support for cooperative learning; interactivity of materials encouraging exploration of ideas and testing of hypotheses; and, user control over what, when and how the material is learned. At each stage prerequisite skills are clearly identified, so that students can remedy any lack thereof, thus increasing each individual's likelihood of success. Using WebCT, a product of UBC, a tracking system is built-in so that each student's progress through the course can be monitored, to help ensure that students are progressing at least at minimum acceptable norms. Thus, the instructor can be informed and intervene if a student is not progressing at a pace that will allow them to complete the course.




Learning Objectives: When we adopted the various design elements of WEBCAL it was with a number of general and specific learning objectives for the students in mind.

General:

• to offer access to learning materials to students at any time they choose, for however long they choose;
• to provide a "transparent" interface to learning materials, including the interactive mathematical components;
• to incorporate the Rule of Four, balancing the symbolic with the graphical, numerical and verbal;
• to incorporate interactive "experiments" in mathematics into the materials;
• to shift the locus of control of pace and sequencing from the teacher towards the student;
• t to incorporate immediate access to definitions/explanations of mathematical terminology and formative quizzes through Java and WebCT tools
• to use technology to promote conceptual change in mathematics;
• to incorporate CSCL.



Specific:

• to develop familiarity with families of functions in the four perspectives;
• t to develop an understanding of the concepts of limits, continuity and derivatives in the four perspectives;
• to develop skill in the calculation of derivative functions;
• to develop the ability to apply the key concepts of Calculus to solve problems using the four perspectives.




Innovation:

• Based on a web search of Calculus courses offered on-line in Canadian Universities and Colleges we see that WEBCAL is a shift in paradigm, combining Reform Calculus, use of CAS technology on-line, CSCL and a DE paradigm to the daytime classes. Thus WEBCAL in itself is innovative.
• An Eric search of examinations of the use of technology in the teaching of Calculus turned up five papers, none of which uses a case study methodology. The case study methodology allows a more in depth analysis that is bound to reveal insight into student perception and attitudes concerning the use of technology in Calculus. It may also reveal setting variables that students perceive as important but so far have not been listed as part of the "good" and "bad" practices, thus enlarging the EvNet list.




Theory:

Constructivist Paradigm: The constructivist paradigm of teaching and learning is emerging as a dominant force in educational psychology. Constructivists believe that learning is a process in which students construct the meaning of new ideas, and doing so requires them to modify their existing knowledge structure. The proponents of this paradigm recognize that while teachers can provide scaffolding, they cannot actually modify any student's knowledge structure, only the student can.





Rote Learning versus Conceptual Change: The process students follow to construct meaning often differs. Ausubel (1963) introduced the concept of "rote learning" or "surface processing" to distinguish it from "meaningful learning". In rote learning about a new concept the information about it is stored without translation to a learner's own vocabulary, without any effort to decide under which existing domain the new concept should be stored, and without linking the new concept to any branches of the hierarchical tree that is the existing knowledge structure. That is, neither assimilation nor accommodation (Piaget, 1954) with an existing knowledge structure is sought. Consequently, the existing knowledge structure is not modified, and students are unable to use it in problem solving or transfer their knowledge to different settings.

On the other hand, students may undergo meaningful learning by relating new ideas to those previously held in various ways: by engaging in the process of reconciliation of previously held views with the data gained through their observation and experimentation; by formulating hypotheses, testing the hypotheses and formulating conclusions; etc. In doing this learners re-examine and rearrange their ideas, detect misconceptions and correspondingly modify their understanding. Thus, they emerge with a well structured knowledge structure which more closely resembles that of an expert than their previous structure. While Ausubel (1993) calls such a learning process meaningful learning, others use terms such as "conceptual change" (Posner, Strike, Hewson, & Gertzog, 1982), "cognitive restructuring" (Saunders, 1992) or "deep processing" (Entwistle, & Tait, 1996) to describe essentially the same process. In our previous work we have settled on conceptual change as our label for this process because students emerge from such activity with a new and changed knowledge structure. For consistency we will continue to use this terminology.

While rote learning requires only memorization as an activity on the part of learners, conceptual change requires that learners be actively cognitively engaged in a number of activities. It is important to recognize that such learning involves a continuous process over a long period of time, and teachers, in their choice of instructional setting, set the pace and timing for students. Thus, constructivists have turned to examining the setting in which learning is taking place, seeing it as one of the essential components enabling conceptual change. While some constructivists advocate a dramatic move from a "sterilized classroom" to "apprenticeship settings" (Prawat, 1996), most constructivists seek more modestly to restructure the instructional setting so as to promote students' active engagement. Researchers (Pintrich, Marx, & Boyle, 1993) have suggested that conceptual change occurs only if both cognitive and motivational conditions are satisfied (hot conceptual change theory). They propose that both instructional setting (e.g., task structures, evaluation structures, course management, teacher modeling and scaffolding) and student characteristics and beliefs (e.g., prior knowledge, epistemological beliefs as applied to Calculus, value and importance of learning Calculus, self-efficacy and control beliefs) have to meet these conditions.

Entwhistle (1995) and Kabele (1998) point out that student interpretation of the instructional setting is influenced by their individual knowledge structure, by the learning strategies that they are accustomed to using and are skilled at, and by the socially shared understanding of it. The further the instructional setting is from the norm that the class is used to, the more varied and individual the perceptions of that new setting are likely to be. Further, in a notion not unlike that of the Heisenberg Uncertainty Principle, Kabele points out that what interpretation a student will make in a situation is probabilistic at best, and those probabilities depend upon good knowledge of the range of possibilities. In a novel situation such as WEBCAL, it is thus critical that the variety of possibilities be "mapped". Winning active participation in the WEBCAL structure from the majority of students may require adaptations to that structure. Adaptations designed to help move students from their multiple different interpretations towards "buying into" an interpretation shared with the designers of that structure.




Cognitive conditions for conceptual change in instructional settings: The cognitive conditions for conceptual change may or may not appear in every aspect of the instructional setting. The task structure that allows students time to wrestle with concepts, gives them opportunities to experiment and observe the outcomes of their experimentation and encourages them to formulate hypotheses, design and carry out tests to validate those hypotheses, is seen as setting cognitive conditions for conceptual change. Similarly, tasks which involve relating different perspectives of concepts, as is done in reform Calculus classes, generates a setting which encourages conceptual change. The evaluation structure is another element of the setting which may or may not encourage students' conceptual change. There are two aspects to this structure, what is being assessed and how it is assessed. Assessment which demands that students demonstrate conceptual understanding, as opposed to demonstrating lower level mechanical skills, encourages them to become involved in their work and to make a conceptual change (Dedic, & Rosenfield, 1994; Mazur, E, 1996). Assessment which serves formative goals encourages students to continue to develop deeper understanding. Assessment which serves summative goals encourages students to bring closure to further construction of meaning. Since summative assessment is a necessary component of any evaluation structure in any instructional setting, it can be said that for an instructional setting to promote conceptual change it must include an evaluation structure which tests conceptual understanding and serves formative goals.

Furthermore, research suggests that learning environments which incorporate cooperative group work (Abrami, Chambers, Poulsen, DeSimone, d'Apollonia, & Howden, 1995; Bosse, & Nandakumar, 1998) promote conceptual change by requiring students to engage in conversations about subject matter. In this sense, students in a CSCL mathematics course are being asked to behave like mathematicians by cooperating in the development of a difficult abstract concept, and according to Lave (Lave, 1991) this prepares them to use their knowledge in the workplace. The construction of meaning is also enhanced by the need to present a completely formulated idea to the group (Harasim, 1987), to defend the idea against criticism, and in turn, to criticize ideas of others. Thus, group discussion helps the learner not only to develop alternative points of view and but also to appreciate alternative points of view and consequently, promotes conceptual change.

The use of technology in the classroom can enhance students' interaction with materials and thus promote active cognitive engagement. It has been shown to promote students' achievement (Mayer, 1997). Abrami and his team (Abrami, Bernard, Chambers, Schmid, & d'Apollonia, 1998) hypothesize that CSCL promotes conceptual change and they are presently testing this hypothesis in various contexts from grade school to university courses to workplace applications.




Motivational conditions for conceptual change in the instructional settings: The motivational conditions for conceptual change may or may not appear in every aspect of an instructional setting. For example, tasks which are at the same time both challenging and accessible, avoid disengagement due to either boredom or frustration and promote students' feelings of self-efficacy. The evaluation structure can produce a similar effect on students' disengagement and expectancy of success as the task structure. Researchers into student motivation increasingly focus on the role of students' assessment of the value of what is being learned, their expectancies of success, and the costs perceived associated with achievement of that success. These individual assessments are not formulated in isolation, but are the result of socially shared assessment among students in a classroom. Abrami and his colleagues (Abrami et al, 1998) hypothesize that CSCL promotes a socially shared assessment of values, expectancy and costs which promotes conceptual change. Most theorists agree that students' perception of control over their learning enhances their cognitive engagement and hence promotes conceptual change. Thus, an instructional setting in which the control of what is learned, when it is learned and how it is learned is shared between the instructor and students promotes beliefs of control among the students and consequently, satisfy one of the motivational conditions for conceptual change.




Cognitive conditions for conceptual change in student characteristics and beliefs

An appropriate prior knowledge structure is a cognitive condition for conceptual change. Dicke and Farrell (1991) and Donald (1994) found that there is a mismatch between students' domain specific prior knowledge structure and teachers' expectations in this area. Consequently, they report a lack of conceptual change. In addition, an appropriate knowledge structure must include strategic knowledge, e.g., the learner must have strategies concerning how to deal with conflicting data. For example, learners who do not have strategic knowledge concerning how to organize concepts cannot conceptualize chemical nomenclature, only memorize it. According to Pintrich and his colleagues (Pintrich, Marx, and Boyle, 1993) the prior knowledge structure influences perception and selective attention to new information. Students may mis-perceive or choose to ignore data that contradicts their prior concepts. In such cases, the students' prior knowledge structure becomes a hindrance to learning.




Motivational conditions for conceptual change in student characteristics and beliefs: The expectancy theory perspective (Tolman, 1932; Vroom, 1964) of students' motivation has three components: value; expectancy; and, cost. These three factors influence students' decisions and approach to learning. Namely, a decision to commit significant time and effort towards learning Calculus will depend upon: how highly the student values knowledge and understanding of Calculus; how successful the student expects to be in this undertaking; and, how high the student perceives the cost(s) involved in attaining that success. Although it is commonly agreed that we are living in a technological age, students are reluctant to engage in the use of technology outside of computer courses or recreation in part because they suspect its use adds additional burdens. Thus, the expectancy theory perspective is particularly apt in our circumstances.

It is important to note that these three factors are not unchangeable personality traits. It is believed that the instructional setting changes them. Similarly, students' control beliefs are believed to be affected by the instructional setting and are of import for cognitive engagement.

Researchers have shown that students' epistemological beliefs and conceptual change are strongly related (Hammer, 1994). However, there is no common agreement as to the components of those beliefs, whether they are developmental or a fixed personality trait, or how to assess such beliefs (varying from interviews to questionnaires), but there are two components that are common across most epistemological theories: knowledge is simple; knowledge is certain. Schommer and her colleagues (Schommer, Crouse & Rhodes, 1992) label students who believe that "to learn is to remember a set of simple facts" as students who believe that knowledge is simple. They have demonstrated that simple knowledge beliefs are negatively correlated with comprehension and meta-comprehension. Further, they have found that the influence of belief in simple knowledge could be mediated by appropriate strategies. Students who believe that knowledge is fixed, not relative, usually residing in the teacher or the text, may be said to have the belief that knowledge is certain. It has been shown that standard science education settings reinforce the belief that knowledge is certain (Paulsen, & Wells, 1998). With this belief students are not disposed to engage in tasks required to construct knowledge, because it is faster and easier to obtain it from the teacher or text. Clearly this is a factor of import for cognitive engagement and whether a conceptual change takes place.




Methodology:

Research design:

Case Study Methodology: There are five components to our use of this methodology: observation in the classroom; analysis of materials (written and electronic) provided to students; evaluation questionnaires; summative assessment of students; and, a series of interviews (beginning of term, mid-term, end of term) with selected students.

A method developed by Kouros (Kouros, d'Apollonia, Poulsen, Howe, & Abrami, 1993) will be used for the classroom observation.  This component will supply data concerning how WEBCAL has implemented various design elements and their overall impact on students.

Course materials will be coded and analysed and the results used to evaluate how WEBCAL meets its general and specific objectives.

Evaluation questionnaires will be analysed for student perception of the impact of various WEBCAL design elements. In particular, we will looking at the use of and perception of the usefulness of student-student and student-teacher communication. To measure students' motivation (value of Calculus, expectancy of success, control beliefs) we will use the Motivational and Strategic Learning Questionnaire (MSLQ) (Pintritch, Smith, Garcia, & McKeachie, 1991). The six motivational scales (intrinsic goal orientation, extrinsic goal orientation, task value, control of learning beliefs, self-efficacy and test anxiety) and three scales which assess the use of learning strategies of MSLQ were validated by the authors (Pintritch, Smith, Garcia, & McKeachie, 1993). MSLQ is a robust instrument and allows both, the adaption of items for Calculus and the use of selected scales. We will use the Epistemological Beliefs Questionnaire (EBQ) (Schommer, 1990) to assess students' beliefs in simple knowledge and beliefs that knowledge is certain. The EBQ was subsequently validated by Schommer et al (Schommer, Crouse, & Rhodes, 1992). Qian et al (Qian, & Alvermann, 1995) refined and tested a slimmed down EBQ by eliminating roughly half of the questions, bringing it down from four factors to two. However, Hofer et al (Hofer, & Pintrich, 1997) raised doubts about the entire notion of testing epistemological beliefs via a questionnaire. Thus, we intend to consider both the refined EBQ and incorporation of such questions into interviews.

Student response to summative assessment (quizzes, term tests, final exam) will be used to evaluate students' attainment of curricular objectives. Assessment items will be coded for testing conceptual understanding versus rote learning. Test items will be shared with instructors of standard Calculus sections.

The interviews will provide in depth information concerning conceptual change, student perceptions and motivation and allow us to provide a rich description of the reaction to WEBCAL of students with varied pre-existing skill levels.





Participants: Our experiments will involve a sample of first year Vanier College pre-university science students taking Calculus I (201-103). We expect the sample size to be approximately 40 students. Our sample of students is intended to represent the population of CÉGEP students enrolled in the pre-university science program. Note that in accordance with accepted practices at Vanier College, students choose their own section.

We will divide the class into high, medium and low achievers in mathematics, using prior grades in mathematics courses for classification purposes. Within each category we will create a randomized list of students, and for interviews we will seek participation from 2 students per category, using the lists.




Dissemination:

• We intend to present our work as it progresses through the Working Papers folder in LearnLink.
• We will download a sample lesson to LearnLink and invite EvNet researchers to critique the lesson from the constructivist perspective and the technology used to promote the conceptual change.
• We would like to use LearnLink to begin a discussion concerning the use of qualitative methodology to evaluate the use of technology in education.
• Steven Rosenfield is a member of the technology subcommittee of the Mathematics Planning and Coordinating committee (MAPCO) that serves to implement the mathematics curriculum in all anglophone secondary schools in Quebec. Through discussion in this subcommittee the mathematics consultants for the three major boards have expressed interest in linking their students to the pre-Calculus module and the Functions Module of WEBCAL, and indicated that they will distribute results of an evaluation of WEBCAL to their teachers to promote the use of these materials.
• Steven Rosenfield was an invited speaker at the second annual conference of Utilisation des logicels de calcul symbolique dans l'enseignment (http://www.etsmtl.ca/seg/colloque/Index.htm) on the topic of the construction of WEBCAL this past fall. He will present the fully developed WEBCAL at the next conference, and the evaluation of WEBCAL the following year.






Training:

Under the careful supervision of the research team a graduate student will participate in all aspects of the research programme, including research meetings, publications, conference presentations, seminars, and materials development. The student will work with the research team in conducting literature searches, retrieving articles, formulating and refining the design of the study, constructing instruments, collecting and analysing data, and writing up the results of the study. The student will be invited to co-author papers based on the project. The student will be expected to participate in the CSLP seminar series. The student will collaborate in the continuing improvement of the WEBCAL course and so acquire experience and develop skill in the use of a variety of computer software packages used in the development of computer based education materials.

Schedule:

The evaluation of WEBCAL will involve the following activities by the team members:

1999 - 2000

• collection of written and electronic materials and development of a coding scheme
• observation of classroom setting and development of a coding scheme
• development of conceptual questions
• development of three protocols for semi-structured interviews, test of protocols, test of conceptual questions and development of coding schema for interviews and questions
• report on work in progress through LearnLink - Working Papers




2000 - 2001

• run the case study in the fall term
• code and analyse data obtained
• prepare papers based on the results and disseminate via Working Papers, conferences and journals






---

Appendix A - Sample of WEBCAL Lesson

The index is a hyper-linked html file that is the initial file that a student sees upon choosing a particular lesson. At the moment we are testing a linear presentation of the components of a lesson since it offers us an opportunity to show students how we see the order of the materials. However, we are debating an alternative that would use a clickable version of the map shown above. In either case, the student will be able to access the materials in the order that they themselves choose. A sample index page in list format is shown in the box below:

The prerequisites page lists the topics that the student should know to be able to complete the lesson, and it is linked to previous lessons where that content can be found. A sample is shown in the box below:



The check-in page is a list of problems that students can view to help them decide for themselves if they really need to work through a particular lesson. A sample is shown in the box below:

 





The lesson page is the main body of a lesson and it may be many screens long. This page includes the interactive examples that we believe are critical to helping students understand the concepts presented. Unfortunately, a text based representation of this page does not explain what it is like, so no example is presented. A visit to the web site, once the MathView plugin has been installed, it the only way to truly sample this experience.




The examples page contains a sequence of examples linked to the lesson page. A sample is shown in the box below:




The assignment page consists of exercises assigned to the student for practice. In some cases the student will be referred to exercises in a textbook, sometimes the exercises will refer to the interactive examples in the lesson page, and sometimes the exercises will be entirely self-contained. A sample is shown in the box below:



The check-out page contains a list of learning objectives, in concrete form, so that the student has a clear opportunity to self-assess and decide if they have mastered the material of that particular lesson. A sample is shown in the box below:


All pages in a module lesson contain a list of keywords pertinent to that lesson in a frame row at the top of the page. A simple click pops up a window with the definition of the keyword. A sample of a keyword definition is shown in the box below:





---

Appendix B - Bibliography



Abrami, P. C., Chambers, B. Poulsen, C., DeSimone, C., d'Apollonia, S., & Howden, J. (1995). Classroom connections: Understanding and using cooperative learning. Harcourt-Brace.

Abrami, P. C., Bernard, R., Chambers, B., Schmid, R., & d'Apollonia, S. (1998). Responsible, effective and active learning through computer supported collaborative learning. FCAR (Programme soutien aux équipes de recherche), (1998-2001).

American Association for the Advancement of Science (A.A.A.S.). (1982): Education in the Sciences: A Developing Crisis. Washington, D.C.: American Association for the Advancement of Science.

American Association for the Advancement of Science (A.A.A.S.). (1984): A Report on the Crisis in Mathematics and Science Education: What Can Be Done Now? New York: J.C. Crimmins.

Ausubel, D.P. (1963).The psychology of meaningful verbal learning.New York: Grune & Stratton, Inc.

Barber, F. & Narayan, J. (1994). Technology, Cooperative Learning and Assessment in the Teaching of Ordinary Differential Equations. Primus, 4, 4, 337-346.

Blumenfeld, P. C., Mergendoller, J., & Puro, P. (1992) Translating motivation into thoughtfulness. In H, Marshall (Ed.), Redefining learning (p. 207-239). Norwood, NJ: Ablex.

Bosse, M. J., & Nandakumar, N. R. (1998). Calculus Ideas Generated through Cooperative Learning. Mathematics and Computer Education, 32, 1, 52-61.

Cooley, L. A. (1997). Evaluating Student Understanding in a Calculus Course Enhanced by a Computer Algebra System. Primus, 7,4, 308-316.

Dedic, H. & Rosenfield, S., 1994, Assessment, the Horse that Pulls the Cart of Learning, L'évaluation évaluée, Fédération autonome du collégial (FAC).

Dickie, L. O. & Farrell, J. E. (1991). The transition from high school to college: An impedance mismatch. The Physics Teacher, 29, 440-445.

Donald, J. G. (1994). Science Students' Learning: Ethnographic Studies in Three Disciplines. In P. R. Pintrich, Brown, D. R. & Weinstein, C. E. (Eds.), Student Motivation, Cognition, and Learning: Essays in Honour of Wilbert J. McKeachie. Hillsdale, NJ: Lawrence Erlbaum Associates.

Entwistle, N., (1995). Introduction: Influences of Instructional Settings on Learning and Cognitive Development - Findings From European Research Programs. Educational Psychologist, 30, 1, 1-3.

Entwistle, N. & Tait, H. (1996). Identifying Students through Ineffective Study Strategies. Higher Education, 31, 1, p.97-116

Ferrini-Mundy, J. & Lauten, D. (1994). Learning about Calculus Learning. The Mathematics Teacher, 87, 2, 115-121.

Guskin, A. E. (1994). Reducing Student Costs and Enhancing Student Learning - The Univeristy Challenge of the 1990s. Change, July/August, 23-29.

Guskin, A. E. (1994). Reducing Student Costs and Enhancing Student Learning - Restructuring The Role of Faculty. Change, September/October, 16-25.

Guskin, A. E. (1996). Facing the Future- The Change Process in Restructuring Universities. Change, July/August, 27-37.

Hammer, D. (1994). Epistemological Beliefs in Introductory Physics. Cognition and Instruction, 13, 3, 401-430.

Harasim, L. (1987). Teaching and Learning On-Line: Issues in Computer Mediated Graduate Courses. Cannadian Journal of Educational Communication, 16, 2, p.117-135.

Hodgson, B. R. (1987). Evolution in the Teaching of Calculus. Calculus for a New Century: A Pump, Not a Filter. M.A.A. Notes, 8, 49-50.

Hofer, B. K. & Pintrich, P. R. (1997). The Development of Epistemological Theories: Belifs About Knowledge and Knowing and Their Relation to Learning. Review of Educational research, 67,1. 88-140.

Johnson, D. W. (1998). (private communication).

Kabele, J. (1998) Perody, Principy Socialního Konstruování, Karolinum: Karlova Universita, Praha.

Kouros, C., d'Apollonia, S., Poulsen, C., Howe, N., & Abrami, P.C. (1993). Observing cooperative group work in college biology classrooms: The use of a structured scheme. Invited article in G. Huber (Ed.). Neue perspektiven der kooperation. Germany: Schneider Verlag Hohengehren.

Lave, J. (1991). Situated Learning in Communities of Practice. In. Resnick, L. B., Levine, J. M. & Teasley, S. D. (Eds.), Perspectives on socially shared cognition. Washington, D.C.: American Psychological Association.

Malcolm, C.; Welch, W., (1981). Case Study Evaluations: A Case in Point. An Illustrative Report and Methodological Analysis of Case Study Evaluations. Minnesota Univ., Minneapolis. Minnesota Research and Evaluation Centre.

Mathematical Association of America (M.A.A.). (1986): Toward a Lean and Lively Calculus: Report of the Conference/Workshop To Develop Curriculum and Teaching Methods for Calculus at the College Level. M.A.A. Notes Number 6.

Mathematical Association of America (M.A.A.). (1987): Calculus for a New Century: A Pump, Not a Filter. M.A.A. Notes Number 8.

Mathematical Association of America (M.A.A.). (1988): Computers and Mathematics: The Use of Computers in Undergraduate Instruction. M.A.A. Notes Number 9.

Mayer, R.E.(1997). Multimedia learning: Are we asking the right questions? Educational Psychologist, 32, 1-19

, Mazur, E. (1996). Understanding of Memorization: Are we teaching the right thing. Proc. Resnick Conference, in press, Wiley, 1996.

Neff, R. K., (1998). What Technology Offers to Education: Some Evidence that Computers Do Improve Learning. Breakfest on Campus, Yniversity of Ottawa, May 25, 1998.

Papert, S., (1980). Mind-storms: Children, Computers and Powerful Ideas. New York, NY: Basic Books.

Paulsen, M. B., & Wells, C. T., (1998). Domain Differences in the Epistemological Beliefs of College Students. Research in Higher Education, 39, 4, p.365-384.

C, J. (1954). The construction of reality in the child. New York, NY: Basic Books.

Pintrich, P. R., Marx, R. W., & Boyle, R. A. (1993). Beyond Cold Conceptual Change: The Role of Motivational Beliefs and Classroom Conceptual Factors in the Process of Conceptual Change. Review of Educational Research, 63, 167-199.

Pintrich, P. R., Smith D. A. F., Garcia, T. & McKeachie, W. J. (1991). A Manual for the Use of the Motivated Strategies for Learning Questionnaire (MSLQ). National Centre for Research to Improve Postsecondary Teaching and Learning, Ann Arbor, MI.

Posner, G., Strike, K., Hewson, P., & Gertzog, W. (1982). Accommodation of a scientific conception: Toward a theory of conceptual change. Science Education, 66, 211-227.

Prawat, R. S., (1996). Constructivism, Modern and Postmodern. Educational Psychologist, 31, (3/4), 215-225

Qian, G., & Alvermann, D. (1995). Role of epistemological beliefs and learned helplessness in secondary school students' learning science concepts from text. Journal of Educational Psychology, 87,2, 282-292.

Saunders, W. L. (1992). The Constructivist Perspective: Implications and Teaching Strategies for Science. School Science and Mathematics, 93, 136-141.

Schommer, M. (1990). Effects of beliefs about the nature of knowledge on comprehension. Journal of Educational Psychology, 82, 498-504.

Schommer, M., Crouse, A.& Rhodes, N. (1992). Epistemological Beliefs and Mathematical Text Comprehension: Believing It Is Simple Does Not Make It So. Journal of Educational Psychology, 84, 435-443.

Tolman, E. C. (1932). Purposive behaviour in animals and men. New York, NY: Appleton-Century.

Vroom, V. H. (1964). Work and Motivation. New York, NY: Wiley.