Article

UniServe Science News Volume 16 July 2000

An Exercise in Course Evaluation

Introduction

In April/May, 2000, one of the authors (CS) was a visiting fellow at UniServe Science and the School of Physics, The University of Sydney, under the Thailand Australia Science and Engineering Assistance Project (TASEAP). As part of that fellowship he undertook to evaluate the effectiveness of a course in computational physics which the School had just introduced. This module, an update of a long-standing component of the experimental physics curriculum for advanced students, introduced them to the topic of 'Oscillations and Chaos' by means of a newly written set of computer exercises based on the mathematical package MatLab¹. The purpose of the evaluation was not only to allow the School to monitor the effectiveness of this particular item in its teaching program, but also to establish protocols for future evaluations of other parts of the curriculum.

Description of the course

The computational physics module, 'Oscillations and Chaos', is one of six in Experimental Physics: Physics 1901 (Advanced). It consists of three-hour sessions in the laboratory, which students are required to attend once a week for three weeks. They work in groups of two or three, through a set of exercises provided in the laboratory notes². Each group has access to a computer for the entire session. Students can obtain help during the session from tutors.

As with many university courses, the primary aims of this course had never been explicitly formulated before it was put together. However consultation with the teachers and tutors reached agreement that the course's main aims were:

to provide an introduction to the role of computation in physics;

to appreciate the role of modelling in physical understanding;

to become comfortable with using computers in general; and

to gain some understanding of oscillatory and chaotic behaviours.

Methodology

The project sought to gather information about the students in three areas: attitudes; knowledge; and skills (see Table 1). Three different ways of collecting information were used: before and after interviews with participating tutors and lecturers; questionnaires for evaluation of the students' performance at the beginning and end of the course; and observation of how students approached their work during class. The objectives of each method are summarized in Table 2.

Design of the Pre-Questionnaire

It was decided that it was important to determine the students' backgrounds with which they came to this course in two categories.

Attitudes and Skills. They were asked to give their own rating on their computer skills, their mathematical ability and their understanding of the relevant physical concepts. Responses were asked for on a 1-5 (low - high) scale.

Knowledge. They were tested on their understanding of the basic facts about simple harmonic motion, and their familiarity with the solution of ordinary differential equations. Testing was done mainly by short open-ended questions.

Design of the Post-Questionnaire

The post questionnaire was designed to check the development of the students in the same two areas.

Attitudes and Skills. They were asked to give their own rating, on a 1-5 scale, of the usefulness of this kind of computation and MatLab in particular, whether their mathematical skills and their physics knowledge had improved, and their attitude towards the material.

Knowledge. They were tested, by short open-ended questions, on their understanding of properties of oscillations and chaos, and computational and mathematical methods. Some of these questions were the same as on the pre-questionnaire.

Results and discussion

The raw data from all of these information-collecting exercises can be viewed on-line⁵. Only the most significant findings are reported here.

Students' attitudes toward their own computer skills

More than 80% of the students felt very comfortable with their basic computer skills such as using Windows, managing files, backing up data and using the mouse and keyboard. This implies that students came to this course with adequate basic computer skills. On the other hand, they lacked experience in writing programs and using mathematical packages, implying they had little idea about using the computer as a tool for solving mathematical problems.

After the students had finished this course, more than 90% agreed that computational methods are helpful in understanding physics phenomena and in solving physics problems. In addition, most agreed that MatLab is useful for tackling many mathematical problems. These results show that, at least in their own estimation, this course did give students a feeling for the role of computational methods in physics - which was one main aim of the course.

Students' attitudes toward their own mathematical and physics skills

At the start of the course most of the students were highly confident of their mathematics ability. At the end, very few felt that this course had helped in improving their mathematical skills. These findings are at least mutually consistent.

On the other hand a much smaller fraction felt they knew much of the physics at the start, particularly chaos; and at the end, more than 90% found the course to have been helpful in improving their physics knowledge. Again these results indicate success in exposing them to ideas about chaos - which was another aim of this course.

Students' attitudes toward the course as a whole

While more than 60% of the students felt that this course was interesting, enjoyable and useful in learning physics, only a very small fraction felt that they were encouraged to do further study. While not exactly contradicting any of the course aims, this feature warrants consideration in future.

Students' performance in test questions

Comparison of the results from the pre-tests and post-tests show that students did indeed improve their knowledge about oscillations; damping, driving, resonance and chaos. In addition, the number of students claiming not to know about solving differential equations decreased after they had finished the course. This has to be a success in terms of the aims.

However, a qualitative analysis of the open-ended questions indicated a marked increase in the tendency to answer those questions from a mathematical, rather than a physical, point of view. An example is the question 'What is the difference in behaviour between a pendulum swinging through a small angle and one swinging through a large angle?', which appeared on both pre- and post- tests. At the start of the course, most students answered in terms of actual behaviour - the period gets longer, or similar. At the end, the number of students answering this question in terms of linearity or the approximation involved had increased dramatically. This point also needs to be investigated more closely, in view of the fact that an aim of the course was deemed to be to appreciate the role of modelling in physical understanding.

Conclusions

In the modern university system, in most countries in the world, increased emphasis is being placed on improving the quality of teaching. It follows that we need to develop ways of evaluating the effectiveness of our courses which are transparent, comprehensive and not too time-consuming. The authors believe that the protocols described above are just that. They could easily be transferred to other courses. There may be doubt in the reader's mind that undue stress has been placed on students' own estimation of the outcomes of the course, rather than on more 'objective' measures. This is, however, in line with current trends in educational circles to carry out evaluations which focus on the student's, rather than the teacher's, perspective, for the reason that all learning is, in the end, mediated by students' perceptions of what they are being asked to do⁶.

One last point should be made however. Before any reasonable evaluation of a course can be done it must be clear what the aims of the course are. As mentioned above, many university courses do not set these down before the course is put together. Too often the aims are defined in terms of the syllabus - 'They need to know x, y and z'. If that is the case, then the first job of the evaluator will be to determine what the aims are, which is indeed what happened here.

References

1. MatLab, copyright The Maths Works Inc., 1984-96
2. The complete course notes can be viewed at http://www.physics.usyd.edu.au/teach_res/CP1Chaos_site/cp1chaos.htm
3. The pre-questionnaire is available in pdf format at http://science.uniserve.edu.au/newsletter/vol16/prequest.pdf
4. The post-questionnaire is available in pdf format at http://science.uniserve.edu.au/newsletter/vol16/pstquest.pdf
5. Complete results are available in pdf format at http://science.uniserve.edu.au/newsletter/vol16/results.pdf
6. See for example, Prosser, M. (2000) Evaluating the New Technologies: A student learning focused perspective, Proceedings of the Fifth UniServe Science workshop: Evaluating the New Teaching Technologies, Sydney: UniServe Science.

Chernchok Soankwan
Department of Physics
Mahidol University
25/25 Moo 3
Phuttamonthon 4 Road
Salaya
Phuttamonthon District
Nakhon Pathom 73170
Thailand
sccsk@mahidol.ac.th
and
Ian Johnston
School of Physics
The University of Sydney
idj@physics.usyd.edu.au

Page Maintained By: PhySciCH@mail.usyd.edu.au
Last Update: Monday, 30-Apr-2012 15:40:33 EST
URL: http://science.uniserve.edu.au/newsletter/vol16/soankwan.html