CAL-laborate Volume 3 October 1999

Testing Conceptual Understanding in Physics Using a Browser-based Computer Managed System

Abstract

Testing Conceptual Understanding in Physics (TCUP) is a CUTSD-funded project which aims to monitor the conceptual understanding of first-year university Physics students. It is designed not as an educational research tool but rather to provide timely feedback on student conceptual understanding. This feedback is available both to students and their instructors. The TCUP tests will help promote more effective teaching, and also help to evaluate the effectiveness of physics courses in teaching basic physics concepts. To promote effective learning, individual students will receive formative feedback regarding their level of conceptual understanding in each key area of their introductory physics course.

Each test (which is not intended to count towards summative assessment) is administered via an easy to operate browser-based computer managed testing system. The system, which is fully automated, also provides statistical information to the instructor regarding the performance of the class.

The TCUP project team

Project leaders: Dr. Alex Mazzolini, Dr. Margaret Mazzolini (Swinburne University of Technology) and Dr. John Humble (University of Tasmania).
Question Bank developer: Prof. Bill Rachinger (Monash University and Swinburne University of Technology).
Web delivery: Dr. Barbara Moss (Swinburne University of Technology).

Introduction

Many university physics teaching staff are often dismayed at how little their students have learned after their first year of physics instruction. A number of physics education researchers, including Redish and Steinberg¹, have suggested that many traditional teaching methods do little to improve students' conceptual understanding. Even worse, tertiary teaching staff are often unaware that a problem exists with their teaching methods. Many concerned and dedicated teachers listen to students but fail to recognise their real difficulties. Here is an example taken from Redish's recent paper: "...in a junior level electronics class, a student asked a question about a comparison of currents at two points on a single branch of a relatively complicated circuit...[The teacher]...listened carefully and recognized that the student was confused. He proceeded to give a detailed description of how the entire complicated circuit worked. However, because the current was necessarily the same throughout the branch of the circuit, it was likely that the student's difficulty was a deep one - and not addressed by the instructor's response; namely, that somehow the current was 'used up' en route [according to the student]. The student politely nodded, no better off than before, and the teacher moved on.". The above experience is not an isolated case but rather an example of what is probably a common occurrence in many university courses throughout Australia.

Thornton and Sokoloff² have quantitatively shown that traditional teaching methods do little to improve the conceptual understanding of students undertaking introductory physics units, and that more student-centred methods of teaching need to be considered. In their study, "fewer than 10% of the students seemed to change their (incorrect) views of dynamics after traditional instruction".

Many tertiary physics academics may not be aware of the extent of the problem for the following reasons:

(i) During a physics course, students are typically given a great deal of practice in problem-solving skills via problem sheets and tutorial exercises. Students concentrate on developing these skills rather than developing conceptual understanding, since problem solving skills are the ones most likely to be tested in the final examination.
(ii) Often there is subtle pressure on academics to set predictable examinations. Unconventional questions invariably lead to poor examination results and pass rate problems that require justification to Heads of Department or Deans. As a consequence, students learn how to answer the predictable question styles and can often pass examinations without a good understanding of the material presented in the course.

The development of the TCUP project, which tried to address the above problems, was based on three assumptions:

(i) that assessing students' conceptual knowledge and preconceptions would be a significant aid to effective teaching;
(ii) that concept development is a crucial learning outcome not sufficiently catered for in standard physics courses; and
(iii) that physics departments need tools to evaluate the success of their courses, if they are to be persuaded to trial alternative methods of instruction.

The TCUP project

TCUP is a CUTSD (Committee for University Teaching and Staff Development) funded project that has assembled an extensive set of question banks designed to test for conceptual understanding in all major areas of an introductory tertiary physics course. The multiple choice tests are delivered electronically via a browser-based computer managed testing package. The computer delivery method provides immediate feedback to students during their studies with minimal commitment of staff time. In the year 2000, the TCUP tests will be available for use in all physics departments at universities throughout Australia. The TCUP project is fully-funded by CUTSD and Swinburne University of Technology; there are no additional charges to the users of the TCUP tests.

TCUP has several strengths:

Course coordinators will be able to provide regular diagnostic feedback to encourage students to improve their understanding of fundamental concepts in their physics courses.

The process of "working through" the TCUP question banks will in itself signal to students that a deeper approach to learning is a requirement for conceptual understanding, and will show that conceptual understanding as well as problem solving skills are necessary outcomes of an introductory physics course.

Teaching staff will also receive immediate, automatically generated diagnostic feedback on how well each student group (as a whole) mastered key concepts. This will allow the staff the option of undertaking remedial action and adjusting their teaching emphasis accordingly.

The TCUP question banks will allow teaching staff to do pre- and post- course concept-mastery testing. This will aid individual staff in judging the degree by which their existing courses are successful in transferring basic physics concepts. It will also be an invaluable tool in the trialling of new courses and alternative techniques of course delivery.

TCUP also has several limitations:

Ideally TCUP questions and choice of multiple choice distractors would be based on open-ended questions, preferably in the context of student interviews. However this is a time-consuming process, and diagnostic testing as a feedback tool will only be effective if it covers most of the "standard" syllabus. As TCUP aims to provide a mechanism for useful feedback rather than to be an educational research tool, it was decided instead to rely on individual expertise, plus pre-existing diagnostic tests where available, (in particular, the expertise of Prof. Bill Rachinger, who has developed diagnostic tests in physics over many years, and his surveys of the literature reporting student interviews).

The TCUP tests are purely multiple choice format. The WWWAssign computer testing system used to deliver the tests can also use numeric and other formats. However, given that the TCUP project required automatic marking of student answers (which ruled out extended answer format) and that TCUP is testing conceptual understanding rather than problem solving techniques (which reduces the need for numeric answer format), it was decided that TCUP questions would be restricted to multiple choice format. Multiple choice, a much-discussed and often maligned testing format, does have the advantage that feedback can be directed specifically at "distractors" which match common student misconceptions.

TCUP question banks

Prior to TCUP, no single, comprehensive set of question banks designed to test conceptual understanding existed across the core introductory Physics courses. Suitable question sets for parts of the core introductory Physics course (as commonly taught in Australian universities) have been developed in other places, especially the United States (e.g. Force Concept Inventory³, Mechanics Base Line Test⁴). Other question sets, such as the Galileo ConcepTests⁵, are extensive in topic coverage but have only small question banks for each topic. The TCUP project undertook an extensive survey of these and other question banks that could be used to test conceptual understanding in physics. The TCUP question banks used existing question sets as much as possible, modifying and supplementing them where necessary to cover a typical Australian introductory syllabus.

Currently, 45 TCUP tests covering electricity, magnetism, electronics, linear mechanics, rotational mechanics, waves, optics, thermal, and nuclear physics have been developed. This extensive list is shown in Table 1. The TCUP Home page is: http://www.swin.edu.au/bsee/mazzo/tcup/

It should be emphasised that the TCUP tests are not intended to form part of a graded student assessment - rather, they are a tool to aid concept development in Physics. Students will evaluate their individual progress via electronic feedback from the tests. The feedback will help them identify which particular key physics concepts require more attention in order for them to master the course material.

Electronic delivery of TCUP tests

TCUP tests are accessed from the World Wide Web via any standard web browser. Students will find the electronic delivery system easy to use, and academic staff will find it easy to administer. Figure 1 shows the operational procedure used to enable and run the TCUP tests. The browser-based, computer managed testing system being used is called WWWAssign and is written in the Perl programming language. WWWAssign was originally written by Prof. Larry Martin from North Park University and modified by Prof. Aaron Titus from North Carolina State University. TCUP has been given permission to use the shareware version of WWWAssign free of charge. It should be pointed out that a full commercial version with a robust database structure and a more sophisticated instructor interface (WebAssign) is also available for academics interested in other computer managed testing applications.

Sample Test (Instructors or students)

Standard TCUP Test (Instructors)

Standard TCUP Test (Students)

Figure 1. Operational procedures for instructors and students

To maintain test bank security, all TCUP tests must be supervised. Teaching staff wishing to use TCUP tests for their students will need to electronically import their class list and enable student access to individual tests. To do this academics must obtain an access password from the TCUP administrator. All results are recorded and processed automatically and immediately to minimise the amount of test administration for staff. The TCUP Instructors' site, which is used to enable tests, create class lists etc., is shown in Figure 2.

Figure 2. Instructors' TCUP administration page

Each test has about 10 to 15 multiple choice questions. The questions may include diagrams, and there is no time limit on completing a particular test. A typical sample test (covering electricity and magnetism) and instructions on how to access the test are available on the web site and can be used with unrestricted access. An example of a typical question taken from the sample test is shown in Figure 3. The sample test can be accessed from the TCUP home page address by clicking on the "Try out a sample TCUP Test" hot link. Other tests are also available, but require an access password.

Figure 3. Typical question from one of the "sample test" TCUP question banks

Conclusion

Over the past decades in Australia there have been considerable changes both in senior high school physics curricula and in the proportion of secondary students continuing to tertiary education, and consequential changes in the skills and deficiencies of students going on to study physics at the tertiary level. Many university physics teaching approaches have not adapted to accommodate these fundamental changes. It is hoped that the TCUP project will be widely used in Australian universities to provide instructors with feedback on student's levels of conceptual understanding, and to provide students with timely feedback on their level of mastery of key concepts. It is also hoped that TCUP will provide statistical data that will assist instructors to trial new teaching methods to enhance their students' conceptual understanding in physics.

References

1. Redish, E. F. and Steinberg, R. N. (1999) Teaching Physics: Figuring Out What Works, Physics Today, January, 24-30.
2. Thornton, R. K. and Sokoloff, D. R. (1998) Assessing student learning of Newton's laws: The Force and Motion Conceptual Evaluation and the Evaluation of Active Learning Laboratory and Lecture Curricula, Am. J. Phys., 66, 338-352.
3. Hestenes, D., Wells, M. and Swackhamer, G. (1992) Force Concept Inventory, The Physics Teacher, 30, March, 141.
4. Hestenes, D. and Wells, M. (1992) A Mechanics Baseline Test, The Physics Teacher, 30, March, 159.
5. Mazur, E. http://galileo.harvard.edu/home.html

Alexander Mazzolini
School of Biophysical Sciences and Electrical Engineering
Swinburne University of Technology
PO Box 218
Hawthorn
Victoria 3122
Australia
amazzolini@swin.edu.au

© 1997 - 2008 UniServe Connections

Page Maintained By: PhySciCH@mail.usyd.edu.au
Last Update: Monday, 20-Jun-2005 16:30:07 EST
URL: http://science.uniserve.edu.au/pubs/callab/vol3/mazzo.html