Computer Based Assessment - The Chemistry QuizSteve Walker
CTI Centre for Chemistry, University of Liverpool, UK
This series of programs has been developed at Liverpool over the last three to four years in response to two convergent needs - insistence on a minimum numeric competency from students and a reduction in the amount of staff time involved in assessment. There is evidence that both criteria are being satisfied.
Chemistry Departments have been struggling for some time now with the reluctance of students to carry out even simple mathematical exercises. Physical chemists have borne the brunt of this problem but matters came to a head in Liverpool a few years ago when inorganic chemists realised that a substantial minority could not work out their titrations. The staff agreed that this constituted a skill that all competent professional chemists must acquire and so instituted a test which students had to pass to proceed further in their course. This resulted in the test being re-taken as often as necessary until a pass mark was obtained. The amount of work required of staff in setting and marking as many tests as were required can be imagined.
At the same time, staff in the computer assisted learning facility were turning their attention to the question of computer based assessment and a quiz on volumetric calculations seemed to provide the ideal vehicle for a pilot study. The objectives were:
The major requirement of this program was that, since the test would have to be taken on several different occasions students must be presented with different calculations to prevent prior knowledge conferring an advantage. The CTI Centre provided details of all the available packages, but despite the excellence of some of them, they all failed this criterion. In a typical example, a numeric question and its answer must be pre-programmed with no provision to calculate the answer from the question nor to use random input. This would entail 100 separate numeric inputs for each question together with the corresponding answers to give a different value for each student - clearly time consuming and probably not worthwhile. Accordingly we wrote our own.
The Chemistry Quiz (CQ) is a program (now a series of programs) used to test students' competence with the everyday numeric manipulations required in chemistry and is designed to complement the existing range of software available. CQ allows the generation of random input data so that every student receives a different value even when attempting the same question. CQ is designed only for this type of question and makes no attempt to incorporate textual input, multiple choice or other features found in other products. All computations use integer arithmetic to avoid rounding errors, with sensible use of significant figures. Quizzes have been produced covering volumetric calculations, spectroscopy, thermochemistry, particles and waves, and algebraic manipulation.
CQ may be set up for student use in one of three different modes:
1) Practice mode
In this mode the student is presented with a total of 20 questions chosen at random from all the available question types for a particular quiz. The total of 20 has been chosen to minimise time and resources and to provide a reasonable length of study. In this mode a student may opt to exit at any time or, indeed, to repeat the quiz an unlimited number of times. The student also has the ability to choose a particular question type at any time - useful for tutorial homework etc. This mode is the default.
2) Directed Practice mode
This is similar to mode 1) with the exception that the available question types (and their relative frequency) are specified by the lecturer in a file (setup.dat) placed in the same directory as the quiz. This is useful if some of the questions are not covered in a particular syllabus, the class requires extensive practice in a particular method, or the quiz needs to be modified during a course as material is progressively put in front of students. Students are unaware, in practice, of the differences between modes 1) and 2).
3) Examination mode
In this mode, the number of questions, their relative frequency and order are completely specified by the lecturer in the file 'setup.dat'. This file also contains information required of students, such as passwords, and contains pointers to the location of students' answer files. In complete contrast to modes 1) and 2), students cannot exit until they have seen all the questions stipulated (although not necessarily answered!) and when they finish their answers are recorded in '.csv' format for compatibility with spreadsheets.
CQ comprises three parts: Quizmaker, which is used by the lecturer only to set up a test, a series of Quizzes for the students to take, and Analyser, which marks the answers.
The Quizmaker program is installed on the lecturer's own computer. It consists of a number of pages which specify the precise quiz format such as the mode, quiz subject, number of questions, question frequency, type of question, time limit, location of the answer files, and so on. It generates the 'setup.dat' file.
When a quiz file is opened, the program looks for the file 'setup.dat' in its own directory. If it fails to find it then it will proceed in practice mode. At all times the student has the option to look at the correct answer. If the set-up file is found (and examination mode was specified) then the student still has the choice of continuing to practice or opting to sit the examination.
In examination mode the student must answer the quiz in the style defined by the lecturer. A particular question can be skipped by clicking on "Next Question". After all the required questions have been presented, the student has the option of reviewing the answers and changing them if desired. This can be repeated as often as required until the student is satisfied. At this point the student should click on "Exit" and the answers will be appended to the data file. If a time limit has been specified then the quiz begins with a reminder to the student that this is in operation. The time remaining is displayed below the question number in minutes. When five minutes are left this display changes colour to red and with one minute to go a message is placed on the screen.
The volumetric quiz
The students have access at all times to a calculator (with the ability to copy and paste into the answer box) and a table of appropriate relative molecular masses (so as not to introduce errors from this calculation). This contains the following question types:
Figure 1. The screen as seen by the student in practice mode with the computed answers displayed
The volumetric quiz was examined using the following specifications:
The students were given a 'window' of two weeks in which to take the test. They were able to do this at any time within this period and there was no attempt to check for collaboration, impersonation, or cheating, nor was any other traditional exam supervision imposed. The examination window was scheduled for a period when all students had had time to learn the computer's operation and become confident in the mechanics.
The marking program (Analyser)
A quite separate marking program has been developed which completely automates the marking process and will, if required, generate a final print out suitable for the notice board. This is optional since the answer format is compatible with all spreadsheets and a custom template can be constructed using your preferred program. Analyser allows the accuracy levels required and the mark awarded to be set for each question. The major difference between computer marking and paper marking is that in the former it's an 'all or nothing' process with full marks for a correct answer and zero for one outside the prescribed accuracy limits. A human marker will give partial marks - for example, for writing a correctly balanced equation. Computer marking is thus likely to give lower marks.
The last year for taking the traditional paper test was 1995. This used exactly the same question types as those specified above for the computer tests and was designed to be taken in a lecture slot of 45 minutes. The results for 1995, contrasted with results for the following two years, are shown in Figure 2.
Without making any attempts to analyse these data, the implications are clear. The major reason for the improvement is practice. In the allowed fortnight, students spent the first week practising and only attempted the test when they felt confident. The end result, whatever the means, is that the class can now carry out their volumetric calculations with confidence. Conversely, since there is no evident improvement in students' basic mathematical skills (from tutorials in the remedial mathematics course), it could be that they are simply being trained to carry out these types of calculation.
The time taken to prepare, invigilate, and mark an examination is now 10 minutes compared to approximately 20 hours for the series of paper-based tests. Needless to say, this is extremely popular with staff. Of course, it ignores the time required to write the programs and staff time devoted to running the computer network.
Students are generally supportive although it's difficult to extract anything sensible when asking their opinion of examinations. They like to practice, they like the two-week window and they like to be trusted. The only genuine negative note concerns their inability to indicate their thought processes and to add textual comment to their answers (a bonus as far as staff are concerned!). It may surprise some that they made no attempt to work together (there was, they said, insufficient time for chat), nor to personate nor, indeed, made any attempt to cheat in this completely unsupervised examination. For 1997 (but not 1996) the computer classroom has been fitted with a remote surveillance camera for security reasons and spot checks confirmed that serious individual work was carried out.
Figure 2. Examination results for three successive years: 1995 when traditional paper tests were given, and 1996/7 when computer tests were given.
CAL-laborate Volume 2 October 1998
Page Maintained By: PhySciCH@mail.usyd.edu.au