CAL-laborate Volume 2 October 1998










*******

Computer Based Assessment - The Chemistry Quiz

Steve Walker
CTI Centre for Chemistry, University of Liverpool, UK

Introduction

This series of programs has been developed at Liverpool over the last three to four years in response to two convergent needs - insistence on a minimum numeric competency from students and a reduction in the amount of staff time involved in assessment. There is evidence that both criteria are being satisfied.

Chemistry Departments have been struggling for some time now with the reluctance of students to carry out even simple mathematical exercises. Physical chemists have borne the brunt of this problem but matters came to a head in Liverpool a few years ago when inorganic chemists realised that a substantial minority could not work out their titrations. The staff agreed that this constituted a skill that all competent professional chemists must acquire and so instituted a test which students had to pass to proceed further in their course. This resulted in the test being re-taken as often as necessary until a pass mark was obtained. The amount of work required of staff in setting and marking as many tests as were required can be imagined.

At the same time, staff in the computer assisted learning facility were turning their attention to the question of computer based assessment and a quiz on volumetric calculations seemed to provide the ideal vehicle for a pilot study. The objectives were:

  1. to test the feasibility of examining a large class of students (ca. 100) using the in-house computer network of 15 machines. This would deal with such matters as timetabling, reliability, security etc.;
  2. to assess the acceptability of such tests from the student point of view;
  3. to assess the acceptability from the staff point of view;
  4. to assess the educational effectiveness; and
  5. to quantify any resultant time savings.

The program

The major requirement of this program was that, since the test would have to be taken on several different occasions students must be presented with different calculations to prevent prior knowledge conferring an advantage. The CTI Centre provided details of all the available packages, but despite the excellence of some of them, they all failed this criterion. In a typical example, a numeric question and its answer must be pre-programmed with no provision to calculate the answer from the question nor to use random input. This would entail 100 separate numeric inputs for each question together with the corresponding answers to give a different value for each student - clearly time consuming and probably not worthwhile. Accordingly we wrote our own.

The Chemistry Quiz (CQ) is a program (now a series of programs) used to test students' competence with the everyday numeric manipulations required in chemistry and is designed to complement the existing range of software available. CQ allows the generation of random input data so that every student receives a different value even when attempting the same question. CQ is designed only for this type of question and makes no attempt to incorporate textual input, multiple choice or other features found in other products. All computations use integer arithmetic to avoid rounding errors, with sensible use of significant figures. Quizzes have been produced covering volumetric calculations, spectroscopy, thermochemistry, particles and waves, and algebraic manipulation.

Modes

CQ may be set up for student use in one of three different modes:

1) Practice mode

In this mode the student is presented with a total of 20 questions chosen at random from all the available question types for a particular quiz. The total of 20 has been chosen to minimise time and resources and to provide a reasonable length of study. In this mode a student may opt to exit at any time or, indeed, to repeat the quiz an unlimited number of times. The student also has the ability to choose a particular question type at any time - useful for tutorial homework etc. This mode is the default.

2) Directed Practice mode

This is similar to mode 1) with the exception that the available question types (and their relative frequency) are specified by the lecturer in a file (setup.dat) placed in the same directory as the quiz. This is useful if some of the questions are not covered in a particular syllabus, the class requires extensive practice in a particular method, or the quiz needs to be modified during a course as material is progressively put in front of students. Students are unaware, in practice, of the differences between modes 1) and 2).

3) Examination mode

In this mode, the number of questions, their relative frequency and order are completely specified by the lecturer in the file 'setup.dat'. This file also contains information required of students, such as passwords, and contains pointers to the location of students' answer files. In complete contrast to modes 1) and 2), students cannot exit until they have seen all the questions stipulated (although not necessarily answered!) and when they finish their answers are recorded in '.csv' format for compatibility with spreadsheets.

CQ comprises three parts: Quizmaker, which is used by the lecturer only to set up a test, a series of Quizzes for the students to take, and Analyser, which marks the answers.

Quizmaker

The Quizmaker program is installed on the lecturer's own computer. It consists of a number of pages which specify the precise quiz format such as the mode, quiz subject, number of questions, question frequency, type of question, time limit, location of the answer files, and so on. It generates the 'setup.dat' file.

Quizzes

When a quiz file is opened, the program looks for the file 'setup.dat' in its own directory. If it fails to find it then it will proceed in practice mode. At all times the student has the option to look at the correct answer. If the set-up file is found (and examination mode was specified) then the student still has the choice of continuing to practice or opting to sit the examination.

In examination mode the student must answer the quiz in the style defined by the lecturer. A particular question can be skipped by clicking on "Next Question". After all the required questions have been presented, the student has the option of reviewing the answers and changing them if desired. This can be repeated as often as required until the student is satisfied. At this point the student should click on "Exit" and the answers will be appended to the data file. If a time limit has been specified then the quiz begins with a reminder to the student that this is in operation. The time remaining is displayed below the question number in minutes. When five minutes are left this display changes colour to red and with one minute to go a message is placed on the screen.

The volumetric quiz

The students have access at all times to a calculator (with the ability to copy and paste into the answer box) and a table of appropriate relative molecular masses (so as not to introduce errors from this calculation). This contains the following question types:

Type 1. Concentration-1. A random weight (from 0.2 to 2 g) of a typical volumetric reagent, chosen at random from a list of 10, is dissolved in water (from 50 cm3 to 1000 cm3). One question is asked: find the molarity of the reagent. Type 2. Concentration-2. Similar to Type 1. Two questions are asked. A) Find the molarity of the reagent. B) Calculate the concentration of one of the elements in the reagent.
Type 3. Concentration-3. Similar to type 1. Three questions are asked. A) Find the molarity of the species. B) Calculate the volume of solution required producing a molarity of exactly 0.10000, 0.01000, 0.00100 etc. as appropriate. C) As for question B, but with a randomly chosen molarity. Type 4. Acid/Base-1. Randomly chooses either the standardisation of sodium hydroxide by potassium hydrogen phthalate or of hydrochloric acid by potassium hydrogen carbonate. Unknown molarities are in the range 0.08 to 0.13 with the unknown solution being titrated against random weights of the standard substance. One question is asked: find the molarity of the unknown.
Type 5. Acid/Base-2. Similar to type 4. Three questions are asked. A) Calculate the molarity of the unknown solution. B) Repeats the calculation for a different weight of standard. C) Asks for the weight of standard required to produce a titre of exactly 25 cm3. Type 6. Permanganate-1. The standardisation of unknown permanganate solutions by sodium oxalate. Molarities are in the range 0.018 to 0.024. One question is asked: find the molarity of the permanganate.
Type 7. Permanganate-2. As type 6. Three questions are asked. A) Find the molarity of the permanganate. B) Repeats the calculation for a different weight of oxalate. C) Asks for the weight of oxalate required to produce a titre of exactly 25 cm3. Type 8. Dichromate. The standardization of potassium dichromate with either iron(II) sulfate or iron(II) ammonium sulfate. Molarities are in the range 0.014 to 0.019. One question is asked: find the molarity of the dichromate.
Type 9. Thiosulfate. The standardisation of sodium thiosulfate with potassium iodate using starch indicator. Molarities are in the range 0.08 to 0.13. One question is asked: find the molarity of the thiosulfate. Type 10. Chromium Ore. The chromium is oxidised to dichromate and reacted with an excess of iron(II) sulfate (25 cm3 of ca. 0.4 M). The remaining iron is back titrated with permanganate. One question is asked: find the weight percentage of chromium (in the range 8 to 13%).
Type 11. Calcium Ore. The calcium (25 to 40%) is precipitated as oxalate, dissolved in acid and titrated with permanganate. One question is asked: find the weight percentage of calcium. Type 12. Argentimetric. Titration of a halide (randomly chloride, bromide, or iodide) against silver nitrate using dichlorofluoroscein. Molarities are in the range 0.08 to 0.13. One question is asked: calculate the percentage of halogen in the unknown.


Figure 1.

Figure 1. The screen as seen by the student in practice mode with the computed answers displayed



Examination implementation

The volumetric quiz was examined using the following specifications:

  1. six questions, one each of types 1, 3, 5, 7, 10, 11 (see above);
  2. order of questions: strictly as set down in 1;
  3. time limit: 45 minutes; and
  4. no passwords required.

The students were given a 'window' of two weeks in which to take the test. They were able to do this at any time within this period and there was no attempt to check for collaboration, impersonation, or cheating, nor was any other traditional exam supervision imposed. The examination window was scheduled for a period when all students had had time to learn the computer's operation and become confident in the mechanics.

The marking program (Analyser)

A quite separate marking program has been developed which completely automates the marking process and will, if required, generate a final print out suitable for the notice board. This is optional since the answer format is compatible with all spreadsheets and a custom template can be constructed using your preferred program. Analyser allows the accuracy levels required and the mark awarded to be set for each question. The major difference between computer marking and paper marking is that in the former it's an 'all or nothing' process with full marks for a correct answer and zero for one outside the prescribed accuracy limits. A human marker will give partial marks - for example, for writing a correctly balanced equation. Computer marking is thus likely to give lower marks.

Results

The last year for taking the traditional paper test was 1995. This used exactly the same question types as those specified above for the computer tests and was designed to be taken in a lecture slot of 45 minutes. The results for 1995, contrasted with results for the following two years, are shown in Figure 2.

Without making any attempts to analyse these data, the implications are clear. The major reason for the improvement is practice. In the allowed fortnight, students spent the first week practising and only attempted the test when they felt confident. The end result, whatever the means, is that the class can now carry out their volumetric calculations with confidence. Conversely, since there is no evident improvement in students' basic mathematical skills (from tutorials in the remedial mathematics course), it could be that they are simply being trained to carry out these types of calculation.

The time taken to prepare, invigilate, and mark an examination is now 10 minutes compared to approximately 20 hours for the series of paper-based tests. Needless to say, this is extremely popular with staff. Of course, it ignores the time required to write the programs and staff time devoted to running the computer network.

Students are generally supportive although it's difficult to extract anything sensible when asking their opinion of examinations. They like to practice, they like the two-week window and they like to be trusted. The only genuine negative note concerns their inability to indicate their thought processes and to add textual comment to their answers (a bonus as far as staff are concerned!). It may surprise some that they made no attempt to work together (there was, they said, insufficient time for chat), nor to personate nor, indeed, made any attempt to cheat in this completely unsupervised examination. For 1997 (but not 1996) the computer classroom has been fitted with a remote surveillance camera for security reasons and spot checks confirmed that serious individual work was carried out.

Paper1995 First test81 Candidates27 Fail (3 absent)Mean mark 52%
1995 First resit26 Candidates16 Fail (1 absent)Mean mark 50%
1995 Second resit16 Candidates2 FailMean mark 55%
Computer1996 First test101 Candidates11 Fail (3 absent)Mean mark 72%
1996 First resit10 Candidates1 Fail (withdrawn)Mean mark 65%
1997 First test91 Candidates7 Fail (3 absent)Mean mark 73%
1997 First resit1 Candidate6 Fail (absent)

Figure 2. Examination results for three successive years: 1995 when traditional paper tests were given, and 1996/7 when computer tests were given.

Steve Walker
CTI Centre for Chemistry
University of Liverpool
Donnan Laboratories
Liverpool
L69 7ZD
United Kingdom
Ctichem@liverpool.ac.uk
http://www.liv.ac.uk/ctichem.html


Return to Contents

CAL-laborate Volume 2 October 1998

[an error occurred while processing this directive]

Page Maintained By: PhySciCH@mail.usyd.edu.au
Last Update: Monday, 30-Apr-2012 15:15:20 AEST
URL: http://science.uniserve.edu.au/pubs/callab/vol2/walker.html