eAssess - Automating Large Group Examination Management and MarkingCraig Burton
The Online Assessment Company
In 1997 Craig Burton attended the UniServe Science workshop at The University of Sydney and demonstrated a software package called NEST, which ran high-risk online assessment tasks at The University of Melbourne. Craig has since left The University of Melbourne to start his own company, The Online Assessment Company, and has created a new assessment system, eAssess. The Online Assessment Company has also produced two related products for KPMG Consulting Australia; an online voting system eLect, and an online survey system eQuest, which are available with eAssess.
eAssess functions as a Web site which allows educators (or other 'authors' who will design tests) to securely login and use the friendly authoring system to build examinations and quizzes. The students (or other system 'users' who come to attend tests) can then login to their own accounts and attempt assessment material prepared for them. Authors can login from anywhere and also work offline.
Figure 1. How an answer is described : eAssess will attempt to match user submissions to Dromidary (the word, exactly). If this matches, they get 5 marks. If not, the system falls back on the next rule, * romidary (where there can be any letters before 'romidary'. Also provided is the ability to hunt for long patterns with any number of matching rules.
The most important function of eAssess is the running of large-group assessment of students in a controlled environment. Together with measures for anti-collusion, secure login access, and a high-performance serving system, eAssess can serve hundreds of simultaneous requests and submissions in a performance-critical examination environment; more than 3 million examination submissions a day on a small Intel server.
Many institutions now enjoy some form of automation of their curriculum delivery and to a certain extent, assessment. However, after benefiting from the greater accessibility and interactive nature of online resources, most students sit their examinations on paper. The cost of running examinations in this manner can be as high as $AUD20 per student per examination with half of this cost being marking.
Final and term examinations are run on paper for security reasons; there has to be tangible evidence of a student's attendance, and the right student has to be identified with the right examination paper. Cheating has to be prevented and examination papers have to be handled securely. However, all of these important aspects of examining students can be automated. While important examinations should still be done in a controlled situation, the formal nature of examination questions and answers lends itself well to interactive computer programs which can serve the same questions and collect many of the same kinds of responses (such as numeric, short answer and selection-type answers).
Access to the examination can be via two-step identification of the student and issue of login information. The examination questions can be served out of a pool of questions, reordered and randomised to prevent collusion. The most tangible benefits are in the automated marking available for many answer types and the ability to schedule and run re-sits of an examination.
eAssess runs on a LAN or extranet to an examination facility or computer laboratory where students are identified on arrival (somewhat like a polling station) and are then issued a computer and login criteria. eAssess can run many examinations at once, and serve the right examination to the right student. The examination access is timed (though students can submit after the close time and are logged out).
The student can later be issued with access to an examination report which shows how all marks were issued by the system and how their part-marks were provided. The system gives an example of an answer which would have won the student full marks. The system will also show manual marks issued by markers for questions which were configured for manual assessment.
Quizzes, surveys, simple tasks
An important step in migrating students to eAssess is the introduction of low-risk tasks which reinforce the utility of the system and build trust. Old eAssess examinations can be easily served as quizzes, for example. The two principal formats are "exams" as I have covered above (where the student receives little or no feedback on their submission during the session) and "quizzes" where the students are told their marks (but not the answers) on submit.
The quiz format allows the author to provide a pool of quiz content with students making multiple attempts at a set of questions, reaching a hurdle (say, 8/10), then requesting a new group of questions.
The marking rules for either format of online assessment allow the author to anticipate various partially incorrect responses. This approach becomes more and more appropriate as the author uses more complex answer types such as short answer, multi-response and long answer types. In these cases, specific patterns of choices or entered words attract marks and penalties for the student. Authors can provide unlimited numbers of rules and eAssess will attempt to award the best mark to the student.
With quizzes, you can choose to view and mark the student's first attempt to the quiz. This is their first attempt at each question. As they have the ability to make multiple attempts, their mark will improve (albeit with brute force), so you may want to mark them on their first go. The submissions tool in eAssess allows you to see the 'first', 'last' or 'all' submissions to the quiz. This gives the author the ability to track the progress of the student through numerous attempts to several quizzes, and to see if they are deriving their marks by brute force, guessing etc.
Figure 2. Report: The author gets 1 out of 5 marks for choosing b, c and d. This matches a rule where b and c are worth a mark each, but the choice of d is penalised by a mark. The highest-value rule is shown as an example of 'full marks'.
The questions in an examination and a quiz can be made up of straight text, text and images, video and audio for the student to use and respond to. Use of these multimedia types is greatly facilitated by eAssess's online multimedia library.
The majority of the work in creating eAssess has gone into making the system friendly and easy to use for the author. There are two ways to author with it: the first is via a simple step-by-step 'wizard' which guides the author through creating questions, using images (or sound, video, etc.), setting up security and the like. The Wizard runs in a browser and so the author can log in from anywhere and create the tasks. For complete control, a second system is a 'command-driven' system (eScript) where the author creates entire examinations or large banks of material with a set of commands. This allows the author to convert legacy testing systems or to write their content offline in a word processor. The author can then upload this as a file and continue working online. Assessment tasks made with the Wizard can be converted to the eScript format for customisations.
Figure 3. The eScript editor with some of the commands visible. eAssess uses a compiler to build the examination and then it runs on its own to serve students. Other buttons in the Editor include a command reference and a syntax checker. eScript enables authors a powerful mechanism for creating examinations with any content and structure.
eScript gives access to complex layout and interactions such as hierarchical assessments, hint-driven assessment, dynamic content serving and other functions.
Whether you create the task via the Wizard or via eScript, you then have the ability to share this with other authors who can contribute their own questions and collaborate. The system handles its assessment tasks in a manner similar to the life cycle of a paper document - there are stages of Draft, Live, Reporting as well as a facility to Archive. It's up to the author to define when the task is ready to become accessible to the students.
Once the assessment task becomes Live, then the users organised for the task can gain access. At this time, you can start reviewing their submissions via a reporting and submissions browsing tool online. This allows you to see how marking was done by the system. eAssess allows you to add new rules and change rules in the case where students might identify an unanticipated answer to a question which is not wrong!
eAssess is distributed by The Online Assessment Company, which is based in Melbourne. The creators of the package invite educators and readers of UniServe Science to contact firstname.lastname@example.org and obtain a trial account on the system.
UniServe Science News Volume 18 October 2001
Page Maintained By: PhySciCH@mail.usyd.edu.au