Assessment: the Key to Changing the Way We LearnBengt Kjöllerström
CITU, Lund University, Sweden
Studying at virtual universities has recently become more widespread. A critical issue yet to be fully addressed is how to deal with assessments and examinations on-line. In conventional education students often focus only on material that is examinable. Too much of the student's attention is given to passing an examination rather than developing a deeper understanding of the subject.
Another problem is the organization of on-line examinations when students are spread throughout the world and hence control may not be guaranteed.
The challenges of tomorrow
High bandwidth communication systems will soon be available to every student anywhere in the world. Hand-held devices and other gadgets are already offered at affordable prices. Satellites in low orbits, as well as communication technologies such as Ericsson's Bluetooth (short range wireless connectivity), are rapidly changing the way people interact with each other. A huge mass of information is available to students everywhere. However, there will be no gatekeepers - everyone is able to publish anything. Therefore, we believe that it is the art of critical thinking that will be the most important skill to master in the future. Students also have the right to get an answer to the question "What's in it for me?".
To cultivate critical thinking and increase students' responsibility for their own learning we should provide them with opportunities to practise these skills.
For the students the challenges of tomorrow are:
What's in it for the professor?
For the virtual university the challenges are to:
The technology and economic pressure will force universities to compete and cooperate. A student will not take a course because it is close to home, but rather because it gives the best value. Students will be free to choose from courses offered at hundreds of virtual universities and educational businesses on-line.
We assume a development similar to when the handicrafts turned into industrialized processes. Thus we think of virtual universities having 10 000 students in a course. How can that be handled?
What do the employers want of the university? If you look at various advertisements for new personnel you find certain skills listed that often fall outside what students can show in their degrees. We think it is essential that the students of tomorrow can master these skills and therefore suggest "skills for life courses" like:
How to determine the quality of learning
One popular way to describe a student's level of learning is given by Bloom's Taxonomy (1956). Back in 1972 one of us used Bloom's Taxonomy to describe what Swedish students were asked to do in assessments in Physics during 1972/73. All Swedish examinations were evaluated to determine what level of knowledge was required to solve a problem. It was found that most of the questions only required the lowest level - knowledge, and only rarely did questions require the higher levels - analysis, synthesis or evaluation. The authors were surprised to find that the number of questions at the knowledge level increased in the more advanced courses of Physics. When the authors met again last summer they concluded that the situation had not improved in the 25 years that have passed.
Bloom's Taxonomy for cognitive skills
SOLO taxonomy for learning
Biggs and Collis (1982) designed another tool for evaluating students' responses: SOLO - Structure of the Observed Learning Outcome.
What kind of examinations do we need tomorrow?
Virtual universities need a new way of assessment since it is the factor that most influences learning. Students often only focus on those aspects that are likely to appear in examinations. "Is this examinable?" Forget about "interesting". The question from most students is "Will we be tested on this?". Traditional examinations often only address the lower levels of cognitive skill in the taxonomy proposed by Bloom. To focus the student's learning and address all aspects of the taxonomy we must develop new forms of assessment. We believe that this is a most important challenge. Success with assessment will change the way students learn.
We would like to shift from assessment as a scary and intimidating experience to assessment as a learning tool. In this transition it is important to ask the questions: "What do we assess?"; "What do employers ask for?"; and "Can the assessment be used as a tool to find a match between student and employer needs?". In a rapidly changing world we must answer these questions ourselves: "What kind of assessment do we really need in the 3rd millennium?".
We think that the portfolio concept will therefore be of central importance: the student will have to be able to convince future employers that he is the right person for the job. Employers will find it increasingly difficult to evaluate grades on a piece of paper from various universities all over the world. Essays and work on-line will enable the student to easily direct an employer to a portfolio of his/her credentials.
Today we require students to take a course from the beginning to the end independent of their previous knowledge. We are working on ideas of placement tests that enable a student to enter a course at an appropriate level or take only the parts that he actually needs.
A proposed example of a new type of examination
To stimulate the discussion we have constructed a test with SAFT - Self Assessed Free Text and an assessment of open-ended questions without the need of a marker - a tool that could reduce costs and increase quality. The test is part of a "skills course" on writing an essay.
Initially the student answers a question in essay form. He is then guided by the test through a series of questions and exercises that aim to further develop his insight into the subject of the test. As feedback to his answer the student receives the teacher's recommended answer to the question. This recommended answer is then used as a basis for further insight from the student. He may for instance be asked to give possible improvements to the combined student-teacher solution. He might be prompted to mark all the sections of the recommended answer that he himself covered in his own answer. At certain points of the test the student might be able to read other students' answers to a follow-up question to inspire his own thinking. These other answers make up a knowledge bank from which students may obtain ideas and insights. In fact the student's own answer to the same follow-up question is in turn added to the bank of information for other students' possible gain. The student will also give a value judgement on his own accomplishment that will be included in the final grade of the test.
Advantages with SAFT
The ideas put forth in the SAFT test have, in our opinion, the same advantages that traditional examinations have today. In creating new opportunities for the student to get involved in his/her examination and giving him/her a chance to express an opinion that is taken seriously we get a whole new approach to viewing the examination process. The vision is clear: We want to be able to use SAFT for formative examinations that are the basis for which grade the student will actually receive in any academic course given at a virtual university. Whether or not this will be possible only the future can tell. Using SAFT for this purpose will mean, however, that we will have to change our entire outlook on how we perceive written examinations. The student can have an interesting and valuable opinion on the subject matter of a course that should be taken very seriously by the teacher. The student's given answer to the test is not right or wrong. It's how he/she draws conclusions and how he/she applies his/her knowledge, analyses the problem, creates his/her own interpretation of the subject and gives meaningful value judgements of the subject that really matters, all in accordance with Bloom's Taxonomy.
It will take some time before SAFT is fully developed and is appreciated and widely used for examinations. To learn more about SAFT and to bypass the problem with identifying the student, this novel approach could be used as a self-study aid. A SAFT test is constructed to give immediate feedback to the student after he/she completes the test or even while the test goes on. One of the purposes of the SAFT test is to make the examination a learning experience and not solely an examination. Imagine that we have a SAFT test with yet another purpose: to give feedback in the form of study references. The idea works almost the same as an ordinary SAFT test except that the purpose of the test is not to set a grade for the student. Instead the test aims to allow the student to undertake a practise test in which the feedback is given in the form of literature references to material covering concepts and understanding that the student has yet to acquire.
An important part of the techniques is the knowledge banks that all students may contribute to and everyone may obtain ideas from. The knowledge banks make the course grow and improve. Other advantages include:
for the student:
and for the teacher:
Other tools developed at CITU
EVA - On-line evaluation (http://bengt2.citu.lu.se/eval/)
EVA is a web-based evaluation tool for editing questions, then publishing the evaluation and its results on the web. This tool is of great use for quick and easy evaluations. One idea could be to use EVA in class after you have met the students three times. Evaluate what the students think about the course, publish the result and have a discussion on how you can improve things. EVA contains over 600 questionnaires with over 12 000 questions. More than 350 000 answers have been posted to the database up till now.
A resource by students, for students has been operating for three years at CITU. The work of summer '99 will soon start at CITU and will aim to include even more student-related material than is available now.
ITcampus includes lecture notes and old examinations (questions and answers) making it absolutely necessary for the teachers to update their lectures and examinations every semester or they will quickly be outdated!
Many resources at ITcampus are collected to develop the quality of education. There are plans for a news section and a course section that will include searchable, updated and accurate information on all courses given at Lund University, virtual as well as physical.
ITcampus is the 37th most visited site in Sweden and it is mainly concerned with issues that involve Lund University.
ITcampus has grown beyond the student world of Lund University to include a cooperation with Aalborg University in Denmark and may still grow further to include other universities and languages.
In an attempt to make the web student-centred we are launching a student portal to the web, EGO for the students at Lund University this semester (fall 1999). While creating several useful web recourses including ITcampus and web email accounts the need for a collective site that gathers all these exciting initiatives in a 'one click away' fashion has become apparent.
The student's account in EGO is registered automatically at the same time he/she is registered at any department of the university. The very first time the student logs onto his/her EGOpage, the page is by default filled with material related to his/her course. He/she can find and contact course mates, course related material, links to on-line bookstores that offer the cheapest literature for the courses that he/she is registered on, a link to the student's web email account and much more.
Knowledge Network for Education in Schools.
LUVIT, Lund University Virtual Interactive Tool (http://www.luvit.com/)
LUVIT is an Internet-based system for interactive education and information management, designed to help teachers create and publish courses on the web, and to guide students to learn and communicate, and to record student progress.
If you want to get some insight into what our ideas could signify in your situation please do the following exercise.
Bloom, B. S. (Ed.). (1959) Taxonomy of educational objectives: The classification of educational goals, Handbook 1: Cognitive domain. New York: David McKay Company.
CAL-laborate Volume 3 October 1999
Page Maintained By: PhySciCH@mail.usyd.edu.au