CAL-laborate Volume 3 October 1999

Assessment: the Key to Changing the Way We Learn

Abstract

Studying at virtual universities has recently become more widespread. A critical issue yet to be fully addressed is how to deal with assessments and examinations on-line. In conventional education students often focus only on material that is examinable. Too much of the student's attention is given to passing an examination rather than developing a deeper understanding of the subject.

Another problem is the organization of on-line examinations when students are spread throughout the world and hence control may not be guaranteed.

The challenges of tomorrow

High bandwidth communication systems will soon be available to every student anywhere in the world. Hand-held devices and other gadgets are already offered at affordable prices. Satellites in low orbits, as well as communication technologies such as Ericsson's Bluetooth (short range wireless connectivity), are rapidly changing the way people interact with each other. A huge mass of information is available to students everywhere. However, there will be no gatekeepers - everyone is able to publish anything. Therefore, we believe that it is the art of critical thinking that will be the most important skill to master in the future. Students also have the right to get an answer to the question "What's in it for me?".

To cultivate critical thinking and increase students' responsibility for their own learning we should provide them with opportunities to practise these skills.

They should work in groups with other students and solve problems together. This is done by making the examinations into Problem Based Learning (PBL).

They should contribute to and give value judgments that are taken seriously. This is done by not setting a fixed answer to any given question. All questions should have a range of possible answers. Open-ended questions could stimulate.

They should "create their own courses" that fit their special needs - education on demand. When the goals set in these courses are achieved that could constitute the examination of the course.

They should be allowed to criticize their teacher.

For the students the challenges of tomorrow are:

to constantly learn when they like, where they like and at their own pace;

to find and take the very best courses on-line;

to be able to demand to learn the bits and pieces of use and interest to them and nothing more (or less); and

to have very small fees for these courses due to the huge competition between virtual universities on-line.

What's in it for the professor?

teaching improvements;

increased communication with students (email allows you to "talk" with many more students than in a regular class);

the challenge of having students from all over the world;

sharing the work to prepare teaching material with others and using on-line resources saves time;

easier collaboration and sharing resources with colleagues all over the world;

less tedious work;

easier administration of students and their records;

automated assessment;

easier tracking of individual students;

new job opportunities as a teacher at other virtual universities;

increased royalty from your published work, as many more will know about it (we anticipate royalties even on papers published on the Internet); and

more time for contact with students.

For the virtual university the challenges are to:

educate many more students most of whom will be lifelong learners;

supply higher quality courses;

focus more on student's individual needs; and

budget a much lower cost.

The technology and economic pressure will force universities to compete and cooperate. A student will not take a course because it is close to home, but rather because it gives the best value. Students will be free to choose from courses offered at hundreds of virtual universities and educational businesses on-line.

We assume a development similar to when the handicrafts turned into industrialized processes. Thus we think of virtual universities having 10 000 students in a course. How can that be handled?

What do the employers want of the university? If you look at various advertisements for new personnel you find certain skills listed that often fall outside what students can show in their degrees. We think it is essential that the students of tomorrow can master these skills and therefore suggest "skills for life courses" like:

how to write a report;

how to present on posters and overheads;

how to get along with the press;

how to give a speech;

how to study, focusing on the student's personal responsibility;

how to cooperate - teamwork; and

how to manage your time.

How to determine the quality of learning

One popular way to describe a student's level of learning is given by Bloom's Taxonomy (1956). Back in 1972 one of us used Bloom's Taxonomy to describe what Swedish students were asked to do in assessments in Physics during 1972/73. All Swedish examinations were evaluated to determine what level of knowledge was required to solve a problem. It was found that most of the questions only required the lowest level - knowledge, and only rarely did questions require the higher levels - analysis, synthesis or evaluation. The authors were surprised to find that the number of questions at the knowledge level increased in the more advanced courses of Physics. When the authors met again last summer they concluded that the situation had not improved in the 25 years that have passed.

Bloom's Taxonomy for cognitive skills

Knowledge - the student remembers facts;

Comprehension - the student understands relations and context;

Application - the student can apply his knowledge to new areas;

Analysis - the student can analyze and find the parts;

Synthesis - the student can create something of his own that is relatively unique; and

Evaluation - the student can give value judgements based on facts in his field.

SOLO taxonomy for learning

Biggs and Collis (1982) designed another tool for evaluating students' responses: SOLO - Structure of the Observed Learning Outcome.

Prestructural - irrelevant response;

Unistructural - the use of one obvious piece of given data;

Multistructural - the sequential use of one obvious piece of given data;

Relational - the integration of the given data to form a unique conclusion or generalization; and

Extended abstract - the use of multiple interacting abstract systems to form a response. This may include forming a general hypothesis, assessing the quality of models and accepting open-ended answers.

What kind of examinations do we need tomorrow?

Virtual universities need a new way of assessment since it is the factor that most influences learning. Students often only focus on those aspects that are likely to appear in examinations. "Is this examinable?" Forget about "interesting". The question from most students is "Will we be tested on this?". Traditional examinations often only address the lower levels of cognitive skill in the taxonomy proposed by Bloom. To focus the student's learning and address all aspects of the taxonomy we must develop new forms of assessment. We believe that this is a most important challenge. Success with assessment will change the way students learn.

We would like to shift from assessment as a scary and intimidating experience to assessment as a learning tool. In this transition it is important to ask the questions: "What do we assess?"; "What do employers ask for?"; and "Can the assessment be used as a tool to find a match between student and employer needs?". In a rapidly changing world we must answer these questions ourselves: "What kind of assessment do we really need in the 3rd millennium?".

We think that the portfolio concept will therefore be of central importance: the student will have to be able to convince future employers that he is the right person for the job. Employers will find it increasingly difficult to evaluate grades on a piece of paper from various universities all over the world. Essays and work on-line will enable the student to easily direct an employer to a portfolio of his/her credentials.

Today we require students to take a course from the beginning to the end independent of their previous knowledge. We are working on ideas of placement tests that enable a student to enter a course at an appropriate level or take only the parts that he actually needs.

A proposed example of a new type of examination

To stimulate the discussion we have constructed a test with SAFT - Self Assessed Free Text and an assessment of open-ended questions without the need of a marker - a tool that could reduce costs and increase quality. The test is part of a "skills course" on writing an essay.

Initially the student answers a question in essay form. He is then guided by the test through a series of questions and exercises that aim to further develop his insight into the subject of the test. As feedback to his answer the student receives the teacher's recommended answer to the question. This recommended answer is then used as a basis for further insight from the student. He may for instance be asked to give possible improvements to the combined student-teacher solution. He might be prompted to mark all the sections of the recommended answer that he himself covered in his own answer. At certain points of the test the student might be able to read other students' answers to a follow-up question to inspire his own thinking. These other answers make up a knowledge bank from which students may obtain ideas and insights. In fact the student's own answer to the same follow-up question is in turn added to the bank of information for other students' possible gain. The student will also give a value judgement on his own accomplishment that will be included in the final grade of the test.

Advantages with SAFT

The ideas put forth in the SAFT test have, in our opinion, the same advantages that traditional examinations have today. In creating new opportunities for the student to get involved in his/her examination and giving him/her a chance to express an opinion that is taken seriously we get a whole new approach to viewing the examination process. The vision is clear: We want to be able to use SAFT for formative examinations that are the basis for which grade the student will actually receive in any academic course given at a virtual university. Whether or not this will be possible only the future can tell. Using SAFT for this purpose will mean, however, that we will have to change our entire outlook on how we perceive written examinations. The student can have an interesting and valuable opinion on the subject matter of a course that should be taken very seriously by the teacher. The student's given answer to the test is not right or wrong. It's how he/she draws conclusions and how he/she applies his/her knowledge, analyses the problem, creates his/her own interpretation of the subject and gives meaningful value judgements of the subject that really matters, all in accordance with Bloom's Taxonomy.

It will take some time before SAFT is fully developed and is appreciated and widely used for examinations. To learn more about SAFT and to bypass the problem with identifying the student, this novel approach could be used as a self-study aid. A SAFT test is constructed to give immediate feedback to the student after he/she completes the test or even while the test goes on. One of the purposes of the SAFT test is to make the examination a learning experience and not solely an examination. Imagine that we have a SAFT test with yet another purpose: to give feedback in the form of study references. The idea works almost the same as an ordinary SAFT test except that the purpose of the test is not to set a grade for the student. Instead the test aims to allow the student to undertake a practise test in which the feedback is given in the form of literature references to material covering concepts and understanding that the student has yet to acquire.

An important part of the techniques is the knowledge banks that all students may contribute to and everyone may obtain ideas from. The knowledge banks make the course grow and improve. Other advantages include:

for the student:

an opportunity to apply his knowledge to new areas;

help to analyze the given situation and find the parts of the problem;

motivation to create something of his own, help to move his thoughts in new directions; and

an opportunity to think more about a given value judgement on his knowledge as well as his own opinions and from this draw conclusions on the nature of a correct answer.

and for the teacher:

help to guide the student towards higher levels of Bloom's Taxonomy;

valuable input from students that may be incorporated into and help develop the course given;

students take responsibility of their own examinations - no teacher correcting will be necessary;

no need to gather all students at once in a big room and perform a written test; and

reduced costs for examinations.

Other tools developed at CITU

EVA - On-line evaluation (http://bengt2.citu.lu.se/eval/)

EVA is a web-based evaluation tool for editing questions, then publishing the evaluation and its results on the web. This tool is of great use for quick and easy evaluations. One idea could be to use EVA in class after you have met the students three times. Evaluate what the students think about the course, publish the result and have a discussion on how you can improve things. EVA contains over 600 questionnaires with over 12 000 questions. More than 350 000 answers have been posted to the database up till now.

ITcampus (http://itcampus.org/)

A resource by students, for students has been operating for three years at CITU. The work of summer '99 will soon start at CITU and will aim to include even more student-related material than is available now.

ITcampus includes lecture notes and old examinations (questions and answers) making it absolutely necessary for the teachers to update their lectures and examinations every semester or they will quickly be outdated!

Many resources at ITcampus are collected to develop the quality of education. There are plans for a news section and a course section that will include searchable, updated and accurate information on all courses given at Lund University, virtual as well as physical.

ITcampus is the 37th most visited site in Sweden and it is mainly concerned with issues that involve Lund University.

ITcampus has grown beyond the student world of Lund University to include a cooperation with Aalborg University in Denmark and may still grow further to include other universities and languages.

EGO (http://ego.itcampus.org/)

In an attempt to make the web student-centred we are launching a student portal to the web, EGO for the students at Lund University this semester (fall 1999). While creating several useful web recourses including ITcampus and web email accounts the need for a collective site that gathers all these exciting initiatives in a 'one click away' fashion has become apparent.

The student's account in EGO is registered automatically at the same time he/she is registered at any department of the university. The very first time the student logs onto his/her EGOpage, the page is by default filled with material related to his/her course. He/she can find and contact course mates, course related material, links to on-line bookstores that offer the cheapest literature for the courses that he/she is registered on, a link to the student's web email account and much more.

KNUT (http://knut.kks.se/)

Knowledge Network for Education in Schools.

LUVIT, Lund University Virtual Interactive Tool (http://www.luvit.com/)

LUVIT is an Internet-based system for interactive education and information management, designed to help teachers create and publish courses on the web, and to guide students to learn and communicate, and to record student progress.

Continue thinking

If you want to get some insight into what our ideas could signify in your situation please do the following exercise.
On your next test or assessment, ask your students to bring and use their own telephone and computer!

How will that change the questions?

How will it change the way students learn?

References

Bloom, B. S. (Ed.). (1959) Taxonomy of educational objectives: The classification of educational goals, Handbook 1: Cognitive domain. New York: David McKay Company.
On-line links to Bloom's Taxonomy:
http://www.stedwards.edu/cte/bloomtax.htm
http://illinois.online.uillinois.edu/model/bloomtaxonomy.htm
Briggs, J. B. and Collis, K. F. (1982) Evaluating the Quality of Learning: The SOLO Taxonomy. Academic Press, New York.
Grimvall, G. and Kjöllerström, B. (1974) Evaluation of all Swedish exams in physics during 1972/73. UKA report 1974:22c.

Bengt Kjöllerström
CITU
Lund University
Box 118
S-221 00 Lund
Sweden
bengt.kjollerstrom@citu.lu.se
and
Magnus Mårtensson
CITU
Lund University
Box 118
S-221 00 Lund
Sweden

© 1997 - 2008 UniServe Connections

Page Maintained By: PhySciCH@mail.usyd.edu.au
Last Update: Monday, 20-Jun-2005 16:30:07 EST
URL: http://science.uniserve.edu.au/pubs/callab/vol3/kjoller.html