Workshop Presentation

Proceedings of Evaluating the New Teaching Technologies Workshop, April 28, 2000

Why am I evaluating this thingummygig?

Introduction

Why do I need to evaluate the new technologies? In my undergraduate days I doubt if anyone evaluated their teaching and my learning was inspite of their teaching! The long slow haul to change the teaching and learning culture has put excessive pressure on some forms of learning experiences and new technologies is one of them. This makes sense when you consider the amount of money we have been investing in developing computer based learning materials. Ever since we, in First Year Biology, started to develop teaching and learning materials using information technologies we have endeavoured to understand how these materials are being used and what, if anything, do the students gain from using such materials. Early on we did a lot of usability studies and checks on accuracy of content (formative evaluations) and so improved the materials. We also developed expertise in instructional design. With this in place we concentrated more on the impact of the materials on student perceptions (Did they like using them? Did they help them in their understanding?) Whilst this is also a type of formative evaluation it gave us some ideas on how students were using the materials. The big issue, however, is 'Do the materials have an effect on student learning outcomes?' such that one would argue they are better than other forms of learning material? This is difficult to answer without fairly exhaustive studies on the use of the computer based learning materials by students. By the time the formative evaluation stage is over, there is often little time available to ask these questions; we are too busy; there are too many students to cope with; etc.

In a recent CUTSD-funded study, Shirley Alexander reviewed 104 teaching development projects and reported that in approximately 90% of cases the project leaders indicated that they had the intention of improving student learning outcomes, but only a third could report this as an actual outcome as only this third actually evaluated student learning outcomes (Alexander, 1999; Alexander and McKenzie, 1998). Alexander goes on to argue that most of the project evaluations fell within the first level of the four levels of outcome on which evaluation evidence should focus, as described by Kirkpatrick (1994), that is, 'reaction to the innovation' and a minority of evaluations fell within the second level 'achievement of learning objectives'. Only one project fitted the third level 'transfer of new skills to the job or task' and no project evaluated the 'impact on the organisation', (Alexander, 1999). It would seem that we all need to lift our performance in this area and ensure that at least levels one and two are fulfilled.

Categories of evaluation

Much has been written on the methodology of evaluation and on evaluation studies themselves, (more recent examples include Flagg, 1990; Kulik and Kulik, 1991; Reeves, 1991; Laurillard, 1993; Draper et al., 1994; Learning Technology Dissemination Initiative (LTDI), 1998). For future evaluation studies I propose using a modification of a model from the Learning Technology Dissemination Initiative (1998), funded by the Scottish Higher Education Funding Council, in which four phases of evaluation are considered (Table 1). These are formative, summative, illuminative and integrative.

In this model summative evaluation is used for external product review, the equivalent of a book review and this is done after the courseware has been finalised. A schematic plan of the relationship of these phases is presented below.

Figure 1. A schematic plan of the relationship of these phases

I believe this model helps to set up a workable methodology for validating expectations of any courseware and how it fits into the overall course or part of a course (integrative evaluation) and in revealing important factors which we have no preconceived ideas about (illuminative evaluation). Illuminative evaluation uses anecdotal reports, observations, interviews and open-ended questionnaires. Integrative evaluation requires analysis of all course materials and their assessment, along with investigating how students use all materials.

Current project

Currently, in First Year Biology, we have an evaluation project to look at the way computer based learning modules have been integrated into a human biology course, which is taken by 800 first year students in second semester. This evaluation is part of an ASCILITE-CUTSD project 'Staff Development in Evaluation of Technology-Based Teaching Development Projects'. The project uses an action inquiry and mentoring approach in which participants will be helped by their mentor to set up appropriate evaluation. The human biology course consists of lectures, laboratory sessions, and independent study modules. The software modules used in this course, 'Nervous System', 'Reproductive System', 'Digestive System', 'Cardiovascular System', and 'Structure and Function of the Ear', have been developed over a number of years. The materials have been produced to replace animal cadavers in university undergraduate classes in line with the current climate on these issues. One of the modules was funded by the New Educational Aids in Medicine and Science (NEAMS) Trust which funds one project each year in Australia for the development of teaching materials to replace animal cadavers. Each module is written as an interactive exploration of the content and with a quiz section for students to self-test their understanding of the content and concepts. All of the modules, along with other teaching materials, are available on the Web in the First Year Biology Virtual Learning Environment. (http://fybio.bio.usyd.edu.au/vle/L1/ResourceCentre/). Students can thus have access to the materials during the formal teaching time and at any other time they choose.

Formative evaluation of the software modules has already been done including external peer review. The module 'Structure and Function of the Ear' was a finalist in the ASCILITE 1998 Awards. The current project will concentrate on trying to answer the question 'Is the inclusion of software modules within the curriculum an effective way to teach human biology?'. Evaluation will involve looking at the overall curriculum, its content and assessment, finding out how the students are using the materials, identifying if the intended learning outcomes are met, and identifying if the innovation is educationally appropriate in the unit. These are summarised in Table 2.

The information gained from the process outlined in Table 2 will be used to answer the question 'Is the inclusion of software modules within the curriculum an effective way to teach human biology?' More importantly, 'Do the materials have an effect on student learning outcomes?'

References

Alexander, S. (1999) An evaluation of innovative projects involving communication and information technology in higher education, Higher Education Research & Development, 18(2), 173-183.
Alexander, S. and McKenzie, J. (1998) An Evaluation of Information Technology Projects for University Learning, Canberra: Australian Government Publishing Services (AGPS).
Draper, S., Brown, M. I., Edgerton, E., Henderson, F. P., McAleer, E., Smith, E. D. and Watt, H. D. (1994) Observing and measuring the performance of educational technology, University of Glasgow Technical Report.
Flagg, B. (1990) Formative Evaluation for Educational Technologies. L. Erlbaum Associates.
Harvey, J. (ed.) (1998) Evaluation Cookbook. Learning Technology Dissemination Initiative, Institute for Computer Based Learning, Edinburgh: Heriot-Watt University. http://www.icbl.hw.ac.uk/ltdi/ltdi-pub.htm
Kirkpatrick, D. L. (1994) Evaluating training programs: The four levels. San Francisco: Berrett-Koehler.
Kulik, C. C. and Kulik, J. A. (1991) Effectiveness of computer-based instruction: An updated analysis, Computers in Human Behaviour, 7, 75-94. Laurillard, D. (1993) Rethinking university teaching: A framework for the effective use of educational technology. London: Routledge.
Reeves, T. (1991) Ten commandments for the evaluation of interactive multimedia in higher education, Journal of Computing in Higher Education, 2, 84-113.

© 1997 - 2009 UniServe Connections

Page Maintained By: PhySciCH@mail.usyd.edu.au
Last Update: Thursday, 12-Oct-2000 11:21:16 EST
URL: http://science.uniserve.edu.au/pubs/procs/wshop5/peat.html