A vital part of the development of any Computer Aided Learning (CAL) package is evaluation of the package during the development of the package. This formative evaluation of a package will feed back into the development process at numerous stages of development. This is critical as much time, effort, and money may be wasted by developing a package that, for any of a thousand reasons, may be unsuitable. If evaluation of materials occurs during the development process then the production of a useful package is more likely. This process of evaluation leading to revision is known as formative evaluation.
Formative evaluation is also known by the names of `developmental testing', `student tryout', and `learner verifications and revision' (Russell and Blake 1988). It is this last point that is critical in the definition of formative evaluation - its purpose is evaluation that will dictate further revisions of material being prepared and thus is undertaken during the development of CAL materials. Formative evaluation is often done by the developers of a package, though if materials are being developed as part of a consortium then formative evaluation by other consortium members or a central group may be appropriate (e.g. Laurillard, 1994a and 1994b).
There are a number of ways to undertake formative evaluation - these can be divided into informal and formal means. Informal formative evaluation takes place in the development space often through asking the opinion of colleagues. For example asking another developer for their opinion of a particular part of a CAL package, or challenging them to find bugs in a CAL package, and getting subject matter experts to review the content. This type of formative evaluation takes place throughout the development of instructional materials and will be repeated many times. Through feedback the design and content of materials will be modified. At a slightly later stage in development (when the obvious technical `bugs' have been ironed out), but still well from the end of the developmental stage, then more formal formative evaluation will take place, that is at a `rough draft' phase of development.
Formal formative evaluation may include having small groups of potential users use the instructional materials in a situation much like those in which the materials will finally be used. These users should be typical of the target user, that is not exceptionally able or challenged students, unless these groups are the target audience. The users should be observed while they work through the CAL package. Often the best feedback from the users can be gained if the users are in pairs and they discuss their perceptions of the package, its navigation, and its content as they work through it. An alternative to having an observer is to videotape users `think aloud' as they work their way through the instructional material. The use of a camera rather than an observer may lead to users being less self-conscious, leaves a record that can be reviewed many times, and allows the users to work out for themselves navigation rather than asking the observer for help. A negative aspect is that reviewing videotape can be extremely time consuming.
The purpose of formative evaluation is to get feedback so that further development of the package can occur and the package can be the best instructional material possible. This will mean that from the results of student/user comments any technical or content problems found will be corrected and the package will be modified to reflect the users' comments. Once these modifications and revisions have occurred, then another round of formal formative evaluation may take place.
However, the instructional materials being prepared must eventually be released. Thus, although formative evaluation is vital and should go through a number of cycles before the CAL package is released, the time frame and budget of a project will determine when the CAL package will be released. It must be accepted that the first release of any instructional materials is unlikely to be perfect (although there should be no content errors or major technical bugs) but that there must be an end point to the development of the first release version Getting the balance between adequate formative evaluation or getting trapped in an endless spiral from which the product (or you!) may never emerge is a constant danger of which one needs to constantly be aware.
Laurillard, D. (1994a) Evaluation Planning, Technology Enhanced Language Learning (TELL), CTI for Modern Languages, University of Hull, UK.
Laurillard, D. (1994b) Evaluation Procedures, Technology Enhanced Language Learning (TELL), CTI for Modern Languages, University of Hull, UK.
Russell, JD and Blake BL (1988) Formative and summative evaluation of instructional products and learners. Educational Technology, Sept. 1988: 22-28.
Evaluation Sites of Interest
Teaching with Independent Learning Technologies
Evaluation of Learning Technology in Higher Education
Table of Contents