The introduction of web-based formative assessment in medical science: An action research projectRobert B. Dow
Medical Sciences Section, School of Life Sciences, Queensland University of Technology, GPO Box 2434, Brisbane, Qld 4001, Australia
A brief review of the literature would suggest that formative assessment (feedback) is an important determinant of student performance (Ramsden 1992). Buchanan (1998) has described formative assessment as procedures, which are designed to give students appropriate feedback in a form, which they may use to guide their future efforts. This is distinct from summative assessment, which is designed to formally rank students, e.g. end of semester examinations. Haematological Techniques 1 (LSA 322) is part of the course program for the Associate Degree in Medical Laboratory Techniques at the Queensland University of Technology. It is a subject that has been traditionally assessed by only final practical and theory examinations, with a fail rate of about 10% (Forster 1999). Comments from previous student evaluations in this subject have reflected favourably on some of the feedback that is presently undertaken for the students, principally 'spot tests' conducted during practicals (QUT 1998). The introduction of some formative assessment, which addresses the theory, will complement present practice and perhaps improve learning outcomes for the students.
Callear and King (1997) have indicated the potential for computerised tests in formative assessments. An important advantage of this style of formative assessment is that feedback is immediate and can be tied specifically to performance. It also allows the students to progress at their own pace and in their own environment, at times suitable to them. The challenge therefore is how to effectively introduce the web-based formative assessment so the best learning outcomes are achieved for the students.
The action research paradigm is one where there is the simultaneous achievement of action (that is, change) and research (that is, understanding) (Dick 1997). It has many forms. Conventional research paradigms can actually inhibit effective change. Action research tends to be cyclic, and similar steps recur in a sequence. It is also participative, the researcher and the researched are involved as partners or as active participants in the research. Action research is more often concerned with qualitative data such as language, comments, and points of view rather than quantitative forms. A key part of the action research process is critical reflection. Critical reflection on the process and outcomes are essential intrinsic components of the cycle. Carr and Kemmis (1986) have described these steps as being: plan -> act -> observe -> reflect -> plan, and so on. The cyclic nature and the critical reflection allow for learning and responsiveness as each stage is progressed. For each step there is planning before action and critical analysis after it and the understanding gained in preceding cycle or cycles modifies each cycle. It is the responsiveness of action research that allows it to be used very effectively for evaluation and pilot research as in this project.
Discussion was held with colleagues as to the suitability of web-based formative assessments in this subject. Discussions were also held with the students to outline the proposed project and to enlist their participation and support. Feedback indicated their strong support and willingness to be involved in the project. The following strategies were then proposed and developed.
A search was made for suitable software to allow for the progressive development and introduction of a small number (four) of quizzes, i.e. four cycles. A suitable freeware package was located, viz. Hot Potatoes, and was used to develop the first quiz (Half Baked Software 1999). This software package was also used to develop the subsequent quizzes. A suitable web site was developed (Dow 1999).
The introduction of the quizzes would be monitored by a variety of feedback strategies. Interaction with the students by the use of focus groups, and questionnaires and feedback from colleagues formed the initial strategies. This feedback would be sought at suitable intervals after the introduction of each quiz, allowing a period for students to access the programs. A web-based feedback process would also be made available.
The implementation of the four computer based formative assessments was planned for weeks 4, 6, 8, 10 of Semester 1, 1999. Each cycle would be monitored by a variety of feedback mechanisms, and use the information in action research cycles (plan, implement, assess, reflect, plan). That is, information obtained in the first cycle would be reflected on and used to modify, as required, the quiz for the second cycle, and so on.
The first quiz was made available via the Internet. Reference was made to the literature when developing the actual questions (Isaacs 1994). An initial fifteen questions were developed and included in quiz one. All participants were emailed, and where appropriate briefed in class that the first quiz was available.
After one week, initial feedback was obtained by use of an open questionnaire. The questions asked were: Please comment on good points, bad points and other comments. Fourteen out of a possible thirty-one students returned the questionnaire. Some students indicated that they had access problems due to system problems and overcrowding. Alternative access to the system was then arranged immediately to allow further access to quiz one and in subsequent cycles. A group discussion was also held. Comments from both the questionnaire and discussion indicated that the quiz had been favourably received. Additional feedback on the quiz was obtained from peers (academics) who again indicated positive support. Many of the participants asked when the next one would be available. During the reflective process it became evident that more care would need to be paid to matching the question content with curriculum progress. It was also observed that there was a real cohesive spirit developing among the group. This was not expected initially and is almost paradoxical when considering that at one level this is very much an individual action, i.e. do the quiz, but couched in the action research/critical reflection paradigm is the participation of the individual in the project, which had the effect of building a more collegiate spirit amongst the group.
Two weeks after the first quiz was implemented the second quiz was developed and made available. Critical reflections from cycle one feedback guided the further development of quiz two. Increasing complexity was introduced into the questions in an attempt to promote more understanding and learning. It was also determined that an attempt to ascertain the group's perceptions of the effect of the formative assessment on their learning would be undertaken in cycles three and four. Re-enforcement of the quiz availability was done in lectures and by email.
A facilitated discussion was held to obtain feedback on the second cycle. Eighty-five percent of the students had now accessed the quizzes. This was much improved from cycle one and there was a strong feeling from the students that the project was valuable for helping them learn. Typical comments were: 'the second quiz was much better than the first'; 'the quizzes help us to review what we have done, and learn to understand'; 'the quizzes are good because the feedback tells me to look in the practical manual'; and 'can more questions and feedback be included'. During the discussion it also surfaced that the student cohort had a particular dislike of the use of online technology as a complete substitute for the teaching process, but appreciated that use of this approach as introducing an element of choice, and as a way to help improve learning outcomes rather than as a total substitute. Peer and self-motivation was observed to be increasing. Feedback was also obtained from some practising medical scientists. This independent feedback confirmed that the questions were also relevant to the work environment.
The third quiz was made available in week seven, rather than in week eight as originally planned. This was in response to the demands of the participants themselves. More feedback was included in the questions, although question number was not increased in an effort to keep a balance between deeper structure and time to complete the quiz. Despite not having covered some of the material in lectures there was a real desire to 'have a go'. This was an unexpected, but pleasing outcome. It was also determined amongst the participants that the quizzes would not stop at four, but carry on for another cycle in the period prior to the examination period.
An online feedback mechanism was introduced prior to the implementation of quiz four. A number of individual interviews were conducted with the students to obtain a further option on how the quiz was helping their understanding. All the students interviewed stated that they felt the quizzes were particularly effective in identifying areas that they understood and the areas that they still had to learn properly. It was also at this time that students started to discuss particular questions at the beginning of the formal lecture sessions. This provided further opportunities to promote understanding. They also felt that the structure and depth of questions was also appropriate. Students had been accessing each quiz about four times. The major suggestion made during the interviews was that even more feedback as part of the answer response would be desirable. This was taken up and implemented in quiz four.
The fourth quiz was placed on the Web during week nine. This quiz was more focussed than the previous quizzes, and incorporated additional feedback as part of the answer response. The major topic covered by this quiz was a particularly challenging one, haemostasis. Some technical issues were also addressed by upgrading the appearance of the web pages, and providing an uncluttered page structure.
A major evaluation of the fourth cycle, and the project at this stage, was undertaken by videotaping a group discussion. At this stage the students demonstrated that they were very much involved with the project and were willing to cooperate in a taped discussion. This tape provided a powerful statement of the value of the project. All the students had by this time accessed the quizzes, and articulated the view that their understanding of the subject had been helped by the project, and in fact requested that a further quiz be made available. Some actual comments were: 'very beneficial in guiding areas not quite sure on'; 'I think the quizzes highlight areas that I think I know, but don't actually know'; 'concentrate on areas not sure about'; and 'questions are good, but could photographs of cells be included'. These comments certainly indicate that the quizzes were guiding the students' learning, the fundamental aim of the formative assessment. Particularly appreciated was the flexible nature of the program and that the students could access the quizzes when it suited them. The students also indicated that the specific feedback provided by the quiz responses was highly valued as it helped to guide their further study in the subject. A Student Evaluation of Unit (SEU) was also conducted to obtain further comment on the project. An improvement for future quizzes would be to incorporate some images.
Discussion and further reflections
Action research is a process by which change and understanding can be pursued at the one time (Dick 1997). At the beginning of this project, the technical aspects relating to establishing a web presence were considered to be a valid part of the action research process. In fact, as the project advanced the technical aspects began to be seen more as merely a tool to allow the action research process to occur. Each cycle provided an opportunity to immediately apply what was learnt from that cycle and so rapid progress was made in enhancing the quizzes, hence the learning outcomes for the group. At the commencement of the project the author was a little skeptical of the action research paradigm. A reflective journal was also started with some apprehension, but it was soon discovered that it was useful to write down immediate responses, observations and feelings before they were forgotten. It was really as a result of the interaction with, and by, the students, and observing the positive impact of the project on the group, that an appreciation of the true value of the methodology became apparent. Action research is a powerful process because of its cyclic and participatory nature, and thus allowing for evaluation, reflection and monitoring of what is actually occurring, and modifying actions based on these observations and reflections. Changes can then be implemented and immediately assessed in the context of the project. This project has led to a demonstration of the effectiveness of web-based formative assessment in this discipline and improved teaching practice. It has also demonstrated that action research is an effective research paradigm for academics researching their own teaching practice in medical science.
This report describes the first four cycles of the project. The project is in fact continuing into a fifth cycle as it has helped to guide the students' learning. This was not part of the initial plan but this has occurred as a result of action research. Innovations are not fixed in action research as they are in a case study framework. For the fifth quiz the number of questions has been increased (from 15 to 26), and more specific feedback provided for the answer responses. In addition, one question has incorporated an image. Future quizzes will have more images. This action was the result of further evaluation and reflection during cycle four.
This project has laid the groundwork for future applications of action research in seeking improvements in teaching practice in medical science. An important benefit is the increased awareness of what is actually happening in the classroom and the mechanism it provides for a valid methodology for inquiring into problems of students' learning generally. Action research has enabled a significant change to be made in a controlled validated manner, which has led to improved practice.
I would like to acknowledge the contribution made by the students of LSA322 Haematological Techniques 1 Semester 1, 1999, to the successful outcome of this project. Thanks also to my colleagues, particularly those undertaking the Graduate Certificate of Education, for their constructive advice, and to Dr Sally Clarke for review of the draft manuscript.
CAL-laborate Volume 10 June 2003
Page Maintained By: firstname.lastname@example.org