Workshop Presentation

Proceedings of Evaluating the New Teaching Technologies Workshop, April 28, 2000

Doing it differently in science: An evaluation of the process

Project background: the learning context

With the benefit of a CUTSD Grant for 1999, the authors sought to develop an integrated programme supporting the development of key learning competencies in undergraduate science students at the University of Western Sydney (UWS) Nepean. The subject chosen to contextualise this programme, was a second year biological sciences subject, Immunology, which in itself aimed to provide students with an understanding of the development and functioning of the immune system, as well as expertise in a range of clinical assessment and research techniques involving immunological principles. The targeted competencies included:

group leadership and membership;

oral and written communication;

critical analysis;

problem solving;

reflective skills; and

independent learning.

We recognised that students do not 'pick up' such competencies as they learn (Collins, Brown and Newman, 1989; Lave, 1988), and so they were specifically taught and practiced throughout the semester, in multiple different contexts. The delivery of the subject was through cooperative learning approach, in order to both assist students with their learning, and to help simulate the collaborative environment common in the science workplace (Johnson, Johnson and Smith, 1998).

To achieve our objectives, the essentials of the knowledge base were delivered external to the allocated lecture periods, workshops replaced lectures and the information normally presented in a lecture format was available via audio tapes, textbook, a study guide and web site. During workshops, students worked cooperatively in teams undertaking a number of different tasks designed to develop the aforementioned competencies as well as explore the knowledge domain. For example, one key activity was to critically analyse three immunology research papers, and subsequently present their analysis to their peers for critical feedback. Teams also worked cooperatively to develop practical solutions to a chosen laboratory problem, and subsequently in their laboratory-based investigation of that problem during an intensive weekend laboratory session. Support for student learning was achieved via direct access to both academic and technical staff as well as on-line through the communication software included in the School of Science Virtual Resource Centre web site (http://cadfl.nepean.uws.edu.au/science/vrc/ - password required), this enabled students to contact each other, and their lecturer/tutor at any time.

Mid-semester and final examinations were retained, though they contained less direct recall material than in previous years. Additional assessment tasks, designed to measure the acquisition of specified generic or professional competencies, were spread throughout the semester and allocated significant marks (see Table 1).

What was there to evaluate?

When any new programme is introduced to students, it is essential that it is thoroughly evaluated, from both their perspective and that of the staff involved in its delivery. In this case, as the programme was developed with the support of CUTSD funds, formal evaluation was an integrated and integral part of its development and integration. Aspects of the subject that required evaluation included: the programme - its design and delivery; the learning environment - including the lecturer/tutor and support services; and the learning outcomes - the clarity of intent and success in achieving them. Each of these was used to make decisions about the programme's future.

Evaluating the evaluation - the person and the process

Many traditional evaluation processes were incorporated into the programme during its design and development, these included the use of:

summative assessment outcomes - including the team collage and poster;

student questionnaires;

follow-up student focus groups;

staff reflection - documented;

staff focus groups - both academic and technical;

video footage - of the weekend laboratory sessions and subsequent student presentations; and

observation - of the cooperative learning process.

During the course of the semester it became apparent that an effective evaluation of the subject required both a process of evaluation, outlined above, and a person whose role it was to undertake/oversee it. It is the role of the latter in this process that has caught our interest.

Initially our CUTSD application promised the use of Observation methods (Guba and Lincoln, 1981) and the presence of an 'Independent Observer', whose task it would be to monitor aspects of cooperative learning during both workshop and laboratory sessions. However, in our project, this person became more of an 'Independent Facilitator' and was ultimately pivotal to both the evaluation and implementation of the project. This facilitator was characterized as independent of both the teaching and support staff who delivered the subject, but being well known to them, a free flowing, two-way communication was established. Though assisting in the development of the subject learning assessment tasks, she was independent of any assessment of these tasks. This resulted in the students communicating thoughts and feelings to her that may not have been possible with the subject lecturer/tutor, and allowed for the sharing of insights that may normally have remained unsaid. This is perceived to have enhanced the depth of subject evaluation that was possible. The facilitator was thus characterized as independent of any vested interests, personal, political or philosophical, allowing for a more dispassionate viewing of the learning environment both during its delivery and after its completion. However, although she was independent of the people and process in these ways, she most certainly had an holistic involvement in the implementation of the subject, therefore could not be called an observer, hence our coining of the term: 'Independent Facilitator'.

We conclude that it can be valuable having such a person involved in the delivery of a new subject, as s/he brings a different viewpoint and focus to the programme, and most importantly, has the time to facilitate its effective delivery. In order to function in a manner conducive to the implementation and evaluation of an effective learning programme, the 'Independent Facilitator' should have the following characteristics:

content knowledge;

an understanding of the learning process;

interpersonal skills;

a flexibility of approach;

an ability to reflect, analyse and solve problems;

an ability to identify needs as they arise; and

the sensitivity not to impose his/her ideas and opinions on others, or to make value judgements.

Evaluation: informing the future

Both the 'Independent Facilitator' and the process of evaluation, provided invaluable feedback about the achievement of the learning goals, and the popularity and viability of the new teaching and learning strategy. The combined feedback, from all evaluation procedures, suggests that the programme has been well received by students, both in the goals we were trying to achieve and the way the learning experience was structured. It was relatively successful in enabling students to develop the learning competencies outlined previously, as determined by both student perceptions of their own skills, and as observed by the staff involved in the delivery of the programme. Though cooperative learning requires a more active engagement by students in the learning process than normal, often proving more difficult in practice than in theory, both staff and the majority of students (79%) recognized the importance of grappling with learning/working in this manner. Any changes that will be made to the delivery will be minor, addressing weaknesses in programme timing and resource management. This will include:

more structured guidance and feedback for the written communication tasks, including more explicit summative assessment of these skills; and

an emphasis on teamwork and oral communication presentation will be retained and an analysis of how such an emphasis affects non-English speaking background (NESB) students will be evaluated.

One staff member believes that we may have sacrificed some content expertise in our desire to develop professional competencies, however she also believes that this cohort of students '...will make great 3rd years, due to their ability to solve problems'. Only time, experimentation and further evaluation, will tell whether we have the balance right. The future of the programme may be less rosy however. Both the subject lecturer and 'Independent Facilitator' have either left the School of Science or are in the process of leaving, with few remaining staff with an interest in either the philosophy of the methodology, or in the types of learning outcomes being addressed, namely generic or professional competencies of science. As with a number of such initiatives, their success and continuation hangs off one or two pivotal members of staff and unless they become institutionalized in the academic unit, they do not continue. In this instance, the programme's evaluation indicates that it has been successful in achieving many of its objectives, and hence such evaluation outcomes can be used as an incentive to those who follow to adopt its recommendations and assist in the continued evolution of the programme as both the student cohorts and learning context change.

References

Collins, A., Brown, J. S. and Newman, S. E. (1989) Cognitive apprenticeship: Teaching the craft of reading, writing and mathematics, in Knowing learning and instructions: Essays in honor of Robert Glaser, ed. L. B. Resnick, 453-494, Hillsdale, NJ: Erlbaum.
Guba, E. G. and Lincoln, Y. S. (1981) Effective evaluation. San Francisco, CA: Jossey-Bass.
Johnson, D. W., Johnson, R. T. and Smith, K. A. (1998) Active learning: cooperation in the college classroom, Edina, MN: Interaction Book Co.
Lave, J. (1988) Cognition in Practice. Boston, MA: Cambridge.

© 1997 - 2009 UniServe Connections

Page Maintained By: PhySciCH@mail.usyd.edu.au
Last Update: Thursday, 12-Oct-2000 12:24:18 EST
URL: http://science.uniserve.edu.au/pubs/procs/wshop5/fraser.html