Summary Paper

Proceedings of Evaluating the New Teaching Technologies Workshop, April 28, 2000

Evaluating the New Technologies: A student learning focused perspective

[In June 2000, Mike was appointed Director of the Institute for Teaching and Learning at The University of Sydney.]

Introduction

There has been a growing impact of the new technologies on the processes and outcomes of teaching and learning. The developments are occurring faster than they can be properly evaluated.

Much of the evaluation that is being conducted has been from the teacher's perspective, focussing on:

learning gains by students on tests produced by teachers; and

improvements in the productivity of teaching and learning (Alexander and McKenzie, 1998).

There has been little research or evaluation focussing on the students' experiences of using the technology - a student rather than teacher focused perspective. For example, questions such as the following are rarely addressed in the evaluation reports I have seen:

how do the students experience the new technologies?

what do they think are the aims?

what do they believe they are learning?

how do they approach the use of such technologies? and

very importantly, how do they see the relationship between the new technologies and other aspects of teaching and learning?

This is surprising, given the impact of the research and evaluation in teaching and learning in higher education from a student learning perspective (Marton, Hounsell and Entwistle, 1997; Ramsden, 1992; Prosser and Trigwell, 1999). It is from this perspective that the major Australian teaching and learning benchmarking instrument - the Course Experience Questionnaire (CEQ) - was developed and is being used (Ramsden, 1991).

One such evaluation of a number of on-line learning packages at La Trobe University recently concluded that there was less variation between packages from a student learning perspective than variation within packages (McShane, 2000). The issue was what was done with the packages by the teachers rather than differences between packages.

This paper will address the issue of evaluation of the new technologies from a student learning perspective. In doing so, it will outline the characteristics of the evaluative research being conducted in university science education from this student learning perspective, show examples of research and evaluation from that perspective, and outline some strategies for the future evaluation of the new technologies in teaching and learning. In passing it will summarise the theory and research underlying the development of the CEQ.

Student learning in higher education

Over twenty years of research in teaching and learning in higher education has shown that student learning outcomes - examination results, concept maps, open-ended responses, etc. - are closely related to how students experience their studies. Figure 1 summarises the result of much of this research in terms of an heuristic model of student learning in higher education. It shows that student learning outcomes are closely related to how they say they approach their studies. How they approach their studies is, in turn, related to how they perceive and understand the teaching and learning context. How they perceive and understand that context is in turn, related to their prior experiences of teaching and learning and to the context itself. The key issue is, however, that students perceive the same context in different ways. These different ways are systematically related to how they approach their studies and to the quality and quantity of their learning outcomes (Prosser and Trigwell, 1999).

Figure 1. Model of student learning

In terms of the evaluation of new technologies in teaching and learning, the model would suggest that student learning outcomes from the use of the new technologies would relate to how the students approached their studies with the new technologies. This would depend, for example, on how they saw the aims of the new technologies in their learning - not how well the new technologies met the teacher's aims, how they saw the use of the new technologies relating to their perceptions of what was to be rewarded in the assessment, how they experienced their workload associated with using the new technologies, etc. These perceptions would depend on how well the new technologies were designed and integrated into the subject and course structure, what the aims were for the use of the new technologies, and importantly what the student's prior experiences were of using similar technologies.

In the remainder of this section I wish to use the results of some of the previous research in university science education to examine this model of student learning before returning explicitly to the evaluation of new technologies.

The research has identified two fundamentally different ways in which students approach their studies in higher education. These are the so-called surface and deep approaches to study. A surface approach is characterised by an intention to reproduce what is being learned for assessment purposes. There is little intention to try to understand that material. On the other hand, a deep approach is characterised by an intention to understand the material being studied. This research suggests that it is the student's intention, rather than the observed or reported strategy, which is important in terms of the quality of student learning outcomes. A given strategy can have a different meaning depending on the intention.

Strategies associated with a surface approach include:

rote memorisation of information needed for assessment;

failure to distinguish principles from examples;

treating tasks as external impositions; and

focussing on discrete elements without integration.

Strategies associated with a deep approach include:

vigorous interaction with content;

relating new ideas to previous knowledge;

relating concepts to everyday experiences; and

relating evidence to conclusions.

What needs to be noted here, especially in the context of science education, is that memorisation can be associated with either a surface or a deep approach. It depends on the intention the student has when engaged in memorisation.

In terms of the use of the new technologies, the quality of learning outcomes depends on the student's intentions when they come to use the new technologies. For example, whether they approach the new technologies with the intention to learn and understand, or whether they approach it with the intention of just completing the task.

The research has identified that the key perceptions of context related to these approaches to study are perceptions of:

1. quality of teaching;
2. clearness and nature of the goals;
3. nature of assessment;
4. heaviness of the workload; and
5. amount of independence in learning.

It needs to be emphasised that it is students' perceptions of these aspects of the teaching and learning context that are important, and that different students in the same context will form different perceptions of that context. These are the scales of the Course Experience Questionnaire, the questionnaire being used to survey all graduates of Australian universities about their experiences of studying.

In the remainder of this section, I intend to draw upon the results of recent research in university science education to illustrate the relations referred to previously.

As part of a recent study involving over 1500 first year university science students in Australia, some colleagues and I were interested in looking at the relations between students' perceptions of their teaching and learning context and their approach to study. We used a modified version of John Biggs' Approaches to Study Questionnaire to get indicators of their approaches to study (Biggs, 1987) and a modified version of Ramsden's CEQ to obtain indicators of their perceptions of their teaching and learning contexts (Ramsden, 1991). The questionnaires were modified to get the students to focus on their approaches and perceptions in relation to the subject they were studying rather than overall indicators. Examples of items from the Approaches to Study Questionnaire are:

Surface Approach
32. Although I generally remember facts and details, I find it difficult to fit them together into an overall picture.
35. The best way for me to understand what technical terms mean is to remember the textbook definitions.

Deep Approach
28. I try to relate ideas in this subject to those in other subjects, wherever possible.
34. In trying to understand new ideas, I often try to relate them to real life situations to which they might apply.

Examples of items for the perceptions of context are:

Good Teaching
3. The teaching staff of this subject motivated me to do my best work.
15. The staff made a real effort to understand difficulties students might be having with their work.

Clear Goals
6. I usually had a clear idea of where I was going and what was expected of me in this subject.
26. The staff made it clear right from the start what they expected from students.

Inappropriate Workload
4. The workload was too heavy.
25. The sheer volume of work in this subject meant that it couldn't all be thoroughly comprehended.

Inappropriate Assessment
8. To do well in this subject, all you really need is a good memory.
19. Too many staff asked me questions just about facts.

Student Independence
16. This subject has encouraged me to develop my own academic interests as far as possible.
20. Students had a great deal of choice over how they learned in this subject.

The results were analysed using factor analyses, and are shown in Table 1.

A factor analysis is designed to show the structure of the relationship between variables. The analysis shows two clear factors. The first relates perceptions of good teaching, clear goals and independence with a deep approach and the second inappropriate workload (too heavy) and inappropriate assessment (measuring rote learnt material) with a surface approach. It is clear that the way students perceive their teaching and learning environment or context is associated with the way they approach their studies in that context.

In a further study at La Trobe University, the University included a set of more client centred questions along with the CEQ in our survey of graduates. Table 2 shows the results of the factor analyses of this data in 3 separate years.

This analysis suggests that students' perceptions of the teaching, goals and generic skills development - those perceptions related to the way students approach their studies, are independent of their perceptions of the administrative procedures, student facilities, teaching and learning facilities, library facilities and student services - the more client or customer centred items. It also shows that their satisfaction with the course is independent of the client or customer centred items, but that their overall experience loads equally with both perspectives. The workload and assessment scales are independent of both sets of perspectives. Thus, combining this with the results of the previous study, it is clear that it is students' perceptions or experiences of the teaching and learning context that are important in terms of the quality of student learning outcomes.

In a separate study some colleagues and I looked at how these perceptions and approaches relate to prior and post knowledge and understanding and achievement in first year university science courses. We used open-ended questions and concept maps to obtain indicators of first year physics students' understanding of key concepts in electricity and magnetism, and their examination results to obtain an indicator of their achievement. We also used the questionnaires previously discussed to obtain indicators of their perceptions and approaches. Table 3 shows the results of a cluster analysis of all these variables. (While a factor analysis looks at the relationship between variables, a cluster analysis results in clusters of individual students with like scores.)

For the purposes of this paper, I wish to focus on cluster 1, the understanding cluster. These results show that in both institutions, those students with the highest achievement and highest quality understanding were those who reported adopting a deep approach, and who perceive the context as providing good teaching, clear goals and independence with appropriate workload and assessment.

What do these studies and this model say about the evaluation of new technologies in teaching and learning? They highlight the importance, in terms of student learning outcomes, of trying to see how the students perceive and understand the use of new technologies, not how they judge them or how we, as teachers and developers, judge them.

But how can this be done without developing specially designed questionnaires or in-depth interviews? I wish to turn now to two examples of work that I have been associated with in mathematics and physics. The first is from a study of first year students' experiences of studying mathematics. In that study we asked students to respond to two open-ended written questions. The students were given these questions on a single page and given half a page each to respond. The questions were:

1. Think about the maths you've done so far. What do you think maths is?
2. How do you go about learning maths?

The first stage of the analysis was to develop a set of categories of description based upon the responses themselves. The second stage was to return to the responses and to categorise them in relation to these categories. The results of the analysis of these two questions are shown in Tables 4 and 5.

The tables show that when we asked students what they thought mathematics was - their experience of mathematics, we identified a range from a very unsophisticated conception - about numbers - to a very sophisticated conception - helping to explain and understand aspects of the world. Similarly, when we asked students how they study mathematics we identified a range of approaches two of which represent surface approaches and 3 represent deep approaches.

The analysis showed a strong relationship between the way they approached their studies of mathematics and how they conceived of it. The important thing to note is that the results were obtained from an analysis of open-ended written statements by students collected in class, and represent the way the students experienced mathematics, not a judgement by them of our predetermined ways of experiencing it.

A similar study was also conducted in first year physics with very similar results. Those results are shown in Tables 6 and 7.

Tables 4-7 show a range of ways students in the same class conceive of the subject they are studying and a range of ways in which they approach their studies. The approaches fall neatly into the surface/deep distinction made earlier, and based upon that earlier research it can reasonably be inferred that those classifications relate to the quality and quantity of learning outcomes. The point being that it is reasonably simple to collect this more student focused evaluative information, and with a reasonably careful analysis, well substantiated results can be produced.

So far I have argued for a more student focused perspective on evaluation - trying to see the object of study - new technologies in teaching and learning - from the students' perspective. Not just finding out how they rate various parts or measure how much they learn but how do they see, perceive, experience the new technology and its place in teaching and learning. In the next section I will take an example of published work in the use of the new technologies in teaching and learning in university science subjects and show how such a student focused approach could have been included in the evaluation.

A case study for evaluation of new technologies

The example I have chosen is reported in a paper by Redfern (1999) in UniServe Science News. In that paper Redfern identifies two key issues in the use of the Web in teaching and learning. They are:

that while there is a substantial amount of computer based learning materials available on the Internet, a key problem is how to integrate that material into a particular teaching environment - how to use it in a particular subject; and

how to find time in traditional courses to integrate the Web and CFL material into the teaching environment.

Redfern decided to explore this issue in the teaching of an Honours statistics course at the University of Leeds. Redfern developed a web site to link all the various components together. The site was used:

as a source of information;

as a means of communicating with students via email; and

to provide links to CBL modules designed to help students understand key statistical concepts.

In the article, Redfern describes the innovation and reports on the results of the evaluation of students reactions to this teaching. He noted that they:

liked the freedom to organise their own learning;

used email because they felt freer to ask questions;

maintained lecture attendance; and

found the CBL material interesting and useful.

But from a student focused learning perspective, he does not seem to have addressed in the evaluation the key issues of concern that he identified in the opening of the paper. There is little or no evaluation of how the students perceived or experienced the integration of the various components or of the aims of each component - the key focus of the innovation. Interestingly, in the article the author focuses on integrating the various components, not on helping students experience an integrated curriculum. The curriculum may have been very well integrated from the teacher's perspective, but not necessarily from the student's perspective. Furthermore, how the students saw and experienced that integration could have been very different to how the teachers designed the integration.

Questions such as the following remained unanswered:

1. How did students see the relationship between the lectures, CBL, web-based material, etc.?
2. What did students understand the role or aims of the CBL and web-based material to be in the subject as a whole?
3. How did students approach their studies in the subject - what were they trying to learn from the various components?
4. Did they feel as though they had enough time to deal with the various components?
5. What components did they perceive to be of most importance and what were of least importance to their learning, and most importantly why?

The questions raised by Redfern in the Introduction are very important ones, but ones not answered by the subsequent evaluative discussion.

From a student learning perspective, a modified version of the CEQ could have been administered to find out how they were perceiving the teaching assessment, workload and goals overall. An open-ended questionnaire, with 4-5 open-ended questions could also have been distributed to find out from the student perspective, how they were experiencing the integration of the various components.

Examples of items included in the questionnaire could have been:

1. If you were to explain to a friend how the CBL helped your learning, what sorts of things would you say?
2. How did you approach your learning using CBL? What sorts of things did you do and why did you do them?
3. What sorts of things did you learn from attending lectures?
4. What sorts of things did you learn by using the CBL?
5. What sorts of things did you learn when using the emails, etc.?

The important point to note from these questions is that they are written from a student perspective, and are designed to elicit from the student how they perceive or experience the innovations, not getting them to make judgements on how the teachers and or designers designed the innovation.

Conclusions

In this paper I have taken as my point of departure a student focused perspective on learning and have argued for a more student focused perspective in the evaluation of new technologies in teaching and learning. I have argued that the quality of student learning outcomes - conceptual understanding and achievement - is closely related to how they perceive and understand the teaching and learning environment that they are in. It is not how well we have articulated the aims and objectives, but how well and what the students understand those aims and objectives to be. It is not how well we have designed assessment to test understanding rather than reproduction, but what our students believe the assessment to be about.

Too much of the evaluation of, and research into, the new technologies in teaching and learning have been conducted from the teacher's or developer's perspective. There has been little research and evaluation looking at the teachers' or students' experience of the new technology and to my knowledge none from a student focused learning perspective.

References

Alexander, S. and McKenzie, J. (1998) An evaluation of information technology projects in Australian higher education. Canberra: Australian Government Publishing Services.
Biggs, J. B. (1987) The Study Process Questionnaire (SPQ) Users' Manual. Hawthorne, Victoria, Australian Council for Educational Research.
Crawford, K., Gordon, S., Nicholas, J. and Prosser, M. (1994) Conceptions of Mathematics and how it is learned: The perspective of students entering university. Learning and Instruction, 4, 331-345.
Harvey, J. (ed.) (1998) Evaluation Cookbook. Learning Technology Dissemination Initiative, Institute for Computer Based Learning, Edinburgh: Heriot-Watt University.
Marton, F., Hounsell, D. and Entwistle, N. J. (eds) (1997) The Experience of Learning: Implications for Teaching and Studying in Higher Education. (2nd edition). Edinburgh: Scottish Academic Press.
McShane, K. (2000) 1999 On-line Learning Project. La Trobe University: Academic Development Unit.
Prosser, M. and Trigwell, K. (1999) Understanding Learning and Teaching: the experience in higher education. Buckingham: Open University Press.
Prosser, M., Trigwell, K., Hazel, E. and Lyons, F. (2000) Students' experiences of studying Physics concepts: the effects of disintegrated perceptions and approaches. European Journal of Educational Psychology, 1, 61-74.
Prosser, M., Walker, P. and Millar, R. (1996) Differences in Students' Perceptions of Learning Physics. Physics Education, 31, 43-48.
Ramsden, P. (1991) A performance indicator of teaching quality in higher education: The Course Experience Questionnaire. Studies in Higher Education, 16, 129-150.
Ramsden, P. (1992) Learning to Teach in Higher Education. London: Routledge.
Redfern, E. (1999) Weaving the Web in learning and teaching statistics. UniServe Science News, 13, 8-13.

© 1997 - 2009 UniServe Science

Page Maintained By: PhySciCH@mail.usyd.edu.au
Last Update: Thursday, 12-Oct-2000 15:00:51 EST
URL: http://science.uniserve.edu.au/pubs/procs/wshop5/prosser.html