Investigation of the Applicability of the SOLO Taxonomy in the Qualitative Content Analysis

Investigation of the applicability of the SOLO taxonomy in the qualitative content analysis of asynchronous online discussion

Dr Christina Mainka

December 2010

A thesis submitted in partial fulfillment of Edinburgh Napier University, for the award of Master of Philosophy.

Acknowledgements

First and foremost I am indebted to my supervisors, Dr. Sandra Cairncross and Dr.Keith Smyth, for their support and endless patience throughout this project. Despite the many delays and unforeseen events over a period that lasted much longer than anyone could have expected, they never failed to listen, guide and encourage. Furthermore, this work could not have been completed without the valuable contributions to the main coding events and subsequent team discussions made by my colleague and friend, Dr. Norrie Brown.

Embarking on this journey was borne out of an idea that turned into much more thanks to the vision and initial support of Prof. Fred Percival, former director of Educational Development at EdinburghNapierUniversity. Prof Percival listened to my questions and recommended I should find the answers-and I think I have, at least some.

The page isn’t long enough to list the additional tasks taken on bycolleagues in Academic Practice and Professional Development during the final stages of this work and I thank them all for putting up with me and shouldering more than their fair share as a result. Finally, with the help and understanding of Dr. Karen Aitchison, Head of Academic Practice, the last hurdle or two could be overcome in order for this research work to be completed.

Thank you.

Abstract

Despite a growing research base in support of higher order thinking processes fostered in asynchronous online dialogue, the research community seems far from a consensus that would unequivocally anchor a higher level of understanding with asynchronous communications technology. Transcripts of online discussion messages provide rich sources of data about a range of communicative issues. Content analysis to date has seen protocols emerge for rate of interaction, critical thinking, collaboration and tutor presence, for example.

The research presented here focuses on depth of understanding as mapped against the SOLO (Structure of the observed learning outcome) taxonomy by Biggs and Collis (1982) which stands out as a content analysis tool as it focuses on structural complexity of text rather than subject specific, manifest content. The taxonomy’s five levels offer a continuum from surface to increasingly deep understanding against which written responses are mapped.A previously published small-scale study in which SOLO was successfully applied in the qualitative content analysis of online discussion transcripts of an environmental science distance education course (Brown, Smyth & Mainka, 2006) prompted the more comprehensivemain investigation reported here.

The SOLO framework was applied to a representative set of data constituting student responses in three constructively aligned, undergraduate physical science distance education courses. Ithas proven to be a robust, replicable and reliable online content analysis tool in a modified format created to includeinteractive levels that takes account of peer-peer and peer-tutor discourse. A simple coding protocol has beendesigned for reuse and further development by researchers and practitioners. In an extended outcome, the significance offormal collaborative opportunities in fully online distance education course design was linked todeeper levels of understanding in the asynchronous discussion which is seen to confirm the role of online collaboration for high quality learning in the distance education context.

Table of contentspage

Chapter 1 Introduction 02

1.1 The nature of dialogue in online education 02

1.2 Learning in the asynchronous discussion02

1.3 SOLO: A rationale 03

1.4 SOLO: An overview 04

1.5 Research focus 05

1.6 Research question 06

1.7 Thesis outline 06

Chapter 2 Literature review 09

2.1 Overview 09

2.2 Quality of learning 10

2.2.1 Approaches to studying 10

2.2.2 The quality of learning 12

2.2.3 Types of knowledge 13

2.3 Learning paradigms 14

2.3.1 Behaviourism 15

2.3.2 Cognitivism 16

2.3.3 Constructivism 17

2.3.4 The constructivist online science classroom 19

2.3.5 Connectivism 20

2.4 Constructive alignment 21

2.4.1 Constructive alignment in online teaching and learning 21

2.5 Asynchronous online communication in education 23

2.5.1 Overview 23

2.5.2 Advantages 24

2.5.3 Disadvantages 25

2.5.4 The student experience 27

2.5.5 The role of the tutor 28

2.5.6 The role of the task 29

2.5.7Asynchronous vs face to face learning instruction 30

2.5.8 Asynchronous online dialogue for achieving understanding 31

2.5.9 Summary 33

2.6 The SOLO taxonomy 34

2.6.1 Overview 34

2.6.2 Background 34

2.6.3 SOLO and the learning process 35

2.6.4 Learning and Piagetian stages 37

2.6.5 Previous research 39

Chapter 3 Research methodology 42

3.1 Overview 42

3.2 Defining research 42

3.2.1 Qualitative and quantitative research 43

3.3 The research paradigm 44

3.4 Content analysis 45

3.4.1 Overview 45

3.4.2 Qualitative and quantitative analysis 45

3.4.3 Review of qualitative methods 47

3.4.4 Newman, Webb and Cochrane 49

3.4.5 Gunawardena, Lowe and Anderson 50

3.5 The SOLO taxonomy50

3.5.1 Overview 50

3.5.2 The SOLO levels 51

3.5.3 Methodological considerations 52

3.5.4 Rationale 54

3.6 Reliability and validity 55

3.7 Developing a content analysis procedure 57

3.7 1 Overview 57

3.7.2 The object of investigation 58

3.7.3 The unit of analysis 59

3.7.4 Developing a coding technique 59

3.7.5 The pre-pilot coding protocol 60

3.7.6 Pre-pilot results 61

Chapter4 Results: Pilot and coding protocol63

4.1 Overview 63

4.2 The research setting 64

4.2.1 Constructively aligned course design 65

4.2.2 The Thought conference 67

4.3 The pilot 68

4.3.1 Data collection 68

4.3.2 Coding guidelines 68

4.3.3 Results pilot study 71

4.4 Results main study 73

4.4.1 Overview 73

4.4.2 The coding protocol 75

4.4.3 Summary 91

Chapter 5 Results: Learning in the Thought conference93

5.1 Overview 93

5.2 General trends 94

5.2.1 Rate of interaction 95

5.2.2 Peer-peer interaction and tutor intervention 96

5.3 Student level of understanding and trends 97

5.4 Individual student trends100

5.5 Summary102

Chapter 6 Discussion104

6.1Overview104

6.2 The SOLO framework for wider use104

6.3. An evaluation106

6.3.1 Replicability106

6.3.2 Reliability106

6.3.3 Objectivity107

6.3.4 Validity108

6.3.5 Summary111

6.4 Understanding in the Thought conference112

6.4.1 Course design issues113

6.5 Summary116

Chapter 7 Conclusion and further recommendations120

7.1 Overview120

7.2 Main aims120

7.3 Recommendations121

7.4 Conclusion123

References125

Appendices

Appendix A Coding guidelines143

Appendix B Sample case study course syllabus153

Appendix C Coding data compiled in Excel spreadsheets I-V159

Appendix D Paper Mainka (2006)159

Appendix E Paper Brown, Smyth & Mainka (2006)166

Appendix F Paper Mainka, Smyth & Brown (2005)174

List of Tables

Table 3.1 Representative characteristics for SOLO level learning outcomes 53

Table 4.1Overview of research activities that inform or comprise this study 64

Table 4.2 Overview coursework and grading criteria for case studies 66

Table 4.3 Sample Thought Questions 67

Table 4.4 Sample coded messages 70

Table 4.5 Pilot study final code frequency 71

Table 4.6 Modified SOLO framework used in main study 73

Table 4.7 Overview Courses A, B, and C 74

Table 4.8 Overview of units coded by coders 75

Table 4.9 Cohen kappa values benchmark 79

Table 4.10 IRR and Cohen’s kappa value for all coding events80

Table 4.11 Coding consistency as % agreement with final code 81

Table 4.12 Coding consistency for coder-coder pairs 82

Table 4.13 Comparison of proportion of C@X codes 84

Table 4.14 Comparison of proportion of high level codes 85

Table 5.1 Participation rate as a number of posts made per week 93

Table 5.2 Final code frequency for Courses A,B, and C 94

Table 5.3 Final code frequency of non-task orientated messages 95

Table 5.4 Comparison peer-peer interactivity and tutor intervention rate 96

Table 5.5 Relational and higher levels for Courses A, B, and C 97

Table 5.6 SOLO level reached for Course A (post) 98

Table 5.7 Relational or higher reached per student for Course A (pre)100

Table 5.8 Relational or higher reached per student for Course A (post)100

Table 5.9 Relational or higher reached per student for Course B (pre)101

Table 5.10 Relational or higher reached per student for Course C (pre)101

Table 6.1 Summary of the evaluation of SOLO coding protocol105