aul Gorsky and Ina Blau - Online Teaching Effectiveness: A Tale of Two Instructors

[Print Version]

June – 2009

Online Teaching Effectiveness: A Tale of Two Instructors

Paul Gorsky and Ina Blau
Open University of Israel

Abstract

Upon completion of a graduate level course at the Open University of Israel, one instructor received very high student ratings while the other received very low ratings. We utilized this exceptional situation to perform ad hoc analyses of their course forums. The objective of this study was to map the dialogic behavior that occurred and to create suggestions for best practice and for worst practice in terms of active and passive participation, instructor response time, and the extent of teaching presence, social presence, and cognitive presence.

Keywords: Effective online teaching; role of online instructor; community of inquiry model; virtual learning community; dialogic behavior; best practice

Introduction

Teaching effectiveness may be defined as how an instructor can best direct, facilitate, and support students toward certain academic ends, such as achievement and satisfaction. Teaching effectiveness has been investigated extensively in traditional classrooms for more than seven decades (for a meta-analysis of empirical studies from 1995-2004, see Seidel & Shavelson, 2007). Over the past five years, research has become directed toward teaching effectiveness in online or virtual classes. As a preface to our study, we discuss findings and conclusions concerning teaching effectiveness in traditional classrooms. We do so for two reasons: practically, since all findings are relevant to online teaching, and theoretically, since current research has shown the theoretical equivalency of all kinds of instructional systems within the framework of a unified theory of instructional design (Gorsky & Caspi, 2005; Gorsky, Caspi, & Chajut, 2007).

One of the most widely cited sources for teacher effectiveness in traditional classrooms is Chickering and Gamson (1987), who suggested seven principles for good practice in undergraduate education. Given their simplicity and eloquence, no further commentary is added. A good teacher does the following:

  1. encourages student-faculty contact,
  2. encourages cooperation among students,
  3. encourages active learning,
  4. gives prompt feedback,
  5. emphasizes time on task,
  6. communicates high expectations,
  7. respects diverse talents and ways of learning.

The authors added this statement:

These principles are intended as guidelines for faculty members, students, and administrators… to improve teaching and learning. They rest on 50 years of research on the way teachers teach and students learn, how students work … with one another, and how students and faculty talk to each other. (Chickering & Gamson, 1987, p.3)

Given their importance for guiding teaching in traditional classrooms, these principles have since been adapted to web-based and virtual classrooms that rely on diverse instructional technologies. A decade after the principles were published, Chickering and Ehrmann (1996) wrote the following:

Since the seven principles of good practice were created in 1987, new communication and information technologies have become major resources for teaching and learning in higher education. If the power of the new technologies is to be fully realized, they should be employed in ways consistent with the seven principles. (Chickering & Ehrmann, 1996, p.2)

These seven principles are defined formally. How can we define these principles operationally so as to be appropriate for research in online classrooms? We propose, as have others (i.e., Shea, Pickett, & Pelz, 2003), that the community of inquiry model (Garrison, Anderson, & Archer, 2000) reflects the principles of good practice in undergraduate education and can accurately quantify them. Relying on a quantitative content analysis technique, the model affords a detailed set of categories and indicators through which to examine issues of pedagogy, dialogue, and interaction. We now present an overview of the model along with representative research.

The Community of Inquiry Model

Garrison, Anderson, and Archer (2000) developed the community of inquiry model as an online learning research tool. The framework consists of three elements – cognitive presence, teaching presence, and social presence – as well as categories and indicators to define each of the presences and to guide the coding of transcripts. Cognitive presence is defined by Garrison, Anderson, and Archer (2001) as the extent to which participants are able to construct meaning through sustained communication. Teaching presence includes subject matter expertise, the design and management of learning, and the facilitation of active learning (Anderson, Rourke, Garrison, & Archer, 2001). Social presence is the perceived presence of others in mediated communication (Rourke, Garrison, & Archer, 1999), which Garrison et al. (2000) contend supports both cognitive and teaching presence through its ability to instigate, to sustain, and to support interaction. It had its genesis in the work of John Dewey and is consistent with all theoretical approaches to learning in higher education. This framework has provided significant insights and methodological solutions for studying online learning (Garrison & Archer, 2003; Garrison, Cleveland-Innes, Koole, & Kappelman, 2006). The structure of the community of inquiry model has been confirmed through factor analysis (Arbaugh & Hwang, 2006; Garrison, Cleveland-Innes, & Fung, 2004; Shea & Bidjerano, 2009).

Social presence is described as the ability to project one’s self and to establish personal and purposeful relationships (Rourke et al., 1999). The three main categories of social presence are affective communication, open communication, and group cohesion. Richardson and Swan (2003) explored perceptions of social presence in online courses and found that students’ perceptions of social presence were highly correlated with perceived learning and satisfaction with their instructors (see also Steinweg, Trujillo, Jeffs, & Hopfengardner-Warren, 2006). Picciano (2002) found relationships between student perceptions of social presence, learning, and interactions in the course discussions. The positive correlation between perceived social presence, seen according to the community of inquiry model as self projection, and most aspects of perceived learning may lead to the conclusion that social presence affords learning by setting a convenient climate (Caspi & Blau, 2008). However, actual interaction in the course discussions in Picciano's (2002) study was not correlated with actual performance (their scores on a multiple choice exam and on a written assignment). Whether and how actual social interaction might or might not affect actual learning online remains unclear and constitutes an important area for future research (Caspi & Blau, 2008; Swan & Shea, 2005). Swan (2002) studied the apparent shift of social presence over time in online course discussions. She reported that open communication indicators (“affective” and “interactive”) of social presence increased over time, while cohesive indicators decreased. One possible explanation is that the use of such references became less necessary as a galvanized classroom community was formed. Another possible explanation addressed the fact that discussion was more exploratory than collaborative. Contrary to the nature of the shift in social presence reported by Swan (2002), Vaughan (2004) and Vaughan and Garrison (2006) found that the frequency of affective and open communication comments decreased, while group cohesion comments increased. It is important to note that the context of Vaughan’s study (2004) was a blended professional development community. The interpretation was that affective and open communication was necessary to establish a sense of community. It was only after the social relationships were established and the group became more focused on purposeful activities that cohesive comments began to take precedence. Social presence online becomes somewhat transparent as the focus shifts to academic purposes and activities.

Cognitive presence is defined as the exploration, construction, resolution, and confirmation of understanding through collaboration and reflection in a community of inquiry (Garrison et al., 2001). Cognitive presence is grounded in the work of Dewey (1933) on reflective thinking. Four phases are defined: Triggering event, exploration, integration, and resolution. Garrison et al. (2001) argue that the third phase, integration, is the most difficult to detect from a teaching or research perspective. This phase requires active teaching presence to diagnose misconceptions, to provide probing questions, comments, and additional information in an effort to ensure continuing cognitive development, and to model the critical thinking process. Often students will be more comfortable remaining in a continuous exploration mode; therefore, teaching presence is essential in moving the process to more advanced stages of critical thinking and cognitive development.

Teaching presence is defined as “the design, facilitation and direction of cognitive and social processes for the purpose of realizing [students’] personally meaningful and educationally worthwhile outcomes” (Anderson et al., 2001, p.5). Vygotsky’s (1978) scaffolding analogies illustrate an assistive role for teachers in providing instructional support to students from their position of greater content knowledge. Although many authors recommend a “guide on the side” approach to moderating student discussions, a key feature of this social cognition model is the adult, the expert, or the more skilled peer who scaffolds a novice’s learning (Anderson et al., 2001). The community of inquiry model defines three categories of teaching presence: design and organization, facilitating discourse, and direct instruction. The categories of teacher presence have been tested by Anderson et al. (2001) in an analysis of the complete transcripts of two online courses and have proved reasonably reliable and useful in identifying differences in both the quantity and quality of the teaching presence projected by different online instructors. How these differences might relate to community has not yet been hypothesized, but the community of inquiry model might provide a starting point for such investigations (Swan & Shea, 2005).

The body of evidence attesting to the importance of teaching presence for successful online learning is growing rapidly (Garrison & Cleveland-Innes, 2005; Meyer, 2003; Murphy, 2004; Pawan, Paulus, Yalcin, & Chang, 2003; Shea, Pickett, & Pelz , 2004; Swan, 2002; Swan & Shih, 2005; Varnhagen, Wilson, Krupa, Kasprzak, & Hunting, 2005; Vaughan, 2004; Wu & Hiltz, 2004). The consensus is that teaching presence is a significant determinate of perceived learning, student satisfaction, and sense of community. Perceived teaching had a strong direct effect on self-reported learning outcomes (LaPointe & Gunawardena, 2004).  Each category of a tutor’s presence is vital to learning and to the establishment of the learning community; tutors' behavior must be such that they are seen to be “posting regularly, responding in a timely manner and modeling good online communication and interaction” (Palloff & Pratt, 2003, p.118). Without an instructor’s explicit guidance and “teaching presence,” students were found to engage primarily in “serial monologues” (Pawan et al., 2003). Baker (2004) discovered that “instructor immediacy, i.e., teaching presence (Rourke et al., 1999), was a more reliable predictor of effective cognitive learning than whether students felt close to each other. Studies have demonstrated that instructor participation in threaded discussion is critical to the development of social presence (Shea, Li, Swan, & Pickett, 2005; Swan & Shih, 2005) and sometimes not fully appreciated by online faculty (Liu, Bonk, Magjuka, Lee, & Su, 2005). Shea, Li, and Pickett (2006) proposed that teaching presence – viewed as the core role of the online instructor – is a promising mechanism for developing learning community in online environments. The majority of students and instructors in Vesely, Bloom, and Sherlock’s (2007) study identified the same elements for building online community, but students ranked instructor modeling as the most important element in building online community, while instructors ranked it fourth.

To conclude, we note a recent study that questioned whether there are three categories of teaching presence. Shea (2006), who completed an extensive study of teaching presence and online learning, concluded that two categories (“design” and “directed facilitation”) sufficed to define the construct. In this study, we opt for the three categories of teaching presence as defined originally by Garrison et al. (2000).

Extensive research into effective online teaching has been carried out without using the community of inquiry model. We now present representative findings concerning the impact of interaction and feedback on students’ perceptions of, and satisfaction with, online learning. Both constructs are closely interrelated with instructor response time, the rate at which instructors respond to students’ posts.

Effective Online Teaching: Research Findings

Regarding interaction, Ridings and Gefen (2004) found that across all types of communities, information exchange was the most commonly cited reason for participation while social support was the second most popular reason for members in communities devoted to professional topics. Communities are used not only for informational purposes but also as an opportunity for social interaction (Kaye, 2005). Kalman, Ravid, Raban, and Rafaeli (2006) argued that interactivity is an essential characteristic of effective online communication and plays an important role in keeping message threads and their authors together. Interactive communication (online as well as in traditional settings) is engaging, and loss of interactivity results in a breakdown of the communicative process.

Research indicates the existence of a relationship between learners’ perceptions of social presence and their motivation for participation in online discussions (Weaver & Albion, 2005). The importance of social interaction to individuals who participate in online communities explains why sociability may be a key element in determining the success or failure of an online community. It has been found that learner satisfaction depends on student-instructor interactions and that students’ perceptions of ‘good’ interactions have a positive impact on their enthusiasm and learning (Swan, 2001; Tricker et al., 2001; Ussher, 2004).

Qualitative data from Weaver and Albion’s (2005) study showed that students placed a high priority on the role of the course instructor as initiator and maintainer of momentum in discussions. This was most evident in the comments of students who experienced infrequent participation by the course instructor. Northrup (2002) found that online learners felt it was important for instructors to promote collaboration and conversation. When interactive activities are carefully planned, they lead not only to greater learning but also to enhanced motivation (Berge 1999; Northrup, 2002).

Interaction in a forum is closely related to the rate at which new messages are posted. Increased frequency of posting in asynchronous communications can lead to more favorable impressions of communication partners (Liu, Ginther, & Zellhart, 2001; Walther & Bunz, 2005). Researchers have suggested that timing of messages can serve as a proxy for a sense of social presence (Blanchard, 2004), as an indication of attentiveness (Walther & Bunz, 2005) or respect (Bargh & McKenna, 2004), and as a clue to the sociability of a community (Maloney-Krichmar & Preece, 2005). As such, the frequency of messages may serve as a signal for how engaged participants are with the community.

Regarding feedback, a survey conducted by McCollum, Calder, Ashby, and Morgan (1995) showed that students ranked feedback as the highest factor in determining course quality. Similar findings were reported by Tricker et al. (2001), Spangle, Hodne, and Schierling (2002) and Young (2006). At the same time, faculty members found interacting with and providing feedback to students in online classes to be more time consuming than in face-to-face classes (Chabon, Cain, & Lee-Wilkerson, 2001; Jennings & McCuller, 2004; Herrmann & Popyack, 2003; Smith, Ferguson, & Caris, 2002).

Kearsley (2000) and Simonson, Smaldino, Albright, and Zvacek (2000) found that instructors can either enhance or decrease course interaction depending on how consistently, quickly, and helpfully they respond. The most successful online courses are those in which instructor-to-student interaction is both frequent and productive (Northrup, 2002; Swan, 2001). Thus, evidence of high instructor engagement ranged from low to high in the quality, speed, and usefulness of feedback to students. Jiang and Ting (2000) further reported that both perceived learning and perceived interaction with instructors were linked to the actual average numbers of responses per student that instructors made. Swan, Shea, Fredericksen, Pickett, Pelz, and Maher (2000) also found a correlation between students’ perceived interaction with their instructors and the actual frequency of instructor participation in online course discussions, and Picciano (1998) reported that instructors’ activity was related to students’ perceived learning from them in an online graduate level course. The number and type of facilitator postings also increased the level of interaction between students (Gilbert & Dabbagh, 2005).

On the other hand, Eom (2006) found that instructor facilitation and feedback did not affect the perceived satisfaction of students who take web-based courses. Eom found that the most significant factors for increasing student satisfaction with online classes are paying attention to students and responding to their concerns.

The Current Study: Rationale and Research Questions

At the end of an ordinary graduate level course at the Open University of Israel, one instructor received very high student ratings while the other received very low ratings, relative to university standards. Given this exceptional situation, we used the quantitative content analysis technique (Garrison et al., 2000) and data logs to perform post-hoc analyses of their forums. The quantitative content analysis technique, based on the community of inquiry model, has been widely used to analyze forums (Garrison, 2007). Given the reliability and validity of this procedure and that all other relevant variables in the learning environment (course policy, content and difficulty, equivalent numbers of instructor assignments, group size, semi-random assignment to groups) were controlled, we expected to identify the impact of the instructors’ actions on the dialogic behavior in the two forums.

We defined two objectives for this study. The first was to test the hypotheses, theory-based and empirically supported, that in a forum led by a very highly rated instructor, as opposed to a forum led by an instructor held in very low esteem, the following would occur:

  1. Active and passive participation would be significantly higher,
  2. Levels of social presence, teaching presence, and cognitive presence would be significantly higher, and
  3. Instructor response time would be significantly shorter.

The second was to create initial suggestions for comparing one asynchronous, virtual learning community with another. Specifically, we hoped to establish tentative boundary conditions for best practice and for worst practice in terms of active and passive participation in the forum, instructor response time, the extent of social presence, teaching presence, and cognitive presence, and the ratios of each.

Methodology

Background

The Open University of Israel is a distance education university that offers undergraduate and graduate studies to students throughout Israel. The learning environment is blended: The University offers a home study system based on textbooks, face-to-face tutorials, and a web-based instructional environment wherein each course has its own website. Course sites simplify organizational procedures and enrich students’ learning opportunities and experiences. Website use is optional or non-mandatory so that equality among students is preserved. It does not replace textbooks or face-to-face tutorials, which are the pedagogical foundations of the Open University. The website provides forums for asynchronous instructor-student and student-student interactions. Each course has a coordinator who is responsible for all administrative and academic activities and instructors who lead tutorials. Instructors and coordinators are available for telephone consultations at specified days and times. Course coordinators define the number of forums made available and their purpose.

In this particular course, three forums were defined: a central forum, led by the course coordinator and available to all students, which dealt with administrative and organizational issues and two smaller forums, led by each of the instructors, which dealt with subject-matter issues. Prior to the start of the course, the coordinator met with both instructors and defined for them the goal of their forums: to provide a haven wherein subject-matter issues could be criticized, reflected upon, discussed, and debated within the context of the course, beyond the context of the course (students’ personal experience), and in any other way deemed appropriate. Both instructors were expressly told to encourage and to facilitate such activities. It appeared that both accepted the task readily and eagerly. In addition, at the start of the course, the course coordinator communicated to all the students (via the general course forum) the purpose of the instructor-led forums. As well, each instructor clearly communicated to her group of students the goals and expectations she held vis-à-vis the forum. This was done by posting messages in the forum and at the face-to-face tutorial sessions.

Participants

Participants were two female instructors and 42 graduate students enrolled in the course. Of the 42 students, thirty (71%) were female. Students were divided equally into two forums led by each of the two instructors. One possibly significant difference between the two instructors was their prior experience in tutoring Open University courses: one instructor, subsequently held in low esteem, had 4 years experience, while the second instructor, subsequently held in high esteem, was new to the task. She was, however, experienced in the ways of distance education. It is especially noteworthy that previous evaluations of the instructor subsequently held in low esteem were generally average, not below average or low.

Instruments and Procedure

Three instruments were employed for obtaining data: (1) survey data, elicited near the course’s end, rated students’ assessments of the instructors and perceived course difficulty; (2) the course log site recorded the amount of active participation, the amount of passive participation (lurking), the number of threads, and instructor response time; and (3) the quantitative content analysis technique, based on the community of inquiry model (Garrison et al., 2000), was used to code and analyze transcriptions from the two forums. This technique has been widely used; it is reliable and valid (Garrison, 2007). Its implementation, however, requires that several methodological issues be resolved.

One issue is the level of coding (e.g., indicator vs. category). Content analysis, as described by Rourke and Anderson (2004), is time consuming, and coding at the indicator level is difficult, often yielding poor reliability (Murphy & Ciszewska-Carr, 2005). In this study, we coded at the category level (Garrison et al., 2006).

Another issue is the unit of analysis. Rourke et al. (1999) identified five units of analysis used in computer conferencing research: proposition units, sentence units, paragraph units, thematic units, and message units. While there has been some discussion around this issue (Garrison et al., 2006; Fahy, 2001; Rourke, Anderson, Garrison, & Archer, 2001), it remains a challenging decision influenced by research question and context. In the present study, we used the message unit, in accord with Anderson et al.’s (2001) study of teaching presence, Garrison et al.’s (2001) study of cognitive presence, and Rourke et al.’s (1999) study of social presence.

Other issues are objectivity, reliability, and replicability (Rourke et al., 2001). No established standards exist for inter-rater reliability (De Wever, Schellens, Valcke, & Van Keer, 2006). There is no consensus for the percent agreement statistic. Often a cut-off figure of 0.75–0.80 is used to determine reliability; others use 0.70 (Neuendorf, 2002; Rourke et al., 2001). To increase reliability and to control errors brought on by inexperience or misinterpretation, Garrison et al. (2006) suggest a negotiated coding approach: researchers code the transcripts and then actively discuss their respective codes with their fellow judges in order to achieve consensus or near consensus. Gros and Silva (2006) propose the use of a research methodology based on the intervention of the participants, especial course instructors, for analyzing computer supported communication. In this study we used the traditional coding approach (without negotiation or participant intervention): 25% of postings were randomly chosen and re-estimated by a second rater; 92% agreement was achieved (Cohen’s k = 0.93).

Findings

The Open University of Israel routinely offers students the opportunity to rate each course they participate in along several dimensions. Questionnaires are posted in the course forum close to the semester’s end, prior to the final examination. Anonymity is assured. For the course involved, 23 of the 43 students (53.5 %) returned questionnaires.

One dimension, instructor satisfaction, includes a cluster of 12 questions; each is rated on a scale of 0-5, with five indicating high satisfaction.  Overall, the highly esteemed instructor received an average rating of 4.3 (SD: 0.26) while the instructor held in low esteem received a rating of 2.3 (SD: 0.42). Figure 1 shows ratings achieved by the two instructors for each of the 12 questions. The relatively small numbers of students involved precluded the use of t test or Mann Whitney statistics. However, differences between students’ satisfaction vis-à-vis the two instructors for all parameters were apparent.

Figure 1

We note that no specific question or questions address how instructors managed their asynchronous course forums. For the purpose of this research, we assumed that student satisfaction vis-à-vis this parameter would be similar to all the other parameters. Indeed, the findings described below support this assumption.

Data obtained from the course log site enabled measurement of active and passive participation for each forum. Passive participation, lurking, occurs when a student logs onto the forum but does not post a message. Highly significant differences were found between the two forums for active and passive participation by both instructors and students. Forum 1, led by the instructor held in low esteem, included a total of 55 postings; forum 2, led by the highly esteemed instructor, included a total of 125 postings (X2 = 27.22, p < 0.001). Table 1 summarizes these findings for the duration of the semester.

Table 1

Regarding active participation, data showed that while the instructor and students in forum 2 posted until the end of semester, the instructor and students in forum 1 stopped posting approximately a month earlier. This may indicate a total loss of interest in the forum as a learning resource. Indeed, threads, which are essentially dialogues, came to a near halt after the third month in the forum led by the instructor held in low esteem. Regarding passive participation, data indicated that students in forum 2 entered in high rates; students in forum 1, despite the lack of activity, continued to enter the forum until the semester’s end, possibly in the hope that something of consequence had happened.

Next, the quantitative content analysis technique was applied in order to determine the extent of teaching, social, and cognitive presence for each forum. The first analytic procedure viewed both forums in their entirety; that is, all instructor and student postings were evaluated. Table 2 presents these data.

Table 2

First, highly significant differences were found between the two forums for instances of teaching presence (X2 = 7.77, p < .01) and for social presence (X2 = 79.50, p < .001); no significant difference was found for cognitive presence (X2 = 0.04, n.s.). Second, a highly significant difference was found between the two forum’s internal distribution of the presences expressed as percentages (X2(2) = 23.05, p <. 001).

In order to gain deeper insights into these findings, we next investigated the teaching, social, and cognitive presence actively engaged in by each of the instructors. Significant differences were found between the two instructors for all three presences: teaching presence (X2 = 17.75, p < .001), social presence (x2 = 64.10, p < .001) and cognitive presence (x2 = 4.26, p < .05.). Tables 3, 4 and 5 summarize occurrences of each type of presence along with the related categories.

Table 3

Table 4

Table 5

No significant differences between the two forums for levels of all three types of presence during the first month of the semester were found. During the second month, for nine of the ten categories, no significant differences were found; interestingly, for one category of cognitive presence (triggering event), the forum led by the instructor held in low esteem was significantly more active. One last item is tabulated (Table 6), the ratio of triggering events with all instances of cognitive presence.

Table 6

The overall picture that emerges from these data is that during the second half of the semester (months 3 and 4), the instructor held in low esteem became nearly dysfunctional in her forum. Over this interval, she posted three times only (3 instances of social presence/open communication). This finding may indeed be the primary causal factor that led to the near cessation of active participation. The highly esteemed instructor was especially active from semester midpoint to semester end; she more than doubled her active participation in both teaching presence (especially discourse and instruction) and social presence (all three categories).

In order to further explore active participation in the forums, we next investigated the teaching, social, and cognitive presence engaged in by students. No significant differences were found between forums for students’ teaching presence and cognitive presence. A highly significant difference was found (X2 = 21.29, p < .001) for students’ social presence: It was higher in the forum led by the highly esteemed instructor. Table 7 presents these data.

Table 7

The final parameter investigated was instructor response rates. These were determined from the course log. Response rates measure the time interval between students’ postings and instructors’ responses. The median response time for the highly esteemed instructor was 3.25 hours (N = 53) while the median response time for the instructor held in low esteem was 26.30 hours (N = 20). The difference in response time is profound. Furthermore, to the detriment of the lowly esteemed instructor, an additional six students’ postings (23% of the total messages posted) remained unanswered; these were not included in the median statistic. Unanswered postings began to occur in the third month. One student, who posted an unanswered query, wrote:  “Why isn't anyone answering? Is anyone here? Should I turn the lights off?” Figure 2 shows the scatter plot for each instructor’s response time.

Figure 2

Discussion

This ad hoc analysis, very carefully controlled in retrospect, identified the roles played by two online instructors (one held in high esteem and the other held in low esteem) and described their impact on two graduate level course forums wherein participation was voluntary.  Based on theory and empirical findings, it was hypothesized that in a forum led by a highly rated instructor, as opposed to a forum led by an instructor held in low esteem, the following would occur:

  1. Active and passive participation would be significantly higher.
  2. Social presence would be significantly higher.
  3. Teaching presence would be significantly higher.
  4. Cognitive presence would be significantly higher.
  5. Instructor response time would be significantly shorter.

All hypotheses, except for hypothesis 4, were supported clearly and unequivocally. In order to understand the implications of these findings, we next analyze active participation in the forum as a function of hypotheses 2-5. We begin by attempting to explain the unsupported hypothesis, namely that cognitive presence will be significantly higher in the forum led by the highly esteemed instructor. We asked ourselves why this did not happen and proceeded in two directions: (1) the need for defining specific goals or tasks for sustaining student engagement, and (2) perceived course difficulty.

The Equivalent Sparsity of Cognitive Presence in Both Forums

The emergence of cognitive presence in an online learning environment depends on many factors. One factor deemed significant by Garrison and Arbaugh (2007) is what they call the “progressive development of inquiry” (p.162). They found that cognitive presence may be defined in terms of a “cyclical process of practical inquiry, where participants move deliberately from understanding the problem or issue through to exploration, integration and application” (p.162). Research has shown that this is oftentimes difficult to achieve (e.g., Luebeck & Bice, 2005; Vaughan & Garrison, 2005).

In our study, specific tasks, tailored especially to the needs of the course forum, were not pre-prepared. The forum was more open-ended regarding the ways by which subject-matter oriented topics were discussed. This, however, does not preclude the progressive development of cognitive presence spontaneously, as a function of the instructor's teaching presence, especially “facilitating discourse.” We now analyze the extent of cognitive presence in both forums. 

On the one hand, during the first two months of the semester, the instructor held in low esteem initiated eight triggering events. This would indicate that she was indeed motivated to provide her students with learning opportunities. These initiatives, however, met with lukewarm response and petered out rapidly. The ratio of triggering events and total instances of cognitive presence was a mere 1 : 2 (Table 6). The number of threads for her forum (a measure of more extensive dialogue) was a mere 18 (Table 1). For this instructor, the lack of specific, progressively structured inquiry tasks and/or the lack of facilitation skills (teaching presence/facilitating discourse) may have contributed to the relatively limited occurrences of cognitive presence.

On the other hand, for the instructor held in high esteem, the ratio of triggering events and total instances of cognitive presence was 1 : 10 (Table 6). The number of threads in her forum was 38 (Table 1). This would indicate that for the highly esteemed instructor, the lack of specific pre-planned inquiry tasks did not apparently restrict achieving high levels of integration and resolution by way of teaching presence/facilitating discourse. In other words, we wish to reaffirm what we cited above: teaching presence, especially the category “facilitating discourse,” seems to play a highly significant role in achieving and sustaining cognitive presence (i.e., learning). 

The Open University questionnaire also elicits from students their evaluation of “course difficulty.” This course was rated as “average” (3 on a scale of 1-5). This non-difficulty may, in part, also account for the general lack of cognitive presence found in both forums, despite the fact that the explicit goal of both forums was to provide a place to learn. Research has shown that students turned to interpersonal dialogue primarily when they couldn’t cope with conceptual difficulties or with difficult tutor assignments (Gorsky, Caspi, & Tuvi-Arad, 2004; Gorsky, Caspi, & Trumper, 2004, 2006; Gorsky, Caspi, & Smidt, 2007). Given the absence of such difficulties and that participation in the forum was non-mandatory, students seemed to have studied on their own.

In other words, there are reasonable, theory-based explanations for the relative lack of cognitive presence found in both forums. This implies that something else accounted for the extreme satisfaction and dissatisfaction experienced by students in the two forums. The something else may be the two exceptional events that occurred during the third month: The instructor held in low esteem became nearly dysfunctional, while the highly esteemed instructor exhibited very high teacher presence and social presence (see Table 3 and 4).

The Impact of Teaching Presence, Social Presence, and Instructor Response Time

We found highly significant relationships between levels of social presence and teaching presence, on the one hand, and students’ active and passive participation in the forum and their satisfaction with it, on the other hand. These findings clearly support previous ones for both social presence (Richardson & Swan, 2002; Steinweg et al., 2006; Weaver & Albion, 2005) and for  teaching presence (Garrison & Cleveland-Innes, 2005; Meyer, 2003; Murphy, 2004; Pawan et al., 2003; Shea et al., 2004; Swan, 2002; Swan & Shih, 2005; Varnhagen et al., 2005; Vaughan, 2004; Wu & Hiltz, 2004). In addition, findings concerning instructor response time (frequent instructor responses correlate with students’ active and passive participation in the forum) are in accord with previous research (Kearsley, 2000; Northrup, 2002; Simonson et al., 2000; Swan, 2001). In other words, these findings corroborate previous research.

Moving beyond these studies, we wish to suggest a tentative two-tier model of causality, based on recently reported research findings. The model appears in Figure 3.

Figure 3

The first tier is based on the well-documented relationship between instructor response time, on the one hand, and student participation and satisfaction, on the other hand. The second tier of the model includes the mediating variable “students’ perceived learning.” Caspi and Blau (2008) and Swan and Richardson (2003) found that students’ perceived social presence in academic forums correlates with their perceived learning. This correlation points to the tentative conclusion that social presence affords learning by setting a convenient climate. In a similar vein, Shea, Pickett, and Pelt (2003) found that students’ perceived teacher presence also correlates with perceived learning as well as with students’ satisfaction with the forum. This correlation points to the tentative conclusion that teaching presence affords learning by setting a convenient climate. In our study, we found that very high versus very low instructor teaching presence and social presence during the third month affected students’ social presence, which, in turn, affected their active and passive participation as well as their sense of satisfaction. Based on these findings and on the three previous ones cited (Caspi & Blau, 2008; Shea et al., 2003; Swan & Richardson, 2003), we suggest that students’ perceived learning in course forums has a significant impact on their participation (both active and passive) and satisfaction. In other words, “students’ perceived learning” may be a mediating variable as illustrated in Figure 3. This hypothesis, based on a combination of findings from different studies is, at best, speculative. Further research is needed to explore this explanation.

Suggestions for Effective Online Teaching

We suggest that teaching presence, social presence, and instructor response time appear to be important factors in the ratings attained by instructors under the very specific conditions described in this study. Table 8 compares findings for these factors along two dimensions: between-forum ratios and within-forum distributions expressed as percentages. These quantities may aid in evaluating the quality of academic forums that are similar to the ones described in this study.

Table 8

These tentative reference points for best and worst practice are for illustrative purposes only. A great deal of additional research is required in order to define the attributes of these forums in terms of academic discipline, learning environment (blended), perceived course difficulty, communication type (asynchronous), duration, participation (non-mandatory), group size, etc. Such a classification is necessary in order to achieve a standard basis for comparison and reasonable confidence levels. However, the table is suggestive of the eventual possibility of having an “objective” tool for evaluating the quality of a given forum.

References

Anderson, T., Rourke, L., Garrison, D. R., & Archer, W. (2001). Assessing teaching presence in a computer conferencing context. Journal of Asynchronous Learning Networks, 5(2). Retrieved April 2, 2009, from http://sloan-c.org/publications/jaln/v5n2/pdf/v5n2_anderson.pdf

Arbaugh, J. B., & Hwang, A. (2006). Does "teaching presence" exist in online MBA courses? The Internet and Higher Education, 9(1), 9-21.

Baker, J. D. (2004). An investigation of relationships among instructor immediacy and affective and cognitive learning in the online classroom. The Internet and Higher Education, 7(1), 1-13. 

Bargh, J. A., & McKenna, K.Y.A. (2004). The Internet and social life. Annual Review of Psychology, 55, 573–590.

Berge, Z.L. (1999). Interaction in post-secondary Web-based learning. Educational Technology, 39(1), 5-11.

Caspi, A., & Blau, I. (2008). Online discussion groups: The relationship between social presence and perceived learning. Social Psychology of Education, 11, 323-346.

Chickering, A. W., & Gamson, Z. F. (1987). Seven Principles of good practice in undergraduate education. AAHE Bulletin, 39(7), 3-7.

Chickering, A., & Ehrmann, S. (1996). Implementing the seven principles: Technology as lever. AAHE Bulletin, 49(2), 3-6.

Chabon, S., Cain, R., & Lee-Wilkerson, D. (2001). Facilitating those dreaded discussions on diversity, through threaded discussions: An inter-institutional, internet-based model. Distance Education, 22(1), 137-143.

De Wever, B., Schellens, T., Valcke, M., & Van Keer, H. (2006). Content analysis schemes to analyze transcripts of online asynchronous discussion groups: A review. Computers & Education, 46, 6-28.

Dewey, J. (1933). How we think (Rev. ed.). Boston: D.C. Heath.

Eom, S. (2006). The role of the instructors as a determinant of students’ satisfaction in university online education. Proceedings of the Sixth IEEE international Conference on Advanced Learning Technologies-ICALT, Washington, DC: IEEE Computer Society, 985-988.

Fahy, P. J. (2001). Addressing some common problems in transcript analysis. International Review of Research in Open and Distance Learning, 1(2). Retrieved April 2, 2009, from http://www.irrodl.org/index.php/irrodl/article/view/321/530

Garrison, D. R. (2007). Online community of inquiry review: Social, cognitive, and teaching presence issues. Journal of Asynchronous Learning Networks, 11(1), 61-72. Available at http://www.ucalgary.ca/~nvaughan/coiissues.pdf

Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education, 2(2-3), 87-105.

Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking and computer conferencing: A model and tool to assess cognitive presence. American Journal of Distance Education, 15(1), 7-23. Available at http://communityofinquiry.com/files/CogPres_Final.pdf

Garrison, D. R., & Arbaugh, J. B. (2007). Researching the community of inquiry framework: Review, issues, and future directions. The Internet and Higher Education, 10(3), 157-172.

Garrison, D. R., & Archer, W. (2003). A community of inquiry framework for online learning. In M. Moore (Ed.), Handbook of distance education. New York: Erlbaum.

Garrison, D. R., & Cleveland-Innes, M. (2005).  Facilitating cognitive presence in online learning: Interaction is not enough. American Journal of Distance Education, 19(3), 133-148.

Garrison, D. R., Cleveland-Innes, M., & Fung, T. (2004). Student role adjustment in online communities of inquiry: Model and instrument validation. Journal of Asynchronous Learning Networks, 8(2), 61-74. Available at http://www.sloan-c.org/publications/jaln/v8n2/pdf/v8n2_garrison.pdf

Garrison, D. R., Cleveland-Innes, M., Koole, M., & Kappelman, J. (2006). Revisiting methodological issues in the analysis of transcripts: Negotiated coding and reliability. The Internet and Higher Education, 9(1), 1-8.

Gilbert, P. K., & Dabbagh, N. (2005). How to structure online discussions for meaningful discourse: A case study. British Journal of Educational Technology, 36(1), 5-18.

Gorsky, P., & Caspi, A. (2005). Dialogue: A theoretical framework for distance education instructional systems. British Journal of Educational Technology, 36(2), 137-144.

Gorsky, P., Caspi, A., & Chajut, E. (2007). The theory of instructional dialogue: Toward a unified theory of instructional design. In R. Zheng & S. Pixy-Ferris (Eds.), Understanding online instructional modeling: Theories and practices (pp. 47-68). Hershey, PA: Idea Group, Inc. Available at http://books.google.com/books?id=WF4Y03EYj08C

Gorsky, P., Caspi, A., & Trumper, R. (2004). University students' use of dialogue in a distance education physics course. Open Learning, 19(3), 265-277.

Gorsky, P., Caspi, A., & Trumper, R. (2006). Campus-based university students’ use of dialogue. Studies in Higher Education, 31(1), 71-87.

Gorsky, P., Caspi, A., & Smidt, S. (2007). Use of instructional dialogue by university students in a difficult distance education physics course. Journal of Distance Education, 22(1), 1-12.

Gorsky, P., Blau, I., Caspi, A., & Antonovsky, A. (in progress). The impact of academic discipline on the dialogic behavior in academic course forums.

Gros, B., & Silva, J. (2006). El problema del análisis de las discusiones asincrónicas en el aprendizaje colaborativo mediado. Revista de Educación a Distancia- RED, 16. Retrieved April 2, 2009, from http://www.um.es/ead/red/16/gros.pdf

Herrmann, N., & Popyack, J. L. (2003). Electronic grading: When the tablet is mightier than the pen. Syllabus: Technology for Higher Education, 16, 1-16.

Jennings , S. E., & McCuller, M. Z. (2004). Meeting the challenges of grading online business communication assignments. Proceedings of the 69th Annual Convention, Association for Business Communication (pp. 9-14). Cambridge, Massachusetts.

Jiang, M., & Ting, E. (2000). A study of factors influencing students’ perceived learning in a web-based course environment. International Journal of Educational Telecommunications, 6(4), 317-338.

Kalman, Y. M., Ravid, G., Raban, D. R., & Rafaeli, S. (2006). Pauses and response latencies: A chronemic analysis of asynchronous CMC. Journal of Computer-Mediated Communication, 12(1). Retrieved April 2, 2009, from  http://jcmc.indiana.edu/vol12/issue1/kalman.html

Kaye, B. K. (2005, August). Web side story: An exploratory study of why weblog users say they use weblogs. Paper presented at the AEJMC Annual Conference, San Antonio, TX. Retrieved April 2, 2009, from  http://www.journalism.wisc.edu/blog-club/Site/Kaye2.pdf

Kearsley, G. (2000). Learning and teaching in cyberspace. Toronto, Ontario: Wadsworth, Thompson Learning.

LaPointe, D. K., & Gunawardena, C. N. (2004). Developing, testing and refining of a model to understand the relationship between peer interaction and learning outcomes in computer-mediated conferencing. Distance Education, 25(1), 83–106.

Liu, X., Bonk, C. J., Magjuka, R. C., Lee, S., & Su, B. (2005). Exploring four dimensions of online instructor roles: A program level case study. Journal of Asynchronous Learning Networks, 9(4), 29-48.

Liu, Y., Ginther, D., & Zelhart, P. (2001). How do frequency and duration of messaging affect impression in computer-mediated communication? Journal of Universal Computing, 7(10), 893–913. Available at http://www.jukm.org/jucs_7_10/how_do_frequency_and/Liu_Y.pdf

Luebeck, J. L., & Bice, L. R. (2005). Online discussion as a mechanism of conceptual change among mathematics and science teachers. Journal of Distance Education, 20(2), 21−39.

Maloney-Krichmar, D., & Preece, J. (2005). A multilevel analysis of sociability, usability, and community dynamics in an online health community. ACM Transactions on Computer-Human Interaction, 12(2), 201–232.

McCollum, A., Calder, J., Ashby, A., Thorpe, M., & Morgan, A. (1995, June). Quality and effectiveness in vocational education. Paper presented at the One World Many Voices, the 17th World Conference for Distance Education, Birmingham, UK.

Meyer, K. A. (2003). Face-to-face versus threaded discussions: The role of time and higher-order thinking. Journal of Asynchronous Learning Networks, 7(3), 55-65.

Murphy, E. (2004). Recognizing and promoting collaboration in an online asynchronous discussion. British Journal of Educational Technology, 35(4), 421-431.

Murphy, E., & Ciszewska-Carr, J. (2005). Sources of difference in reliability: Identifying sources of difference in reliability in content analysis of online asynchronous discussions. International Review of Research in Open and Distance Learning, 6(2). Retrieved April 2, 2009, from http://www.irrodl.org/index.php/irrodl/article/view/233/855

Neuendorf, K. A. (2002). The content analysis guidebook. Thousand Oaks, CA: Sage Publications.

Northrup, P. (2002). Online learner’s preferences for interaction. Quarterly Review of Distance Education, 3(2), 219-229.

Palloff, R., & Pratt, K. (2003). The virtual student: A profile and guide to working with online learners. San Francisco, CA: Jossey-Bass Inc.

Pawan, F. T., Paulus, M., Yalcin, S., & Chang, C. (2003). Online learning: Patterns of engagement and interaction among in-service teachers. Language Learning & Technology, 7(3), 119-140.

Picciano, A. G. (1998). Developing an asynchronous course model at a large, urban university. Journal of Asynchronous Learning Networks, 2(1), 1-14.

Picciano, A.G. (2002). Beyond student perceptions: Issues of interaction, presence, and performance in an online course. Journal of Asynchronous Learning Networks, 6(1). Retrieved April 2, 2009, from http://www.aln.org/publications/jaln/v6n1/pdf/v6n1_picciano.pdf

Richardson, J.C., & Swan, K. (2003). Examining social presence in online courses in relation to students' perceived learning and satisfaction. Journal of Asynchronous Learning Networks, 7(1). Retrieved April 2, 2009, from http://www.aln.org/publications/jaln/v7n1/pdf/v7n1_richardson.pdf

Ridings, C. M., & Gefen, D. (2004). Virtual community attraction: Why people hang out online. Journal of Computer-Mediated Communication, 10(1). Retrieved April 2, 2009, from http://jcmc.indiana.edu/vol10/issue1/ridings_gefen.html

Rourke, L., & Anderson, T. (2004). Validity issues in quantitative computer conference transcript analysis. Educational Technology Research and Development, 52(1) 5-18.

Rourke, L., Anderson, T., Garrison, D. R., & Archer, W. (1999). Assessing social presence in asynchronous test-based computer conferencing. Journal of Distance Education, 14(2), 50-71. Available at http://www.jofde.ca/index.php/jde/article/viewArticle/153/341

Rourke, L., Anderson, T., Garrison, D. R., & Archer, W. (2001). Methodological issues in the content analysis of computer conference transcripts. International Journal of Artificial Intelligence in Education 12(1), 8-22.

Seidel, T., & Shavelson, R. (2007). Teaching effectiveness research in the past decade: The role of theory and research design in disentangling meta-analysis Results. Journal of Applied Behavioral Science, 43(1), 89-107.

Shea, P. (2006). A study of students' sense of learning community in online environments. Journal of Asynchronous Learning Networks, 10(10). Retrieved April 2, 2009, from http://www.sloan-c.org/publications/jaln/v10n1/v10n1_4shea_member.asp

Shea, P., & Bidjerano, T. (2009). Community of inquiry as a theoretical framework to foster “epistemic engagement” and “cognitive presence” in online education. Computers and Education, 52(3), 543-553.

Shea, P., Li, C. S., & Pickett, A. M. (2006). A study of teaching presence and student sense of learning community in fully online and web-enhanced college courses. The Internet and Higher Education, 9(3), 175−190.

Shea, P., Li, C. S., Swan, K., & Pickett, A. M. (2005). Developing learning community in online asynchronous college courses: the role of teaching presence. Journal of Asynchronous Learning Networks, 9(4). Retrieved April 2, 2009, from http://www.sloan-org/publications/jaln/v9n4/pdf/v9n4_shea.pdf

Shea, P., Pickett, A. M., & Pelt, W. E (2003). A follow-up investigation of teaching presence in the SUNY Learning Network. Journal of the Asynchronous Learning Network, 7(2). Retrieved April 2, 2009, from http://www.sloan-c.org/publications/jaln/v7n2/pdf/v7n2_shea.pdf

Shea, P. J., Pickett, A. M, & Pelz, W. E. (2004). Enhancing student satisfaction through faculty development: The importance of teaching presence. In J. Bourne & J. C. Moore (Eds.), Elements of quality online education: Into the mainstream, Volume 5 (pp.39-59). Needham, MA: Sloan-C.

Simonson, M., Smaldino, S. Albright, M., & Zvacek, S. (2000). Teaching and learning at a distance: Foundations of distance education. Upper Saddle River, NJ: Merrill, Prentice Hall.

Smith, G., Ferguson, D.,& Caris, M. (2002). Teaching over the web versus in the classroom: Differences in the instructor experience. International Journal of Instructional Media, 29(1), 61-67.

Spangle, M., Hodne, G., & Schierling, D. (2002, November). Approaching value-centered education through the eyes of an electronic generation: Strategies for distance learning. Paper presented at the Annual Meeting of the National Communication Association. New Orleans, LA.

Steinweg, S. B., Trujillo, L., Jeffs, T., & Hopfengardner-Warren, S. (2006). Maintaining the personal touch in a growing program: Strategies for establishing social presence in online classes. Journal of the Research Center for Educational Technology, 2(2). Retrieved April 2, 2009, from http://www.rcetj.org/?type=art&id=79598&

Swan, K. (2001). Virtual interaction: Design factors affecting student satisfaction and perceived learning in asynchronous online courses. Distance Education, 22, 306-331.

Swan, K. (2002). Building communities in online courses: The importance of interaction. Education, Communication and Information, 2(1), 23-49. Available at http://www.kent.edu/rcet/Publications/upload/SocPres%20ECI.pdf

Swan, K., Shea, P., Fredericksen, E., Pickett, A., Pelz, W., & Maher, G. (2000). Building knowledge building communities: Consistency, contact and communication in the virtual classroom. Journal of Educational Computing Research, 23(4), 389-413.

Swan, K., & Shea, P. (2005). The development of virtual learning communities. In S.R. Hiltz & R. Goldman, Asynchronous learning networks: The research frontier (pp. 239-260). New York: Hampton Press.

Swan, K, & Shih, L. F. (2005). On the nature and development of social presence in online course discussions. Journal of Asynchronous Learning Networks, 9(3). Retrieved April 2, 2009, from http://www.sloan-c.org/publications/JALN/v9n3/v9n3_swan.asp

Tricker, M., Rangecroft, M., & Long, P. (2001). Evaluating distance education courses: the student perception. Assessment and Evaluation in Higher Education, 26(2), 165–177.

Ussher, B. (2004, July). Interactions, student enthusiasm and perceived learning in an online teacher education degree. Paper presented at the Third Pan-Commonwealth Forum – PCF Conference. Hamilton, New Zealand. Retrieved April 2, 2009, from http://www.col.org/pcf3/Papers/PDFs/Ussher_Bill.pdf

Varnhagen, S., Wilson, D., Krupa, E., Kasprzak, S., & Hunting, V. (2005). Comparison of student experiences with different online graduate courses in health promotion. Canadian Journal of Learning and Technology, 31(1), 99-117.

Vaughan, N. (2004). Investigating how a blended learning approach can support an inquiry process within a faculty learning community. Doctoral dissertation, University of Calgary. Retrieved April 2, 2009, from http://www.ucalgary.ca/~nvaughan/norm/nvaughandissertation.pdf

Vaughan, N., & Garrison, D. R. (2005). Creating cognitive presence in a blended faculty development community. Internet and Higher Education, 8, 1−12.

Vaughan, N., & Garrison, D. R. (2006). How blended learning can support a faculty development community of inquiry. Journal of Asynchronous Learning Networks, 10(4). Retrieved April 2, 2009, from http://www.sloan-c.org/publications/JALN/v10n4/v10n4_vaughan.asp

Vesely, P., Bloom, L., & Sherlock, J. (2007). Key elements of building online community: Comparing faculty and student perceptions. MERLOT Journal of Online Learning and Teaching, 3(3). Retrieved April 2, 2009, from http://jolt.merlot.org/vol3no3/vesely.pdf

Vygotsky, L. (1978). Mind in Society: The development of higher psychological processes. Cambridge MA: Harvard University Press.

Walther, J. B., & Bunz, U. (2005). The rules of virtual groups: Trust, liking, and performance in computer-mediated communication. Journal of Communication, 55(4), 828–846. Available at  http://bunz.comm.fsu.edu/JoC2005_55_4_virtual.pdf

Weaver, C. M., & Albion, P. R. (2005). Momentum in online discussions: The effect of social presence on motivation for participation.  Proceedings of Australasian Society for Computers in Learning in Tertiary Education Conference (ASCILITE 2005), Balance, Fidelity, Mobility: Maintaining the momentum? (pp. 703-706). Sydney, Australia. Available at http://www.ascilite.org.au/conferences/brisbane05/blogs/proceedings/81_Weaver.pdf

Wu, D., & Hiltz, S. R. (2004). Predicting learning from asynchronous online discussions. Journal of Asynchronous Learning Networks, 8(2), 139-152. Available at http://www.aln.org/publications/jaln/v8n2/pdf/v8n2_wu.pdf

Young, S. (2006). Student views of effective online teaching in higher education American. Journal of Distance Education, 20(2), 65-77.